Initial design ideas for A/B/n experiments

[sriumcp]

Problem: Service meshes have a whole bunch of features when it comes to traffic splitting and collection of system metrics such as latency and error-rate. However, they provide no support when it comes to observing and collecting business metrics (which originate deep inside application code, and not easily observable by ingress/service mesh/networking components).

This poses a problem for A/B/n testing. How is a user supposed to split traffic and record metrics during A/B/n tests.

Design proposal:

Assumption: User application has a frontend and backend component (as shown below) and we want to A/B/n test the backend. Frontend service is another name for a service exposed externally. Backend service is another name for a cluster local service.

Note: For A/B/n testing of a frontend service, the current ways of using service mesh to split traffic, and current ways of reading metrics from external DBs (like elastic) work fine. K8s provides "backward APIs" which can be used to effectively for this scenario. MLServing use-cases do not generally fall in this category; ML services are generally backend services which is the main motivation for this entire discussion / use-case.

---

The design proposal for A/B/n testing of backend services is illustrated in the following picture.

Brief description of components:

1. A/B/n SDKs: new component
   • Method to route to a backend version
   • Method to update business metrics
   • Method to (periodically) register a backend (supports zero-downtime releases)
2. A/B/n service: new component
   • "Tracks" new versions of backend services (supports zero-downtime releases)
   • Collects metrics for backend service versions
   • Responds to RegisterBackend() call
     • which versions are currently available

Design considerations:

1. Front-end code should be decoupled from backend versions (i.e., not change when new backend versions become available, are rolled forward or rolled back).
2. User should be able to get started with "default" A/B/n experiments with very little understanding of Iter8 (e.g., simply specify required confidence level, and name of the backend app, and kubernetes resource type for the backend, while configuring the A/B/n service).

[sriumcp]

Source files used for pics above ...
what-is-iter8.pptx