Scenario: Hybrid (A/B + SLOs) testing and progressive rollout of Knative services
Hybrid (A/B + SLOs) testing enables you to combine A/B or A/B/n testing with a reward metric on the one hand with SLO validation using objectives on the other. Among the versions that satisfy objectives, the version which performs best in terms of the reward metric is the winner. In this tutorial, you will:
Perform hybrid (A/B + SLOs) testing.
Specify user-engagement as the reward metric.
Specify latency and error-rate based objectives; these metrics will be collected using Iter8's built-in metrics collection feature.
Combine hybrid (A/B + SLOs) testing with progressive rollout. Iter8 will progressively shift traffic towards the winner and promote it at the end as depicted below.
apiVersion:v1kind:Namespacemetadata:name:iter8-knative---apiVersion:iter8.tools/v2alpha2kind:Metricmetadata:name:user-engagementnamespace:iter8-knativespec:params:-name:nrqlvalue:|SELECT average(duration) FROM Sessions WHERE version='$name' SINCE $elapsedTime sec agodescription:Average duration of a sessiontype:GaugeheaderTemplates:-name:X-Query-Keyvalue:t0p-secret-api-keyprovider:newrelicjqExpression:".results[0]|.[]|tonumber"urlTemplate:https://my-newrelic-service.commock:-name:sample-app-v1level:15.0-name:sample-app-v2level:20.0
Metrics in your environment
You can define and use custom metrics from any database in Iter8 experiments.
For your application, replace the mocked user-engagement metric used in this tutorial with any custom metric you wish to optimize in the hybrid (A/B + SLOs) test. Documentation on defining custom metrics is here.
apiVersion:iter8.tools/v2alpha2kind:Experimentmetadata:name:hybrid-expspec:target:default/sample-appstrategy:testingPattern:A/BdeploymentPattern:Progressiveactions:loop:-task:metrics/collectwith:versions:-name:sample-app-v1url:http://sample-app-v1.default.svc.cluster.local-name:sample-app-v2url:http://sample-app-v2.default.svc.cluster.localfinish:# run the following sequence of tasks at the end of the experiment-if:CandidateWon()run:"kubectlapply-fhttps://raw.githubusercontent.com/iter8-tools/iter8/master/samples/knative/quickstart/candidate.yaml"-if:not CandidateWon()-run:"kubectlapply-fhttps://raw.githubusercontent.com/iter8-tools/iter8/master/samples/knative/quickstart/baseline.yaml"criteria:rewards:# Business rewards-metric:iter8-knative/user-engagementpreferredDirection:High# maximize user engagementobjectives:-metric:iter8-system/mean-latencyupperLimit:50-metric:iter8-system/latency-95th-percentileupperLimit:100-metric:iter8-system/error-rateupperLimit:"0.01"requestCount:iter8-system/request-countduration:maxLoops:3intervalSeconds:1iterationsPerLoop:1versionInfo:# information about app versions used in this experimentbaseline:name:sample-app-v1weightObjRef:apiVersion:serving.knative.dev/v1kind:Servicename:sample-appnamespace:defaultfieldPath:.spec.traffic[0].percentcandidates:-name:sample-app-v2weightObjRef:apiVersion:serving.knative.dev/v1kind:Servicename:sample-appnamespace:defaultfieldPath:.spec.traffic[1].percent