Skip to content

SLO Validation with Traffic Shifting

Scenario: SLO validation with progressive traffic shift

This tutorial illustrates an SLO validation experiment with two versions; the candidate version will be promoted after Iter8 validates that it satisfies service-level objectives (SLOs). You will:

  1. Specify latency and error-rate based service-level objectives (SLOs). If the candidate version satisfies SLOs, Iter8 will declare it as the winner.
  2. Use Iter8's built-in capabilities for collecting latency and error-rate metrics.
  3. Combine SLO validation with progressive traffic shifting.

SLO validation with progressive traffic shift

Platform setup
  1. Setup Kubernetes cluster
  2. Install Iter8 in Kubernetes cluster
  3. Install Knative in Kubernetes cluster
  4. Get Helm 3.4+.
  5. Get iter8ctl
  6. Fork the Iter8 GitHub repo. Clone your fork, and set the ITER8 environment variable as follows. 
    export USERNAME=<your GitHub username>

    git clone git@github.com:$USERNAME/iter8.git
cd iter8
export ITER8=$(pwd)

Follow these steps to install Iter8 and Knative in your K8s cluster.

1. Create app versions

Deploy two versions of a Knative app.

kubectl apply -f $ITER8/samples/knative/quickstart/baseline.yaml
kubectl apply -f $ITER8/samples/knative/quickstart/experimentalservice.yaml
kubectl wait --for=condition=Ready ksvc/sample-app
Look inside baseline.yaml
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
apiVersion: serving.knative.dev/v1
kind: Service
metadata:
  name: sample-app
  namespace: default
spec:
  template:
    metadata:
      name: sample-app-v1
    spec:
      containers:
      - image: gcr.io/knative-samples/knative-route-demo:blue 
        env:
        - name: T_VERSION
          value: "blue"
Look inside experimentalservice.yaml
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
apiVersion: serving.knative.dev/v1
kind: Service
metadata:
  name: sample-app
  namespace: default
spec:
  template:
    metadata:
      name: sample-app-v2
    spec:
      containers:
      - image: gcr.io/knative-samples/knative-route-demo:green 
        env:
        - name: T_VERSION
          value: "green"
  traffic:
  - tag: current
    revisionName: sample-app-v1
    percent: 100
  - tag: candidate
    latestRevision: true
    percent: 0

2. Launch experiment

Launch the SLO validation experiment. This experiment will generate requests for your application versions, collect latency and error-rate metrics, and progressively shift traffic and promote the candidate version after verifying that it satisfies SLOs.

kubectl apply -f $ITER8/samples/knative/quickstart/experiment.yaml
Look inside experiment.yaml
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
apiVersion: iter8.tools/v2alpha2
kind: Experiment
metadata:
  name: quickstart-exp
spec:
  target: default/sample-app
  strategy:
    testingPattern: Canary
    deploymentPattern: Progressive
    actions:
      loop:
      - task: metrics/collect
        with:
          versions: 
          - name: sample-app-v1
            url: http://sample-app-v1.default.svc.cluster.local
          - name: sample-app-v2
            url: http://sample-app-v2.default.svc.cluster.local
      finish: # run the following sequence of tasks at the end of the experiment
      - if: CandidateWon()
        run: "kubectl apply -f https://raw.githubusercontent.com/iter8-tools/iter8/master/samples/knative/quickstart/candidate.yaml"
      - if: not CandidateWon()
        run: "kubectl apply -f https://raw.githubusercontent.com/iter8-tools/iter8/master/samples/knative/quickstart/baseline.yaml"
  criteria:
    requestCount: iter8-system/request-count
    objectives: 
    - metric: iter8-system/mean-latency
      upperLimit: 50
    - metric: iter8-system/latency-95th-percentile
      upperLimit: 100
    - metric: iter8-system/error-rate
      upperLimit: "0.01"
  duration:
    maxLoops: 3
    intervalSeconds: 1
    iterationsPerLoop: 1
  versionInfo:
    # information about app versions used in this experiment
    baseline:
      name: sample-app-v1
      weightObjRef:
        apiVersion: serving.knative.dev/v1
        kind: Service
        name: sample-app
        namespace: default
        fieldPath: .spec.traffic[0].percent
    candidates:
    - name: sample-app-v2
      weightObjRef:
        apiVersion: serving.knative.dev/v1
        kind: Service
        name: sample-app
        namespace: default
        fieldPath: .spec.traffic[1].percent

3. Observe experiment

Follow these steps to observe your experiment.

4. Cleanup

kubectl delete -f $ITER8/samples/knative/quickstart/experiment.yaml
kubectl delete -f $ITER8/samples/knative/quickstart/experimentalservice.yaml
Back to top