SLO validation¶
Scenario: Safely rollout new version of a Knative app with SLO validation
Dark launch a candidate version of your Knative application, validate that the candidate satisfies latency and error-based objectives (SLOs), and promote the candidate.
This tutorial illustrates the Helm-based experiment pattern.
Setup K8s cluster with Knative and local environment
- Get Helm 3.4+
- Setup K8s cluster.
- Install Knative in K8s cluster. This tutorial can be tried with any Knative networking layer.
- Install Iter8 in K8s cluster
- Get
iter8ctl
- Get the Iter8 Helm repo
1. Create baseline version¶
Deploy the baseline version of the hello world
application using Helm.
helm install my-app iter8/knslo \
--set baseline.dynamic.tag="1.0" \
--set baseline.dynamic.id="v1" \
--set candidate=null
View K8s resources created by Helm
Use the command below to view the K8s resources created by Helm during the release of my-app
.
helm get manifest my-app
Verify that baseline version is 1.0.0
Ensure that the Knative app is ready.
kubectl wait ksvc/hello --for=condition=Ready
Port-forward the ingress service for Knative. With the Kourier
networking layer, you can do this as follows.
# do this in a separate terminal
kubectl port-forward svc/kourier -n knative-serving 8080:80
curl localhost:8080 -H "Host: hello.default.example.com"
# output will be similar to the following (notice 1.0.0 version tag)
# hostname will be different in your environment
Hello, world!
Version: 1.0.0
Hostname: hello-bc95d9b56-xp9kv
2. Create candidate version¶
Deploy the candidate version of the hello world
application using Helm.
helm upgrade my-app iter8/knslo \
--set baseline.dynamic.tag="1.0" \
--set baseline.dynamic.id="v1" \
--set candidate.dynamic.tag="2.0" \
--set candidate.dynamic.id="v2" \
--install
The above command creates an Iter8 experiment alongside the candidate version of the hello world
application. The experiment will collect latency and error rate metrics for the candidate, and verify that it satisfies the mean latency (50 msec), error rate (0.0), 95th percentile tail latency (100 msec) SLOs.
Verify that candidate version is 2.0.0
Ensure that the Knative app is ready.
kubectl wait ksvc/hello --for=condition=Ready
# this command reuses the port-forward from the first step
curl localhost:8080 -H "Host: candidate-hello.default.example.com"
# output will be similar to the following (notice 2.0.0 version tag)
# hostname will be different in your environment
Hello, world!
Version: 2.0.0
Hostname: hello-bc95d9b56-xp9kv
3. Observe experiment¶
Describe the results of the Iter8 experiment. Wait ~1 min before trying the following command. If the output is not as expected, try again after a few seconds.
iter8ctl describe
Experiment results will look similar to this ...
****** Overview ******
Experiment name: my-experiment
Experiment namespace: default
Target: my-app
Testing pattern: Conformance
Deployment pattern: Progressive
****** Progress Summary ******
Experiment stage: Completed
Number of completed iterations: 1
****** Winner Assessment ******
> If the version being validated; i.e., the baseline version, satisfies the experiment objectives, it is the winner.
> Otherwise, there is no winner.
Winning version: my-app
****** Objective Assessment ******
> Identifies whether or not the experiment objectives are satisfied by the most recently observed metrics values for each version.
+--------------------------------------+--------+
| OBJECTIVE | MY-APP |
+--------------------------------------+--------+
| iter8-system/mean-latency <= | true |
| 50.000 | |
+--------------------------------------+--------+
| iter8-system/error-rate <= | true |
| 0.000 | |
+--------------------------------------+--------+
| iter8-system/latency-95th-percentile | true |
| <= 100.000 | |
+--------------------------------------+--------+
****** Metrics Assessment ******
> Most recently read values of experiment metrics for each version.
+--------------------------------------+--------+
| METRIC | MY-APP |
+--------------------------------------+--------+
| iter8-system/mean-latency | 1.233 |
+--------------------------------------+--------+
| iter8-system/error-rate | 0.000 |
+--------------------------------------+--------+
| iter8-system/latency-95th-percentile | 2.311 |
+--------------------------------------+--------+
| iter8-system/request-count | 40.000 |
+--------------------------------------+--------+
| iter8-system/error-count | 0.000 |
+--------------------------------------+--------+
4. Promote winner¶
Assert that the experiment completed and found a winning version. If the conditions are not satisfied, try again after a few seconds.
iter8ctl assert -c completed -c winnerFound
Promote the winner as follows.
helm upgrade my-app iter8/knslo \
--set baseline.dynamic.tag="2.0" \
--set baseline.dynamic.id="v2" \
--set candidate=null \
--install
Verify that baseline version is 2.0.0
Ensure that the Knative app is ready.
kubectl wait ksvc/hello --for=condition=Ready
curl localhost:8080 -H "Host: hello.default.example.com"
# output will be similar to the following (notice 2.0.0 version tag)
# hostname will be different in your environment
Hello, world!
Version: 2.0.0
Hostname: hello-bc95d9b56-xp9kv
5. Cleanup¶
helm uninstall my-app
Next Steps
Use in production
The knslo
Helm chart is located in the $ITER8/helm
folder. Modify the chart as needed by your application for production usage.
Try other Iter8 Knative tutorials