GitOps with Argo CD¶

This tutorial assumes a basic understanding of Iter8. See, for example, the Istio quick start tutorial.

git clone git@github.com:kalantar/iter8.git
cd iter8
export ITER8=$(pwd)

export YOUR_ORG=fill-in
find $ITER8/samples/istio/gitops -name "*" -type f | xargs sed -i '' "s/MY_ORG/$YOUR_ORG/"
git commit -a -m "update references"
git push origin head

export YOUR_ORG=fill-in
find $ITER8/samples/istio/gitops -name "*" -type f | xargs sed -i "s/MY_ORG/$YOUR_ORG/"
git commit -a -m "update references"
git push origin head

1. Create Baseline Version¶

Install the bookinfo application by creating an Argo CD application as follows:

kubectl apply -f $ITER8/samples/istio/gitops/argocd-app.yaml

The GitOps application defined by argocd-app.yaml identifies a part of your repository, https://github.com/[YOUR_ORG]/iter8, to be the environment repository. Argo CD will immediately begin to synchronize the configuration (bookinfo application config) found there. You can monitor the progress of the syncrhonization through the Argo CD UI. When the state is both Healthy and Synced, it is ready; this might take a few minutes.

2. Create Candidate Version¶

When changes are merged into a code repository, a CI pipeline to, for example, lint, build, test, and push newly built images to an image repository runs. It then writes configuration changes to the environment repository indicating changes are needed to the deployed application. In this tutorial, we simulate the execution of a CI pipeline, by executing a simplified GitHub workflow: https://github.com/[YOUR_ORG]/iter8/actions/workflows/gitops-ci.yaml

Navigate to the workflow and click the button "Run workflow"

You can use the Argo CD UI to monitor progress of the deployment. By default, Argo CD is configured to run every three minutes. If you don't want to wait, manually refresh the application so that the changes are immediately synced to the cluster.

3. Observe experiment¶

The experiment should run for a few minutes once it starts, and you can track its progress with this command:

kubectl get experiments.iter8.tools --watch

More detail is available using the Iter8 CLI:

iter8ctl describe

4. Promote winner¶

Once the experiment finishes, review the pull requests on the environment repository, https://github.com/[YOUR_ORG]/iter8/pulls. Iter8 will have created a new pull request titled Deploy version recommended by Iter8. Changes to productpage.yaml capture the recommended version changes. Other changes remove artifacts specific to the experiment including the candidate version, the experiment and any load generation.

Merge the pull request to complete the experiment. Argo CD will detect the changes and synchronize the cluster to the new desired state.

5. Cleanup¶

kubectl delete -f $ITER8/samples/istio/gitops/
kubectl delete ns istio-system
kubectl delete ns iter8-system
kubectl delete ns argocd

Additional details¶

Environment repository¶

It is generally considered good practice to use different Git repositories for code and for environment configuration. In cases where the same repository is being used for both, one needs to be careful when configuring CI/CD pipeline tools so code changes can be differentiated from configuration changes, so that one doesn't inadvertently create infinite loops.

The environment repository can be organized in many different ways. With tools such as Helm and Kustomize becoming widely used, it becomes even simpler for CI pipeline tools to update an environment repository to roll out new app versions. In this tutorial, we consciously decided to use the simplest directory structure (i.e., all YAML files within a single directory without subdirectories) without the use of any higher level templating tools. Adapting the basic directory structure to Helm/Kustomize is straight forward.

When organizing the directory structure, one needs to keep in mind that the CI pipeline tool will be creating new resources in the environment repository to start an Iter8 experiment. And when the experiment finishes, Iter8 (specifically, Iter8 tasks) will delete the added resources and update the baseline version in the environment repository. In other words, the invariant here is the directory structure, which should stay the same before and after an experiment.

GitOps support for multiple environments¶

Some users might use GitOps to manage multiple environments, e.g., dev, staging, prod, so changes can always propagate from environment to environment, minimizing the chance of defects from reaching the prod environment. In this case, the Iter8 GitOps task would need to be modified so that environment repository changes are done at the correct places. For example, if different environments are managed by different environment repositories, the task would need to make multiple git commits, one for each of the respositories. This could be done all within a single task, or across multiple tasks.

GitOps Guarantees¶

Unlike other progressive delivery tools, Iter8 adheres to GitOps' guarantees by ensuring the actual state is always in sync with the desired state. App versions that fail promotion criteria will never get promoted, even if the cluster has to be recreated from scratch. This important GitOps property is often not guaranteed by other tools!

Caveats¶

1. Both CI pipeline tools and Iter8 need to write to the environment repository. If not coordinated, race conditions can occur. In this tutorial, we assume repo changes are done via pull requests, which is a common practice, so the chance of having a race condition is minimized, if not eliminated. However, other means to coordinate writes to the environment repository by different entities can be done so Iter8 can operate in fully automated pipelines.

2. When a new app version becomes available while an experiment is still running, Iter8 will preempt the existing experiment with the new one. We currently don't support test-every-commit behavior by queuing new experiments, but this could be supported in the future if it turrned out to be more common than we are currently expecting.

3. Iter8 task could fail, just like everything else. Iter8 tasks are currently fail-stop without retries. Please take this into account when writing Iter8 tasks and error handling code.