Learn how to use GitOps to deploy and synchronize a Consul cluster on Kubernetes with Argo CD.
A GitOps tool like Argo CD can help centralize the automation, installation, and configuration of services onto multiple Kubernetes clusters. Rather than apply changes using a Kubernetes CLI or CI/CD, a GitOps workflow detects changes in version control and applies the changes automatically in the cluster. You can use a GitOps workflow to deploy and manage changes to a Consul cluster, while orchestrating the configuration of Consul service mesh for peering, network policy, and gateways.
This approach to managing your Consul cluster and configuration has two benefits. First, a GitOps tool handles the order-of-operations and automation of cluster updates before configuration updates. Second, your Consul configuration uses version control as a source of truth that GitOps enforces across multiple Kubernetes clusters.
This post demonstrates a GitOps workflow for deploying a Consul cluster, configuring its service mesh, and upgrading its server with Argo CD. Argo CD annotations for sync waves and resource hooks enable orchestration of Consul cluster deployment followed by service mesh configuration with Custom Resource Definitions (CRDs). Updating a Consul cluster on Kubernetes involves opening a pull request with changes to Helm chart values or CRDs and merging it. Argo CD synchronizes the configuration to match version control and handles the order of operations when applying the changes.
Argo CD UI of observability application deploying Consul and Jaeger
The example in this post deploys a Consul server and configures defaults for peering and service mesh using mesh and proxy-defaults configuration entries. You can use Argo CD sync waves to additionally deploy applications that rely on Consul, such as Jaeger or other observability tools, with the App of Apps pattern.
Argo CD supports declarative setup of many resources and the installation of Helm charts. Create a repository for Consul Helm chart values. Consul has a number of configuration options and components when running on Kubernetes. Placing these under version control ensures that you pin Consul versions and define its components in a controlled manner.
In the repository, define values for the Consul Helm chart. As of Helm chart version 1.2.2, the values include a global.argocd.enabled
attribute to configure Consul for Argo CD. The values use Consul version 1.16.0 as part of the initial deployment:
Make note of the repository storing the Consul Helm chart values, as you will use it to define a Consul application for Argo CD.
ArgoCD also supports deployment of Kubernetes application manifests. You typically deploy global configuration entries after the Consul server starts successfully. Using Argo CD, you automate the workflow of starting the Consul cluster and configuring defaults for the service mesh or gateways. For example, you can define the default proxy configuration using the proxy-defaults configuration entry
. Using an Argo CD sync wave, you ensure that Argo CD applies the proxy-defaults
configuration entry after the Consul server starts.
To ensure Argo CD follows the order of operations, annotate CRDs for Consul configuration entries with a sync wave greater than 2. The Consul Helm chart includes annotations for helm.sh/hook-weight
, which Argo CD maps to sync waves. A sync wave greater than 2 for Consul configuration entries ensures that Argo CD fully updates the Consul cluster before applying configuration entries:
Store these CRDs in a repository, either the same one as the Helm chart values or a different one dedicated to Consul configuration. By defining a sync wave, you can orchestrate the deployment of different Consul configuration entries after their components start.
Consul requires several cluster-wide permissions. To isolate its management from other workloads, define an Argo CD project dedicated to Consul and its global configuration entries. Creating a project limits Consul’s access to Kubernetes and limits the source repositories to the Consul Helm chart, Helm chart values, and configuration entries.
Define a project named consul
in the consul-project.yaml
file:
Apply the project to your Kubernetes cluster:
$ kubectl apply -f consul-project.yaml
Define an Argo CD application for Consul in the consul-application.yaml
file. The application manifest has three tasks. First, Argo CD creates a namespace for Consul. Then, it synchronizes changes from the Helm chart and its input values using syncPolicy.automated
. It also deploys proxy-defaults
and mesh
configuration entries to the Consul cluster. Note that the Argo CD application for Consul uses multiple sources.
This example initially uses Consul version to 1.16.0 and chart version to 1.2.2 to ensure that Argo CD enforces each version and their compatibility. To upgrade Consul, you can update the version in version control and Argo CD will roll out a new version of Consul or the chart.
Apply the Consul application to your Kubernetes cluster:
$ kubectl apply -f consul-application.yaml
Argo CD deploys the Consul server and other components defined by the Helm chart before applying the mesh
and proxy-defaults
configuration entries.
Argo CD UI of Consul application with mesh gateway, server, and configuration entries
To examine these resources, log into Argo CD. Using the Argo CD CLI, get information about the Consul application:
If you log into Consul and check the proxy-defaults
configuration entry, you will find that Argo CD applied the ProxyDefaults
CRD:
Use version control to commit any changes to the Consul Helm chart, its values, or Consul CRDs. Argo CD automatically synchronizes and applies the changes to these resources.
As you run your Consul cluster, you may choose to upgrade your servers or Helm chart to the next version. Before you commit the new versions to version control and Argo CD synchronizes configuration, make sure that you back up the persistent volume attached to the Consul servers. Before upgrading, review the documentation for upgrading Consul on Kubernetes and verify upgrade instructions for each version of Consul.
Imagine you need to upgrade your Consul cluster from 1.16.0 to 1.16.2 to roll out a fix to snapshots. In the Helm chart values, update Consul’s image to 1.16.2 and set server.updatePartition
to equal the number of server replicas. This ensures that upgrades do not occur immediately and you can control the rollout:
Commit this to version control and wait for Argo CD to synchronize. Argo CD notices the differences in version control and the live configuration and identifies which components to update.
Argo CD UI of showing difference of Consul server statefulset to version 1.16.2
Argo CD will upgrade Consul components such as the injector and gateways first and avoid upgrading the servers because of the server.updatePartition
attribute passed to the Consul Helm chart. Verify that services can still access each other and Consul remains healthy.
Set server.updatePartition
to 2 and commit this configuration to version control:
Argo CD passes this value to the Consul Helm chart and upgrades a single server to 1.16.2.
Argo CD UI upgrading one server to 1.16.2
Verify one server has upgraded to 1.16.2:
Continue decrementing server.updatePartition
and committing the new value to version control until the value reaches zero. This ensures a gradual rollout of the new Consul server version. While decrementing, verify that services can still access each other and Consul remains healthy:
When you upgrade the Helm chart, you can verify changes on the Argo CD server. You need to disable auto-sync for the Consul application. Commit and push the Helm chart version update to version control. Argo CD recognizes the configuration as out of sync. If you examine the application details, you can verify the changes to Consul resources.
Alternatively, you can define Consul as a single-source Argo CD application and run a local command for Argo CD to synchronize a dry run and identify the changes.
Once you verify the changes, update the Consul application for Argo CD with a target revision for the Helm release:
To preserve the state of the Consul servers, set server.updatePartition
to 3. Commit the Helm values to version control. This ensures that Argo CD synchronizes changes to other components before the servers:
Then, apply the Consul Argo CD application to your Kubernetes cluster:
$ kubectl apply -f consul-application.yaml
Decrement server.updatePartition
and commit the Helm values to version control until it reaches zero. This applies gradual updates from the Helm chart to the Consul servers. Verify that services can still access each other and Consul remains healthy.
A GitOps workflow allows you to deploy and manage changes to a Consul cluster while orchestrating the configuration of a Consul service mesh for peering, network policy, and gateways. By ensuring that version control remains the primary source of truth for Consul cluster and service mesh configuration, you can consistently deploy configuration across different Consul clusters, clouds, and environments and standardize management of Consul across multiple Kubernetes clusters. Automation with Argo CD hooks and sync waves help synchronize Consul resources on Kubernetes, coordinate changes to different resources, and facilitate the upgrade of Consul servers and components.
For a detailed set of configurations, check out my demo repository. For more information about Argo CD, explore its documentation. To learn more about Consul on Kubernetes, check out our documentation and this tutorial on how to install Consul via Helm chart. Additional instructions can be found in our documentation on Upgrading Consul on Kubernetes components.
(Thank you to Christian Hernandez, Head of Community Engineering at Akuity, for educating and clarifying Argo CD details for this post.)
HCP Consul Central’s new observability features were recently used by a customer to help troubleshoot an issue with assistance from HashiCorp engineers.
If you’re attending AWS re:Invent in Las Vegas, Nov. 27 - Dec. 1, visit us for breakout sessions, expert talks, and product demos to learn how to accelerate your adoption of a cloud operating model.
Consul 1.17 enhances unified workflow management, reliability, scale, and security for service networking.