Learn how to use GitOps to deploy and synchronize a Consul cluster on Kubernetes with Argo CD.
A GitOps tool like Argo CD can help centralize the automation, installation, and configuration of services onto multiple Kubernetes clusters. Rather than apply changes using a Kubernetes CLI or CI/CD, a GitOps workflow detects changes in version control and applies the changes automatically in the cluster. You can use a GitOps workflow to deploy and manage changes to a Consul cluster, while orchestrating the configuration of Consul service mesh for peering, network policy, and gateways.
This approach to managing your Consul cluster and configuration has two benefits. First, a GitOps tool handles the order-of-operations and automation of cluster updates before configuration updates. Second, your Consul configuration uses version control as a source of truth that GitOps enforces across multiple Kubernetes clusters.
This post demonstrates a GitOps workflow for deploying a Consul cluster, configuring its service mesh, and upgrading its server with Argo CD. Argo CD annotations for sync waves and resource hooks enable orchestration of Consul cluster deployment followed by service mesh configuration with Custom Resource Definitions (CRDs). Updating a Consul cluster on Kubernetes involves opening a pull request with changes to Helm chart values or CRDs and merging it. Argo CD synchronizes the configuration to match version control and handles the order of operations when applying the changes.
The example in this post deploys a Consul server and configures defaults for peering and service mesh using mesh and proxy-defaults configuration entries. You can use Argo CD sync waves to additionally deploy applications that rely on Consul, such as Jaeger or other observability tools, with the App of Apps pattern.
Argo CD supports declarative setup of many resources and the installation of Helm charts. Create a repository for Consul Helm chart values. Consul has a number of configuration options and components when running on Kubernetes. Placing these under version control ensures that you pin Consul versions and define its components in a controlled manner.
In the repository, define values for the Consul Helm chart. As of Helm chart version 1.2.2, the values include a global.argocd.enabled
attribute to configure Consul for Argo CD. The values use Consul version 1.16.0 as part of the initial deployment:
global: name: consul datacenter: dc1 image: hashicorp/consul:1.16.0 # omitted for clarity argocd: enabled: true # omitted for clarity server: replicas: 3 updatePartition: 0
Make note of the repository storing the Consul Helm chart values, as you will use it to define a Consul application for Argo CD.
ArgoCD also supports deployment of Kubernetes application manifests. You typically deploy global configuration entries after the Consul server starts successfully. Using Argo CD, you automate the workflow of starting the Consul cluster and configuring defaults for the service mesh or gateways. For example, you can define the default proxy configuration using the proxy-defaults configuration entry
. Using an Argo CD sync wave, you ensure that Argo CD applies the proxy-defaults
configuration entry after the Consul server starts.
To ensure Argo CD follows the order of operations, annotate CRDs for Consul configuration entries with a sync wave greater than 2. The Consul Helm chart includes annotations for helm.sh/hook-weight
, which Argo CD maps to sync waves. A sync wave greater than 2 for Consul configuration entries ensures that Argo CD fully updates the Consul cluster before applying configuration entries:
apiVersion: consul.hashicorp.com/v1alpha1kind: ProxyDefaultsmetadata: name: global annotations: argocd.argoproj.io/sync-wave: "3"spec: meshGateway: mode: 'local' config: # omitted for clarity
Store these CRDs in a repository, either the same one as the Helm chart values or a different one dedicated to Consul configuration. By defining a sync wave, you can orchestrate the deployment of different Consul configuration entries after their components start.
Consul requires several cluster-wide permissions. To isolate its management from other workloads, define an Argo CD project dedicated to Consul and its global configuration entries. Creating a project limits Consul’s access to Kubernetes and limits the source repositories to the Consul Helm chart, Helm chart values, and configuration entries.
Define a project named consul
in the consul-project.yaml
file:
consul-project.yaml
apiVersion: argoproj.io/v1alpha1kind: AppProjectmetadata: name: consul namespace: argocd finalizers: - resources-finalizer.argocd.argoproj.iospec: description: HashiCorp Consul sourceRepos: - 'https://helm.releases.hashicorp.com' - 'https://github.com/joatmon08/consul-minikubes.git' # Only permit applications to deploy to the consul namespace in the same cluster destinations: - namespace: consul server: https://kubernetes.default.svc name: in-cluster clusterResourceWhitelist: - group: '' kind: Namespace - group: 'rbac.authorization.k8s.io' kind: ClusterRole - group: 'rbac.authorization.k8s.io' kind: ClusterRoleBinding - group: 'apiextensions.k8s.io' kind: CustomResourceDefinition - group: 'admissionregistration.k8s.io' kind: MutatingWebhookConfiguration # Allow all namespaced-scoped resources to be created, except for ResourceQuota, LimitRange, NetworkPolicy namespaceResourceBlacklist: - group: '' kind: ResourceQuota - group: '' kind: LimitRange - group: '' kind: NetworkPolicy # Enables namespace orphaned resource monitoring. orphanedResources: warn: false roles: # A role which provides read-only access to all applications in the project - name: read-only description: Read-only privileges to consul policies: - p, proj:consul:read-only, applications, get, consul/*, allow groups: - consul
Apply the project to your Kubernetes cluster:
$ kubectl apply -f consul-project.yaml
Define an Argo CD application for Consul in the consul-application.yaml
file. The application manifest has three tasks. First, Argo CD creates a namespace for Consul. Then, it synchronizes changes from the Helm chart and its input values using syncPolicy.automated
. It also deploys proxy-defaults
and mesh
configuration entries to the Consul cluster. Note that the Argo CD application for Consul uses multiple sources.
apiVersion: argoproj.io/v1alpha1kind: Applicationmetadata: name: consul namespace: argocdspec: # Automatically synchronize changes from sources, create Consul namespace syncPolicy: automated: prune: true selfHeal: true syncOptions: - CreateNamespace=true project: consul sources: # Deploy Consul Helm chart - chart: consul repoURL: https://helm.releases.hashicorp.com targetRevision: 1.2.2 helm: releaseName: consul valueFiles: - $values/argocd/consul-helm-values.yaml # Use values from demo repository for Helm chart - repoURL: 'https://github.com/joatmon08/consul-minikubes.git' targetRevision: argocd ref: values # Use Consul CRDs to configure proxy-defaults and mesh - repoURL: 'https://github.com/joatmon08/consul-minikubes.git' path: argocd/consul-config targetRevision: argocd # Deploy to the consul namespace in same cluster destination: server: "https://kubernetes.default.svc" namespace: consul
This example initially uses Consul version to 1.16.0 and chart version to 1.2.2 to ensure that Argo CD enforces each version and their compatibility. To upgrade Consul, you can update the version in version control and Argo CD will roll out a new version of Consul or the chart.
Apply the Consul application to your Kubernetes cluster:
$ kubectl apply -f consul-application.yaml
Argo CD deploys the Consul server and other components defined by the Helm chart before applying the mesh
and proxy-defaults
configuration entries.
To examine these resources, log into Argo CD. Using the Argo CD CLI, get information about the Consul application:
$ argocd app get consul Name: argocd/consulProject: consulServer: https://kubernetes.default.svcNamespace: consulURL: https://localhost:8080/applications/consulRepo: https://helm.releases.hashicorp.comTarget: 1.2.2Path: Helm Values: $values/argocd/consul-helm-values.yamlSyncWindow: Sync AllowedSync Policy: Automated (Prune)Sync Status: Synced to 1.2.2Health Status: Healthy GROUP KIND NAMESPACE NAME STATUS HEALTH HOOK MESSAGE # omitted for clarity Service consul consul-connect-injector Synced Healthy Service consul consul-dns Synced Healthy Service consul consul-mesh-gateway Synced Healthy Service consul consul-server Synced Healthy # omitted for clarity apps StatefulSet consul consul-server Synced Healthy consul.hashicorp.com Mesh consul mesh Synced consul.hashicorp.com ProxyDefaults consul global Synced # omitted for clarity
If you log into Consul and check the proxy-defaults
configuration entry, you will find that Argo CD applied the ProxyDefaults
CRD:
$ consul config read -kind proxy-defaults -name global { "Kind": "proxy-defaults", "Name": "global", "TransparentProxy": {}, "Config": { # omitted for clarity }, "MeshGateway": { "Mode": "local" }, "Expose": {}, "AccessLogs": {}, "Meta": { "consul.hashicorp.com/source-datacenter": "dc1", "external-source": "kubernetes" }}
Use version control to commit any changes to the Consul Helm chart, its values, or Consul CRDs. Argo CD automatically synchronizes and applies the changes to these resources.
As you run your Consul cluster, you may choose to upgrade your servers or Helm chart to the next version. Before you commit the new versions to version control and Argo CD synchronizes configuration, make sure that you back up the persistent volume attached to the Consul servers. Before upgrading, review the documentation for upgrading Consul on Kubernetes and verify upgrade instructions for each version of Consul.
Imagine you need to upgrade your Consul cluster from 1.16.0 to 1.16.2 to roll out a fix to snapshots. In the Helm chart values, update Consul’s image to 1.16.2 and set server.updatePartition
to equal the number of server replicas. This ensures that upgrades do not occur immediately and you can control the rollout:
global: name: consul datacenter: dc1 image: hashicorp/consul:1.16.2 # update Consul version # omitted for clarity server: replicas: 3 updatePartition: 3 # number of servers to avoid updating
Commit this to version control and wait for Argo CD to synchronize. Argo CD notices the differences in version control and the live configuration and identifies which components to update.
Argo CD will upgrade Consul components such as the injector and gateways first and avoid upgrading the servers because of the server.updatePartition
attribute passed to the Consul Helm chart. Verify that services can still access each other and Consul remains healthy.
Set server.updatePartition
to 2 and commit this configuration to version control:
global: name: consul datacenter: dc1 image: hashicorp/consul:1.16.2 # update Consul version # omitted for clarity server: replicas: 3 updatePartition: 2 # number of servers to avoid updating
Argo CD passes this value to the Consul Helm chart and upgrades a single server to 1.16.2.
Verify one server has upgraded to 1.16.2:
$ kubectl get pods -n consul \ -l component=server \ -o jsonpath='{range .items[*]}{"\n"}{.metadata.name}{":\t"}{range .spec.containers[*]}{.image}{", "}{end}{end}' consul-server-0: hashicorp/consul:1.16.0, consul-server-1: hashicorp/consul:1.16.0, consul-server-2: hashicorp/consul:1.16.2,
Continue decrementing server.updatePartition
and committing the new value to version control until the value reaches zero. This ensures a gradual rollout of the new Consul server version. While decrementing, verify that services can still access each other and Consul remains healthy:
$ kubectl get pods -n consul \ -l component=server \ -o jsonpath='{range .items[*]}{"\n"}{.metadata.name}{":\t"}{range .spec.containers[*]}{.image}{", "}{end}{end}' consul-server-0: hashicorp/consul:1.16.2, consul-server-1: hashicorp/consul:1.16.2, consul-server-2: hashicorp/consul:1.16.2,
When you upgrade the Helm chart, you can verify changes on the Argo CD server. You need to disable auto-sync for the Consul application. Commit and push the Helm chart version update to version control. Argo CD recognizes the configuration as out of sync. If you examine the application details, you can verify the changes to Consul resources.
Alternatively, you can define Consul as a single-source Argo CD application and run a local command for Argo CD to synchronize a dry run and identify the changes.
Once you verify the changes, update the Consul application for Argo CD with a target revision for the Helm release:
apiVersion: argoproj.io/v1alpha1kind: Applicationmetadata: name: consul namespace: argocdspec: # omitted for clarity sources: # Deploy Consul Helm chart - chart: consul repoURL: https://helm.releases.hashicorp.com targetRevision: 1.x.x # Update to latest Helm chart helm: releaseName: consul valueFiles: - $values/argocd/consul-helm-values.yaml # omitted for clarity
To preserve the state of the Consul servers, set server.updatePartition
to 3. Commit the Helm values to version control. This ensures that Argo CD synchronizes changes to other components before the servers:
global: name: consul datacenter: dc1 image: hashicorp/consul:1.16.2 # omitted for clarity server: replicas: 3 updatePartition: 3 # number of servers to avoid updating
Then, apply the Consul Argo CD application to your Kubernetes cluster:
$ kubectl apply -f consul-application.yaml
Decrement server.updatePartition
and commit the Helm values to version control until it reaches zero. This applies gradual updates from the Helm chart to the Consul servers. Verify that services can still access each other and Consul remains healthy.
A GitOps workflow allows you to deploy and manage changes to a Consul cluster while orchestrating the configuration of a Consul service mesh for peering, network policy, and gateways. By ensuring that version control remains the primary source of truth for Consul cluster and service mesh configuration, you can consistently deploy configuration across different Consul clusters, clouds, and environments and standardize management of Consul across multiple Kubernetes clusters. Automation with Argo CD hooks and sync waves help synchronize Consul resources on Kubernetes, coordinate changes to different resources, and facilitate the upgrade of Consul servers and components.
For a detailed set of configurations, check out my demo repository. For more information about Argo CD, explore its documentation. To learn more about Consul on Kubernetes, check out our documentation and this tutorial on how to install Consul via Helm chart. Additional instructions can be found in our documentation on Upgrading Consul on Kubernetes components.
(Thank you to Christian Hernandez, Head of Community Engineering at Akuity, for educating and clarifying Argo CD details for this post.)
Do cloud right with The Infrastructure Cloud from HashiCorp. Unlock developer potential while controlling cloud costs and risk.
A recap of HashiCorp infrastructure and security news and developments from Google Cloud Next, from scaling infrastructure as code to fighting secrets sprawl and more.
Try this example method for transitioning from Consul service discovery to service mesh without affecting uptimes or development teams.