Deploy Consul on Kubernetes with Argo CD
Learn how to use GitOps to deploy and synchronize a Consul cluster on Kubernetes with Argo CD.
A GitOps tool like Argo CD can help centralize the automation, installation, and configuration of services onto multiple Kubernetes clusters. Rather than apply changes using a Kubernetes CLI or CI/CD, a GitOps workflow detects changes in version control and applies the changes automatically in the cluster. You can use a GitOps workflow to deploy and manage changes to a Consul cluster, while orchestrating the configuration of Consul service mesh for peering, network policy, and gateways.
This approach to managing your Consul cluster and configuration has two benefits. First, a GitOps tool handles the order-of-operations and automation of cluster updates before configuration updates. Second, your Consul configuration uses version control as a source of truth that GitOps enforces across multiple Kubernetes clusters.
This post demonstrates a GitOps workflow for deploying a Consul cluster, configuring its service mesh, and upgrading its server with Argo CD. Argo CD annotations for sync waves and resource hooks enable orchestration of Consul cluster deployment followed by service mesh configuration with Custom Resource Definitions (CRDs). Updating a Consul cluster on Kubernetes involves opening a pull request with changes to Helm chart values or CRDs and merging it. Argo CD synchronizes the configuration to match version control and handles the order of operations when applying the changes.
Argo CD UI of observability application deploying Consul and Jaeger
The example in this post deploys a Consul server and configures defaults for peering and service mesh using mesh and proxy-defaults configuration entries. You can use Argo CD sync waves to additionally deploy applications that rely on Consul, such as Jaeger or other observability tools, with the App of Apps pattern.
»Define Consul Helm chart values
Argo CD supports declarative setup of many resources and the installation of Helm charts. Create a repository for Consul Helm chart values. Consul has a number of configuration options and components when running on Kubernetes. Placing these under version control ensures that you pin Consul versions and define its components in a controlled manner.
In the repository, define values for the Consul Helm chart. As of Helm chart version 1.2.2, the values include a global.argocd.enabled attribute to configure Consul for Argo CD. The values use Consul version 1.16.0 as part of the initial deployment:
global:
name: consul
datacenter: dc1
image: hashicorp/consul:1.16.0
# omitted for clarity
argocd:
enabled: true
# omitted for clarity
server:
replicas: 3
updatePartition: 0
Make note of the repository storing the Consul Helm chart values, as you will use it to define a Consul application for Argo CD.
»Define Consul configuration entries
ArgoCD also supports deployment of Kubernetes application manifests. You typically deploy global configuration entries after the Consul server starts successfully. Using Argo CD, you automate the workflow of starting the Consul cluster and configuring defaults for the service mesh or gateways. For example, you can define the default proxy configuration using the proxy-defaults configuration entry. Using an Argo CD sync wave, you ensure that Argo CD applies the proxy-defaults configuration entry after the Consul server starts.
To ensure Argo CD follows the order of operations, annotate CRDs for Consul configuration entries with a sync wave greater than 2. The Consul Helm chart includes annotations for helm.sh/hook-weight, which Argo CD maps to sync waves. A sync wave greater than 2 for Consul configuration entries ensures that Argo CD fully updates the Consul cluster before applying configuration entries:
apiVersion: consul.hashicorp.com/v1alpha1
kind: ProxyDefaults
metadata:
name: global
annotations:
argocd.argoproj.io/sync-wave: "3"
spec:
meshGateway:
mode: 'local'
config:
# omitted for clarity
Store these CRDs in a repository, either the same one as the Helm chart values or a different one dedicated to Consul configuration. By defining a sync wave, you can orchestrate the deployment of different Consul configuration entries after their components start.
»Create an Argo CD project for Consul
Consul requires several cluster-wide permissions. To isolate its management from other workloads, define an Argo CD project dedicated to Consul and its global configuration entries. Creating a project limits Consul’s access to Kubernetes and limits the source repositories to the Consul Helm chart, Helm chart values, and configuration entries.
Define a project named consul in the consul-project.yaml file:
consul-project.yaml
apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
name: consul
namespace: argocd
finalizers:
- resources-finalizer.argocd.argoproj.io
spec:
description: HashiCorp Consul
sourceRepos:
- 'https://helm.releases.hashicorp.com'
- 'https://github.com/joatmon08/consul-minikubes.git'
# Only permit applications to deploy to the consul namespace in the same cluster
destinations:
- namespace: consul
server: https://kubernetes.default.svc
name: in-cluster
clusterResourceWhitelist:
- group: ''
kind: Namespace
- group: 'rbac.authorization.k8s.io'
kind: ClusterRole
- group: 'rbac.authorization.k8s.io'
kind: ClusterRoleBinding
- group: 'apiextensions.k8s.io'
kind: CustomResourceDefinition
- group: 'admissionregistration.k8s.io'
kind: MutatingWebhookConfiguration
# Allow all namespaced-scoped resources to be created, except for ResourceQuota, LimitRange, NetworkPolicy
namespaceResourceBlacklist:
- group: ''
kind: ResourceQuota
- group: ''
kind: LimitRange
- group: ''
kind: NetworkPolicy
# Enables namespace orphaned resource monitoring.
orphanedResources:
warn: false
roles:
# A role which provides read-only access to all applications in the project
- name: read-only
description: Read-only privileges to consul
policies:
- p, proj:consul:read-only, applications, get, consul/*, allow
groups:
- consul
Apply the project to your Kubernetes cluster:
$ kubectl apply -f consul-project.yaml
»Deploy Consul with Argo CD
Define an Argo CD application for Consul in the consul-application.yaml file. The application manifest has three tasks. First, Argo CD creates a namespace for Consul. Then, it synchronizes changes from the Helm chart and its input values using syncPolicy.automated. It also deploys proxy-defaults and mesh configuration entries to the Consul cluster. Note that the Argo CD application for Consul uses multiple sources.
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: consul
namespace: argocd
spec:
# Automatically synchronize changes from sources, create Consul namespace
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- CreateNamespace=true
project: consul
sources:
# Deploy Consul Helm chart
- chart: consul
repoURL: https://helm.releases.hashicorp.com
targetRevision: 1.2.2
helm:
releaseName: consul
valueFiles:
- $values/argocd/consul-helm-values.yaml
# Use values from demo repository for Helm chart
- repoURL: 'https://github.com/joatmon08/consul-minikubes.git'
targetRevision: argocd
ref: values
# Use Consul CRDs to configure proxy-defaults and mesh
- repoURL: 'https://github.com/joatmon08/consul-minikubes.git'
path: argocd/consul-config
targetRevision: argocd
# Deploy to the consul namespace in same cluster
destination:
server: "https://kubernetes.default.svc"
namespace: consul
This example initially uses Consul version to 1.16.0 and chart version to 1.2.2 to ensure that Argo CD enforces each version and their compatibility. To upgrade Consul, you can update the version in version control and Argo CD will roll out a new version of Consul or the chart.
Apply the Consul application to your Kubernetes cluster:
$ kubectl apply -f consul-application.yaml
Argo CD deploys the Consul server and other components defined by the Helm chart before applying the mesh and proxy-defaults configuration entries.
Argo CD UI of Consul application with mesh gateway, server, and configuration entries
To examine these resources, log into Argo CD. Using the Argo CD CLI, get information about the Consul application:
$ argocd app get consul
Name: argocd/consul
Project: consul
Server: https://kubernetes.default.svc
Namespace: consul
URL: https://localhost:8080/applications/consul
Repo: https://helm.releases.hashicorp.com
Target: 1.2.2
Path:
Helm Values: $values/argocd/consul-helm-values.yaml
SyncWindow: Sync Allowed
Sync Policy: Automated (Prune)
Sync Status: Synced to 1.2.2
Health Status: Healthy
GROUP KIND NAMESPACE NAME STATUS HEALTH HOOK MESSAGE
# omitted for clarity
Service consul consul-connect-injector Synced Healthy
Service consul consul-dns Synced Healthy
Service consul consul-mesh-gateway Synced Healthy
Service consul consul-server Synced Healthy
# omitted for clarity
apps StatefulSet consul consul-server Synced Healthy
consul.hashicorp.com Mesh consul mesh Synced
consul.hashicorp.com ProxyDefaults consul global Synced
# omitted for clarity
If you log into Consul and check the proxy-defaults configuration entry, you will find that Argo CD applied the ProxyDefaults CRD:
$ consul config read -kind proxy-defaults -name global
{
"Kind": "proxy-defaults",
"Name": "global",
"TransparentProxy": {},
"Config": {
# omitted for clarity
},
"MeshGateway": {
"Mode": "local"
},
"Expose": {},
"AccessLogs": {},
"Meta": {
"consul.hashicorp.com/source-datacenter": "dc1",
"external-source": "kubernetes"
}
}
Use version control to commit any changes to the Consul Helm chart, its values, or Consul CRDs. Argo CD automatically synchronizes and applies the changes to these resources.
»Upgrade Consul with Argo CD
As you run your Consul cluster, you may choose to upgrade your servers or Helm chart to the next version. Before you commit the new versions to version control and Argo CD synchronizes configuration, make sure that you back up the persistent volume attached to the Consul servers. Before upgrading, review the documentation for upgrading Consul on Kubernetes and verify upgrade instructions for each version of Consul.
Imagine you need to upgrade your Consul cluster from 1.16.0 to 1.16.2 to roll out a fix to snapshots. In the Helm chart values, update Consul’s image to 1.16.2 and set server.updatePartition to equal the number of server replicas. This ensures that upgrades do not occur immediately and you can control the rollout:
global:
name: consul
datacenter: dc1
image: hashicorp/consul:1.16.2 # update Consul version
# omitted for clarity
server:
replicas: 3
updatePartition: 3 # number of servers to avoid updating
Commit this to version control and wait for Argo CD to synchronize. Argo CD notices the differences in version control and the live configuration and identifies which components to update.
Argo CD UI of showing difference of Consul server statefulset to version 1.16.2
Argo CD will upgrade Consul components such as the injector and gateways first and avoid upgrading the servers because of the server.updatePartition attribute passed to the Consul Helm chart. Verify that services can still access each other and Consul remains healthy.
Set server.updatePartition to 2 and commit this configuration to version control:
global:
name: consul
datacenter: dc1
image: hashicorp/consul:1.16.2 # update Consul version
# omitted for clarity
server:
replicas: 3
updatePartition: 2 # number of servers to avoid updating
Argo CD passes this value to the Consul Helm chart and upgrades a single server to 1.16.2.
Argo CD UI upgrading one server to 1.16.2
Verify one server has upgraded to 1.16.2:
$ kubectl get pods -n consul \
-l component=server \
-o jsonpath='{range .items[*]}{"\n"}{.metadata.name}{":\t"}{range .spec.containers[*]}{.image}{", "}{end}{end}'
consul-server-0: hashicorp/consul:1.16.0,
consul-server-1: hashicorp/consul:1.16.0,
consul-server-2: hashicorp/consul:1.16.2,
Continue decrementing server.updatePartition and committing the new value to version control until the value reaches zero. This ensures a gradual rollout of the new Consul server version. While decrementing, verify that services can still access each other and Consul remains healthy:
$ kubectl get pods -n consul \
-l component=server \
-o jsonpath='{range .items[*]}{"\n"}{.metadata.name}{":\t"}{range .spec.containers[*]}{.image}{", "}{end}{end}'
consul-server-0: hashicorp/consul:1.16.2,
consul-server-1: hashicorp/consul:1.16.2,
consul-server-2: hashicorp/consul:1.16.2,
When you upgrade the Helm chart, you can verify changes on the Argo CD server. You need to disable auto-sync for the Consul application. Commit and push the Helm chart version update to version control. Argo CD recognizes the configuration as out of sync. If you examine the application details, you can verify the changes to Consul resources.
Alternatively, you can define Consul as a single-source Argo CD application and run a local command for Argo CD to synchronize a dry run and identify the changes.
Once you verify the changes, update the Consul application for Argo CD with a target revision for the Helm release:
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: consul
namespace: argocd
spec:
# omitted for clarity
sources:
# Deploy Consul Helm chart
- chart: consul
repoURL: https://helm.releases.hashicorp.com
targetRevision: 1.x.x # Update to latest Helm chart
helm:
releaseName: consul
valueFiles:
- $values/argocd/consul-helm-values.yaml
# omitted for clarity
To preserve the state of the Consul servers, set server.updatePartition to 3. Commit the Helm values to version control. This ensures that Argo CD synchronizes changes to other components before the servers:
global:
name: consul
datacenter: dc1
image: hashicorp/consul:1.16.2
# omitted for clarity
server:
replicas: 3
updatePartition: 3 # number of servers to avoid updating
Then, apply the Consul Argo CD application to your Kubernetes cluster:
$ kubectl apply -f consul-application.yaml
Decrement server.updatePartition and commit the Helm values to version control until it reaches zero. This applies gradual updates from the Helm chart to the Consul servers. Verify that services can still access each other and Consul remains healthy.
»Next steps
A GitOps workflow allows you to deploy and manage changes to a Consul cluster while orchestrating the configuration of a Consul service mesh for peering, network policy, and gateways. By ensuring that version control remains the primary source of truth for Consul cluster and service mesh configuration, you can consistently deploy configuration across different Consul clusters, clouds, and environments and standardize management of Consul across multiple Kubernetes clusters. Automation with Argo CD hooks and sync waves help synchronize Consul resources on Kubernetes, coordinate changes to different resources, and facilitate the upgrade of Consul servers and components.
For a detailed set of configurations, check out my demo repository. For more information about Argo CD, explore its documentation. To learn more about Consul on Kubernetes, check out our documentation and this tutorial on how to install Consul via Helm chart. Additional instructions can be found in our documentation on Upgrading Consul on Kubernetes components.
(Thank you to Christian Hernandez, Head of Community Engineering at Akuity, for educating and clarifying Argo CD details for this post.)
Sign up for the latest HashiCorp news
More blog posts like this one
Consul 1.21 improves service discovery, Consul on Kubernetes, and supports OpenShift 4.17
HashiCorp Consul 1.21 simplifies deployment of Consul service discovery for external services, adds OpenShift 4.16 & 4.17 support, and extends snapshot support of Consul on Kubernetes to Google Cloud Storage.
HashiCorp at AWS re:Invent: Your blueprint to cloud success
If you’re attending AWS re:Invent in Las Vegas, Dec. 2 - Dec. 6th, visit us for breakout sessions, expert talks, and product demos to learn how to take a unified approach to Infrastructure and Security Lifecycle Management.
Consul 1.20 improves multi-tenancy, metrics, and OpenShift deployment
HashiCorp Consul 1.20 is a significant upgrade for the Kubernetes operator and developer experience, including better multi-tenant service discovery, catalog registration metrics, and secure OpenShift integration.