Products & Technology

Deploy Consul on Kubernetes with Argo CD

Learn how to use GitOps to deploy and synchronize a Consul cluster on Kubernetes with Argo CD.

Sep 25 2023Rosemary Wang

A GitOps tool like Argo CD can help centralize the automation, installation, and configuration of services onto multiple Kubernetes clusters. Rather than apply changes using a Kubernetes CLI or CI/CD, a GitOps workflow detects changes in version control and applies the changes automatically in the cluster. You can use a GitOps workflow to deploy and manage changes to a Consul cluster, while orchestrating the configuration of Consul service mesh for peering, network policy, and gateways.

This approach to managing your Consul cluster and configuration has two benefits. First, a GitOps tool handles the order-of-operations and automation of cluster updates before configuration updates. Second, your Consul configuration uses version control as a source of truth that GitOps enforces across multiple Kubernetes clusters.

This post demonstrates a GitOps workflow for deploying a Consul cluster, configuring its service mesh, and upgrading its server with Argo CD. Argo CD annotations for sync waves and resource hooks enable orchestration of Consul cluster deployment followed by service mesh configuration with Custom Resource Definitions (CRDs). Updating a Consul cluster on Kubernetes involves opening a pull request with changes to Helm chart values or CRDs and merging it. Argo CD synchronizes the configuration to match version control and handles the order of operations when applying the changes.

Argo CD UI of observability application deploying Consul and Jaeger

The example in this post deploys a Consul server and configures defaults for peering and service mesh using mesh and proxy-defaults configuration entries. You can use Argo CD sync waves to additionally deploy applications that rely on Consul, such as Jaeger or other observability tools, with the App of Apps pattern.

»Define Consul Helm chart values

Argo CD supports declarative setup of many resources and the installation of Helm charts. Create a repository for Consul Helm chart values. Consul has a number of configuration options and components when running on Kubernetes. Placing these under version control ensures that you pin Consul versions and define its components in a controlled manner.

In the repository, define values for the Consul Helm chart. As of Helm chart version 1.2.2, the values include a global.argocd.enabled attribute to configure Consul for Argo CD. The values use Consul version 1.16.0 as part of the initial deployment:

global:
  name: consul
  datacenter: dc1
  image: hashicorp/consul:1.16.0
 
  # omitted for clarity
 
  argocd:
    enabled: true
 
# omitted for clarity
 
server:
  replicas: 3
  updatePartition: 0

Make note of the repository storing the Consul Helm chart values, as you will use it to define a Consul application for Argo CD.

»Define Consul configuration entries

ArgoCD also supports deployment of Kubernetes application manifests. You typically deploy global configuration entries after the Consul server starts successfully. Using Argo CD, you automate the workflow of starting the Consul cluster and configuring defaults for the service mesh or gateways. For example, you can define the default proxy configuration using the proxy-defaults configuration entry. Using an Argo CD sync wave, you ensure that Argo CD applies the proxy-defaults configuration entry after the Consul server starts.

To ensure Argo CD follows the order of operations, annotate CRDs for Consul configuration entries with a sync wave greater than 2. The Consul Helm chart includes annotations for helm.sh/hook-weight, which Argo CD maps to sync waves. A sync wave greater than 2 for Consul configuration entries ensures that Argo CD fully updates the Consul cluster before applying configuration entries:

apiVersion: consul.hashicorp.com/v1alpha1
kind: ProxyDefaults
metadata:
  name: global
  annotations:
    argocd.argoproj.io/sync-wave: "3"
spec:
  meshGateway:
    mode: 'local'
  config:
    # omitted for clarity

Store these CRDs in a repository, either the same one as the Helm chart values or a different one dedicated to Consul configuration. By defining a sync wave, you can orchestrate the deployment of different Consul configuration entries after their components start.

»Create an Argo CD project for Consul

Consul requires several cluster-wide permissions. To isolate its management from other workloads, define an Argo CD project dedicated to Consul and its global configuration entries. Creating a project limits Consul’s access to Kubernetes and limits the source repositories to the Consul Helm chart, Helm chart values, and configuration entries.

Define a project named consul in the consul-project.yaml file:

consul-project.yaml

apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: consul
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  description: HashiCorp Consul
  sourceRepos:
  - 'https://helm.releases.hashicorp.com'
  - 'https://github.com/joatmon08/consul-minikubes.git'
 
  # Only permit applications to deploy to the consul namespace in the same cluster
  destinations:
  - namespace: consul
    server: https://kubernetes.default.svc
    name: in-cluster
 
  clusterResourceWhitelist:
  - group: ''
    kind: Namespace
  - group: 'rbac.authorization.k8s.io'
    kind: ClusterRole
  - group: 'rbac.authorization.k8s.io'
    kind: ClusterRoleBinding
  - group: 'apiextensions.k8s.io'
    kind: CustomResourceDefinition
  - group: 'admissionregistration.k8s.io'
    kind: MutatingWebhookConfiguration
 
  # Allow all namespaced-scoped resources to be created, except for ResourceQuota, LimitRange, NetworkPolicy
  namespaceResourceBlacklist:
  - group: ''
    kind: ResourceQuota
  - group: ''
    kind: LimitRange
  - group: ''
    kind: NetworkPolicy
 
  # Enables namespace orphaned resource monitoring.
  orphanedResources:
    warn: false
 
  roles:
  # A role which provides read-only access to all applications in the project
  - name: read-only
    description: Read-only privileges to consul
    policies:
    - p, proj:consul:read-only, applications, get, consul/*, allow
    groups:
    - consul

Apply the project to your Kubernetes cluster:

$ kubectl apply -f consul-project.yaml

»Deploy Consul with Argo CD

Define an Argo CD application for Consul in the consul-application.yaml file. The application manifest has three tasks. First, Argo CD creates a namespace for Consul. Then, it synchronizes changes from the Helm chart and its input values using syncPolicy.automated. It also deploys proxy-defaults and mesh configuration entries to the Consul cluster. Note that the Argo CD application for Consul uses multiple sources.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: consul
  namespace: argocd
spec:
  # Automatically synchronize changes from sources, create Consul namespace
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
    - CreateNamespace=true
 
  project: consul
 
  sources:
    # Deploy Consul Helm chart
    - chart: consul
      repoURL: https://helm.releases.hashicorp.com
      targetRevision: 1.2.2
      helm:
        releaseName: consul
        valueFiles:
          - $values/argocd/consul-helm-values.yaml
 
    # Use values from demo repository for Helm chart
    - repoURL: 'https://github.com/joatmon08/consul-minikubes.git'
      targetRevision: argocd
      ref: values
 
    # Use Consul CRDs to configure proxy-defaults and mesh
    - repoURL: 'https://github.com/joatmon08/consul-minikubes.git'
      path: argocd/consul-config
      targetRevision: argocd
 
  # Deploy to the consul namespace in same cluster
  destination:
    server: "https://kubernetes.default.svc"
    namespace: consul

This example initially uses Consul version to 1.16.0 and chart version to 1.2.2 to ensure that Argo CD enforces each version and their compatibility. To upgrade Consul, you can update the version in version control and Argo CD will roll out a new version of Consul or the chart.

Apply the Consul application to your Kubernetes cluster:

$ kubectl apply -f consul-application.yaml

Argo CD deploys the Consul server and other components defined by the Helm chart before applying the mesh and proxy-defaults configuration entries.

Argo CD UI of Consul application with mesh gateway, server, and configuration entries

To examine these resources, log into Argo CD. Using the Argo CD CLI, get information about the Consul application:

$ argocd app get consul
 
Name:               argocd/consul
Project:            consul
Server:             https://kubernetes.default.svc
Namespace:          consul
URL:                https://localhost:8080/applications/consul
Repo:               https://helm.releases.hashicorp.com
Target:             1.2.2
Path:               
Helm Values:        $values/argocd/consul-helm-values.yaml
SyncWindow:         Sync Allowed
Sync Policy:        Automated (Prune)
Sync Status:        Synced to 1.2.2
Health Status:      Healthy
 
GROUP                         KIND                          NAMESPACE  NAME                                      STATUS  HEALTH   HOOK  MESSAGE
 
# omitted for clarity
 
                              Service                       consul     consul-connect-injector                   Synced  Healthy        
                              Service                       consul     consul-dns                                Synced  Healthy        
                              Service                       consul     consul-mesh-gateway                       Synced  Healthy        
                              Service                       consul     consul-server                             Synced  Healthy
 
# omitted for clarity
 
apps                          StatefulSet                   consul     consul-server                             Synced  Healthy        
consul.hashicorp.com          Mesh                          consul     mesh                                      Synced                 
consul.hashicorp.com          ProxyDefaults                 consul     global                                    Synced            
 
# omitted for clarity

If you log into Consul and check the proxy-defaults configuration entry, you will find that Argo CD applied the ProxyDefaults CRD:

$ consul config read -kind proxy-defaults -name global
 
{
  "Kind": "proxy-defaults",
  "Name": "global",
  "TransparentProxy": {},
  "Config": {
    # omitted for clarity
  },
  "MeshGateway": {
    "Mode": "local"
  },
  "Expose": {},
  "AccessLogs": {},
  "Meta": {
    "consul.hashicorp.com/source-datacenter": "dc1",
    "external-source": "kubernetes"
  }
}

Use version control to commit any changes to the Consul Helm chart, its values, or Consul CRDs. Argo CD automatically synchronizes and applies the changes to these resources.

»Upgrade Consul with Argo CD

As you run your Consul cluster, you may choose to upgrade your servers or Helm chart to the next version. Before you commit the new versions to version control and Argo CD synchronizes configuration, make sure that you back up the persistent volume attached to the Consul servers. Before upgrading, review the documentation for upgrading Consul on Kubernetes and verify upgrade instructions for each version of Consul.

Imagine you need to upgrade your Consul cluster from 1.16.0 to 1.16.2 to roll out a fix to snapshots. In the Helm chart values, update Consul’s image to 1.16.2 and set server.updatePartition to equal the number of server replicas. This ensures that upgrades do not occur immediately and you can control the rollout:

global:
  name: consul
  datacenter: dc1
  image: hashicorp/consul:1.16.2 # update Consul version
 
  # omitted for clarity
 
server:
  replicas: 3
  updatePartition: 3 # number of servers to avoid updating

Commit this to version control and wait for Argo CD to synchronize. Argo CD notices the differences in version control and the live configuration and identifies which components to update.

Argo CD UI of showing difference of Consul server statefulset to version 1.16.2

Argo CD will upgrade Consul components such as the injector and gateways first and avoid upgrading the servers because of the server.updatePartition attribute passed to the Consul Helm chart. Verify that services can still access each other and Consul remains healthy.

Set server.updatePartition to 2 and commit this configuration to version control:

global:
  name: consul
  datacenter: dc1
  image: hashicorp/consul:1.16.2 # update Consul version
 
  # omitted for clarity
 
server:
  replicas: 3
  updatePartition: 2 # number of servers to avoid updating

Argo CD passes this value to the Consul Helm chart and upgrades a single server to 1.16.2.

Argo CD UI upgrading one server to 1.16.2

Verify one server has upgraded to 1.16.2:

$ kubectl get pods -n consul \
  -l component=server \
  -o jsonpath='{range .items[*]}{"\n"}{.metadata.name}{":\t"}{range .spec.containers[*]}{.image}{", "}{end}{end}'
 
consul-server-0:        hashicorp/consul:1.16.0, 
consul-server-1:        hashicorp/consul:1.16.0, 
consul-server-2:        hashicorp/consul:1.16.2,

Continue decrementing server.updatePartition and committing the new value to version control until the value reaches zero. This ensures a gradual rollout of the new Consul server version. While decrementing, verify that services can still access each other and Consul remains healthy:

$ kubectl get pods -n consul \                        
  -l component=server \
  -o jsonpath='{range .items[*]}{"\n"}{.metadata.name}{":\t"}{range .spec.containers[*]}{.image}{", "}{end}{end}'
 
consul-server-0:        hashicorp/consul:1.16.2, 
consul-server-1:        hashicorp/consul:1.16.2, 
consul-server-2:        hashicorp/consul:1.16.2,

When you upgrade the Helm chart, you can verify changes on the Argo CD server. You need to disable auto-sync for the Consul application. Commit and push the Helm chart version update to version control. Argo CD recognizes the configuration as out of sync. If you examine the application details, you can verify the changes to Consul resources.

Alternatively, you can define Consul as a single-source Argo CD application and run a local command for Argo CD to synchronize a dry run and identify the changes.

Once you verify the changes, update the Consul application for Argo CD with a target revision for the Helm release:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: consul
  namespace: argocd
spec:
  # omitted for clarity
 
  sources:
    # Deploy Consul Helm chart
    - chart: consul
      repoURL: https://helm.releases.hashicorp.com
      targetRevision: 1.x.x # Update to latest Helm chart
      helm:
        releaseName: consul
        valueFiles:
          - $values/argocd/consul-helm-values.yaml
 
    # omitted for clarity

To preserve the state of the Consul servers, set server.updatePartition to 3. Commit the Helm values to version control. This ensures that Argo CD synchronizes changes to other components before the servers:

global:
  name: consul
  datacenter: dc1
  image: hashicorp/consul:1.16.2
 
  # omitted for clarity
 
server:
  replicas: 3
  updatePartition: 3  # number of servers to avoid updating

Then, apply the Consul Argo CD application to your Kubernetes cluster:

$ kubectl apply -f consul-application.yaml

Decrement server.updatePartition and commit the Helm values to version control until it reaches zero. This applies gradual updates from the Helm chart to the Consul servers. Verify that services can still access each other and Consul remains healthy.

»Next steps

A GitOps workflow allows you to deploy and manage changes to a Consul cluster while orchestrating the configuration of a Consul service mesh for peering, network policy, and gateways. By ensuring that version control remains the primary source of truth for Consul cluster and service mesh configuration, you can consistently deploy configuration across different Consul clusters, clouds, and environments and standardize management of Consul across multiple Kubernetes clusters. Automation with Argo CD hooks and sync waves help synchronize Consul resources on Kubernetes, coordinate changes to different resources, and facilitate the upgrade of Consul servers and components.

For a detailed set of configurations, check out my demo repository. For more information about Argo CD, explore its documentation. To learn more about Consul on Kubernetes, check out our documentation and this tutorial on how to install Consul via Helm chart. Additional instructions can be found in our documentation on Upgrading Consul on Kubernetes components.

(Thank you to Christian Hernandez, Head of Community Engineering at Akuity, for educating and clarifying Argo CD details for this post.)

Kubernetes