Kubernetes secrets management with HCP Vault Secrets

Learn how to sync secrets from HCP Vault Secrets to Kubernetes with Vault Secrets Operator.

Most Kubernetes resources and workloads reference the Kubernetes Secret object for credentials, API tokens, certificates, and other confidential data. Kubernetes stores secrets unencrypted by default and requires role-based access control (RBAC) rules to ensure least-privilege access. However, it does not offer a straightforward method for tracking the lifecycle and distribution of the secret. Rather than store secrets in Kubernetes, you can use a centralized secrets management solution like HashiCorp Cloud Platform (HCP) Vault Secrets to audit and manage secrets.

This post demonstrates how to use the Vault Secrets Operator (VSO) to retrieve dynamic secrets from HCP Vault Secrets and write them to a Kubernetes Secret for other workloads and resources to reference. HCP Vault Secrets stores a set of secrets and supports the management and audit of their lifecycle and distribution. The demo app repository for this tutorial uses HashiCorp Terraform to write the secrets into HCP Vault Secrets, deploy the Vault Secrets Operator to the Kubernetes cluster, and deploy the custom resources to synchronize a secret for Argo CD to create a private repository. This workflow minimizes the need to refactor applications to access a secrets manager directly by using native Kubernetes Secrets.

»Store secrets in HCP Vault Secrets

HCP Vault Secrets enables you to manage the lifecycle of credentials and track their usage. For example, a GitOps tool like Argo CD requires credentials to access a private repository on GitHub. These credentials may include a private key, username and password, or token to allow Argo CD to read from a repository. If you set up the GitHub App, you need to store the application identifier and private key for Argo CD to use. Storing the credentials in HCP Vault Secrets and installing the Vault Secrets Operator ensures the credentials get synchronized with a Kubernetes Secret, which Argo CD will reference.

Workflow for Vault Secrets Operator & HCP Vault Secrets

Workflow for Vault Secrets Operator & HCP Vault Secrets

The configuration example below uses the HCP provider for Terraform to store the GitHub App’s private key, application identifier, and application installation identifier in a HCP Vault Secrets application named argocd:

resource "hcp_vault_secrets_app" "argocd" { app_name    = "argocd" description = "Secrets related to running Argo CD on Kubernetes"} resource "hcp_vault_secrets_secret" "argocd_github_app_id" { app_name     = hcp_vault_secrets_app.argocd.app_name secret_name  = "githubAppID" secret_value = var.argocd_github_app.id} resource "hcp_vault_secrets_secret" "argocd_github_app_installation_id" { app_name     = hcp_vault_secrets_app.argocd.app_name secret_name  = "githubAppInstallationID" secret_value = var.argocd_github_app.installation_id} resource "hcp_vault_secrets_secret" "argocd_github_app_private_key" { app_name     = hcp_vault_secrets_app.argocd.app_name secret_name  = "githubAppPrivateKey" secret_value = base64decode(var.argocd_github_app.private_key)} resource "hcp_vault_secrets_secret" "argocd_github_url" { app_name     = hcp_vault_secrets_app.argocd.app_name secret_name  = "url" secret_value = var.argocd_github_app.url}

After applying the Terraform configuration, you can access the secrets in HCP Vault Secrets under the argocd application. You then need to synchronize these secrets from the argocd HCP Vault Secrets application into Kubernetes for Argo CD to reference.

Argo CD secrets in HCP Vault Secrets

Argo CD secrets in HCP Vault Secrets

»Install Vault Secrets Operator

Vault Secrets Operator helps synchronize secrets from HCP Vault Secrets or Vault into Kubernetes Secrets. The Operator handles the creation of custom resources that define the authentication to, and retrieval of, secrets from HCP Vault Secrets.

Here’s how to install Vault Secrets Operator using its Helm chart:

$ helm repo add hashicorp https://helm.releases.hashicorp.com
$ helm install -n vault-secrets-operator --create-namespace \  vault-secrets-operator hashicorp/vault-secrets-operator

The operator’s Helm chart includes a set of custom resource definitions for authenticating to HCP Vault Secrets.

»Authenticate to HCP Vault Secrets from Kubernetes

The Vault Secrets Operator needs a service principal to authenticate to HCP Vault Secrets. You can use Terraform to create a separate service principal with the viewer role (read-only access) that gives the operator read access to the secret:

resource "hcp_service_principal" "argocd" { name = "argocd"} resource "hcp_service_principal_key" "argocd" { service_principal = hcp_service_principal.argocd.resource_name} resource "hcp_project_iam_binding" "argocd" { project_id   = var.hcp_project_id principal_id = hcp_service_principal.argocd.resource_id role         = "roles/viewer"}

Save the HCP principal’s client ID and key to a Kubernetes Secret in the argocd namespace:

apiVersion: v1data:  clientID: REDACTED  clientSecret: REDACTEDkind: Secretmetadata:  name: hvs-service-principal  namespace: argocdtype: Opaque

The Vault Secrets Operator refers to the HCPAuth resource to authenticate to an HCP project with the read-only service principal you created in the argocd namespace:

apiVersion: secrets.hashicorp.com/v1beta1kind: HCPAuthmetadata:  name: default  namespace: argocdspec:  method: servicePrincipal  organizationID: HCP_ORG_ID  projectID: HCP_PROJECT_ID  servicePrincipal:    secretRef: hvs-service-principal

After deploying the HCPAuth resource to the cluster, you can now define a resource to synchronize secrets from HCP Vault Secrets to Kubernetes.

»Sync secrets from HCP Vault Secrets

Use the HCPVaultSecretsApp resource to define the secrets VSO synchronizes from Vault to Kubernetes. You can define a destination Kubernetes Secret for the credentials and each key in HCP Vault Secrets will map to a key in the secret. If the name of the HCP Vault Secrets key does not match the required Kubernetes secret key you need for a workload, you can configure transformations for each key in the secret. VSO will also refresh the secret on an interval defined in the refreshAfter attribute.

For example, Argo CD creates private repositories by scanning for Kubernetes Secrets with the argocd.argoproj.io/secret-type label. The HCPVaultSecretsApp resource for this tutorial’s GitHub repository includes that label in the destination. It also reads each secret from HCP Vault Secrets and maps it to the keys required by Argo CD, such as githubAppID and githubAppPrivateKey. The repository secret for Argo CD also requires the type key, which is set to git.

apiVersion: secrets.hashicorp.com/v1beta1kind: HCPVaultSecretsAppmetadata:  name: github-creds  namespace: argocdspec:  appName: argocd  destination:    create: true    labels:      argocd.argoproj.io/secret-type: repo-creds      hvs: "true"    name: github-creds    overwrite: false    transformation:      templates:        githubAppID:          name: githubAppID          text: '{{- get .Secrets "githubAppID" -}}'        githubAppInstallationID:          name: githubAppInstallationID          text: '{{- get .Secrets "githubAppInstallationID" -}}'        githubAppPrivateKey:          name: githubAppPrivateKey          text: '{{- get .Secrets "githubAppPrivateKey" -}}'        type:          name: type          text: git        url:          name: url          text: '{{- get .Secrets "url" -}}'  hcpAuthRef: default  refreshAfter: 1h

While the key names in HCP Vault Secrets do match the required keys for Argo CD, you add the transformations to demonstrate the value of re-mapping secrets and adding required fields, such as the type. In general, use the transformation field to create a Kubernetes Secret that conforms to the expected schema of any resource that uses it.

Once you apply the resource, VSO creates a Kubernetes Secret named github-creds with the fields and values defined in the transformation.

$ kubectl get secrets -n argocd github-creds -o yaml
apiVersion: v1data:  _raw: REDACTED  githubAppID: ODU4OTMx  githubAppInstallationID: NDg2Mzg5OTI=  githubAppPrivateKey: REDACTED  type: Z2l0  url: REDACTEDkind: Secretmetadata:  labels:    app.kubernetes.io/component: secret-sync    app.kubernetes.io/managed-by: hashicorp-vso    app.kubernetes.io/name: vault-secrets-operator    argocd.argoproj.io/secret-type: repo-creds    hvs: "true"  name: github-creds  namespace: argocd  ownerReferences:  - apiVersion: secrets.hashicorp.com/v1beta1    kind: HCPVaultSecretsApp    name: github-creds    uid: 729d7860-0065-4802-b892-dffbe15bbffbtype: Opaque

Argo CD recognizes the secret because of the argocd.argoproj.io/secret-type: repo-creds label. It creates a repository resource linked to the repository URL and GitHub App.

Repositories in Argo CD UI for code

Repositories in Argo CD UI for code

To verify any changes or review access to the secret, you can use the activity logs for the application in HCP Vault Secrets. The activity logs in HCP Vault Secrets indicate that the argocd service principal used by VSO has listed secrets under the argocd application.

Activity log in HCP Vault Secrets

Activity log in HCP Vault Secrets

If you create a new GitHub App and update the secrets in HCP Vault Secrets, VSO updates the github-creds secret with the new application IDs and private keys the next time it refreshes the secret. Argo CD updates the repository resource to use the new secrets without disrupting the repository connection. If you need to make changes to many GitHub Apps or credentials, you can update them all in HCP Vault Secrets without searching through Kubernetes namespaces and clusters.

»Learn more

If you currently store credentials in Kubernetes Secrets, you can copy them to HCP Vault Secrets and create resources to synchronize them into your Kubernetes cluster. This process avoids significant refactoring of Kubernetes workloads and lets you manage and track the secret lifecycles in a central location. While the example demonstrates how to synchronize secrets for Argo CD, you can use this pattern for other Kubernetes workloads and resources.

To get started, you can:

Sign up for the latest HashiCorp news

By submitting this form, you acknowledge and agree that HashiCorp will process your personal information in accordance with the Privacy Policy.