vault

Manage Kubernetes Secrets for Flux with HashiCorp Vault

Configure the Secrets Store CSI driver with HashiCorp Vault to securely inject secrets into Flux or other GitOps tools on Kubernetes.

When you use GitOps to deploy your services to Kubernetes, you need a way for services to securely access secrets such as database credentials, deploy tokens, or certificates. GitOps works by using Git as the single source of truth for declarative infrastructure and applications. When you first implement GitOps, you may start by encrypting and committing your secrets to version control and enable a GitOps tool to decrypt them. For example, Flux natively supports Vault through SOPS.

However, encrypting and committing your secrets to version control comes with the risk of having to remediate compromised secrets if you accidentally commit them unencrypted. This approach also does not scale as your development teams, Kubernetes clusters, applications, and infrastructure grow. How do you dynamically inject secrets into your services deployed through GitOps?

This post will show two approaches to configuring a Kubernetes application deployed by the Flux framework to use secrets from HashiCorp Vault. When you use Kubernetes secrets, you have the additional management overhead of distributing and auditing hundreds of secrets across different namespaces and setting up role-based access control (RBAC) to limit access to secrets. Instead, you can use a file-based approach to injecting secrets by adding annotations for the Vault Agent sidecar injector or defining a volume for the Vault Container Storage Interface (CSI) provider.

However, you may have applications such as Flux that depend on Kubernetes secrets and cannot use the Vault Agent sidecar injector or CSI provider. This post introduces a second approach, in which the Vault CSI provider synchronizes secrets from Vault to a Kubernetes secret, allowing applications to reference the secret without refactoring. The approach offers an intermediate solution to let Kubernetes deployments migrate to using Vault while minimizing refactoring impact and integrating with tools that depend on Kubernetes secrets.

»Recommended Approach: File-Based Secrets Injection

In general, avoid using unencrypted, plaintext Kubernetes secrets. Instead, use a file-based approach to retrieve dynamic secrets. A file-based-approach limits access to secrets to the application’s container and service account. Writing the secret to a file in a container volume removes the need to manage separate Kubernetes RBAC permissions for different secrets and automatically updates the secret when you rotate it in Vault. The approach reduces potential errors resulting from incorrect RBAC permissions or revoked secrets.

Our recommended Vault integration approach writes secrets to a file in a volume mount using the Vault Agent sidecar injector or CSI provider. For more information on the differences between these methods, review our blog post comparing Kubernetes-Vault Integration via Sidecar Agent vs. CSI Provider.

With a file-based approach, you must refactor Kubernetes manifests for your applications to use these methods. After refactoring the application’s Kubernetes manifest for a file-based secrets injection approach, you can deploy the applications normally with GitOps tools like Flux. You do not need to configure Flux to retrieve secrets from Vault, as the application will automatically authenticate and retrieve the secret when it starts.

»Vault Agent Sidecar Injector

For the Agent sidecar injector, annotate your deployments with the path to the Vault credentials and a template to write a file with secrets.

HashiCorp Vault agent sidecar injector injects Vault agent based on annotations, which writes secrets to a container volume

The following example uses Vault annotations to inject a sidecar for a product-api. The Vault Agent sidecar will write the database credentials to a local file in a container volume called conf.json. The application container reads the file from /vault/secrets/conf.json.

apiVersion: apps/v1
kind: Deployment
metadata:
 name: product-api
 labels:
   app: product-api
spec:
 replicas: 1
 selector:
   matchLabels:
     app: product-api
 template:
   metadata:
     labels:
       app: product-api
     annotations:
       vault.hashicorp.com/agent-inject: "true"
       vault.hashicorp.com/role: "product"
       vault.hashicorp.com/agent-inject-secret-conf.json: "hashicups/database/creds/product"
       vault.hashicorp.com/agent-inject-template-conf.json: |
         {
           "bind_address": ":9090",
           "metrics_address": ":9103",
           {{ with secret "hashicups/database/creds/product" -}}
           "db_connection": "host=postgres port=5432 user={{ .Data.username }} password={{ .Data.password }} dbname=products sslmode=disable"
           {{- end }}
         }
   spec:
     serviceAccountName: product-api
     containers:
       - name: product-api
         image: hashicorpdemoapp/product-api:v0.0.20
         ports:
           - containerPort: 9090
           - containerPort: 9103
         env:
           - name: "CONFIG_FILE"
             value: "/vault/secrets/conf.json"
apiVersion: apps/v1kind: Deploymentmetadata: name: product-api labels:   app: product-apispec: replicas: 1 selector:   matchLabels:     app: product-api template:   metadata:     labels:       app: product-api     annotations:       vault.hashicorp.com/agent-inject: "true"       vault.hashicorp.com/role: "product"       vault.hashicorp.com/agent-inject-secret-conf.json: "hashicups/database/creds/product"       vault.hashicorp.com/agent-inject-template-conf.json: |         {           "bind_address": ":9090",           "metrics_address": ":9103",           {{ with secret "hashicups/database/creds/product" -}}           "db_connection": "host=postgres port=5432 user={{ .Data.username }} password={{ .Data.password }} dbname=products sslmode=disable"           {{- end }}         }   spec:     serviceAccountName: product-api     containers:       - name: product-api         image: hashicorpdemoapp/product-api:v0.0.20         ports:           - containerPort: 9090           - containerPort: 9103         env:           - name: "CONFIG_FILE"             value: "/vault/secrets/conf.json"

»Vault CSI Provider

For the Vault CSI provider, you must have the Secrets Store CSI driver installed on your Kubernetes cluster. Follow the instructions in our Learn tutorial to install and configure the Vault CSI provider.

HashiCorp Vault CSI provider gets secret from Vault based on SecretProviderClass and mounts the CSI volume to the pod

Define a SecretProviderClass for the Vault CSI provider to retrieve secrets from Vault.

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
 name: products-db
spec:
 provider: vault
 parameters:
   roleName: 'product'
   vaultAddress: 'http://vault:8200'
   objects: |
     - objectName: "db-username"
       secretPath: "hashicups/database/creds/product"
       secretKey: "username"
     - objectName: "db-password"
       secretPath: "hashicups/database/creds/product"
       secretKey: "password"
apiVersion: secrets-store.csi.x-k8s.io/v1kind: SecretProviderClassmetadata: name: products-dbspec: provider: vault parameters:   roleName: 'product'   vaultAddress: 'http://vault:8200'   objects: |     - objectName: "db-username"       secretPath: "hashicups/database/creds/product"       secretKey: "username"     - objectName: "db-password"       secretPath: "hashicups/database/creds/product"       secretKey: "password"

Add the SecretProviderClass as a volume mount for the deployment. The volume contains one file, called username, with the database username. It contains another file, called password, with the database password.

kind: Deployment
metadata:
 name: product-api
 labels:
   app: product-api
spec:
 replicas: 1
 selector:
   matchLabels:
     app: product-api
 template:
   metadata:
     labels:
       app: product-api
   spec:
     serviceAccountName: product-api
     containers:
       - name: product-api
         image: hashicorpdemoapp/product-api:v0.0.20
         ports:
           - containerPort: 9090
           - containerPort: 9103
         volumeMounts:
           - name: products-db
             mountPath: '/mnt/secrets-store'
             readOnly: true
     volumes:
       - name: products-db
         csi:
           driver: 'secrets-store.csi.k8s.io'
           readOnly: true
           volumeAttributes:
             secretProviderClass: products-db
kind: Deploymentmetadata: name: product-api labels:   app: product-apispec: replicas: 1 selector:   matchLabels:     app: product-api template:   metadata:     labels:       app: product-api   spec:     serviceAccountName: product-api     containers:       - name: product-api         image: hashicorpdemoapp/product-api:v0.0.20         ports:           - containerPort: 9090           - containerPort: 9103         volumeMounts:           - name: products-db             mountPath: '/mnt/secrets-store'             readOnly: true     volumes:       - name: products-db         csi:           driver: 'secrets-store.csi.k8s.io'           readOnly: true           volumeAttributes:             secretProviderClass: products-db

»File-Based Approach with Environment Variables

If your application uses environment variables, you can refactor its Kubernetes manifest to source environment variables from a file. For example, the following database deployment includes a command to source an environment variable storing the database password.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: expense-db-mysql
  labels:
    app: expense-db-mysql
spec:
  replicas: 1
  selector:
    matchLabels:
      app: expense-db-mysql
  template:
    metadata:
      annotations:
        vault.hashicorp.com/agent-inject: "true"
        vault.hashicorp.com/role: "expense-db-mysql"
        vault.hashicorp.com/agent-inject-secret-db: "expense/static/data/mysql"
        vault.hashicorp.com/agent-inject-template-db: |
          {{ with secret "expense/static/data/mysql" -}}
          export MYSQL_ROOT_PASSWORD="{{ .Data.data.db_login_password }}"
          {{- end }}
      labels:
        app: expense-db-mysql
    spec:
      serviceAccountName: expense-db-mysql
      containers:
        - name: expense-db-mysql
          image: "joatmon08/expense-db:mysql-8"
          ports:
            - containerPort: 3306
          command: ["/bin/bash"]
          args: ["-c", "source /vault/secrets/db && /usr/local/bin/docker-entrypoint.sh mysqld"]
apiVersion: apps/v1kind: Deploymentmetadata:  name: expense-db-mysql  labels:    app: expense-db-mysqlspec:  replicas: 1  selector:    matchLabels:      app: expense-db-mysql  template:    metadata:      annotations:        vault.hashicorp.com/agent-inject: "true"        vault.hashicorp.com/role: "expense-db-mysql"        vault.hashicorp.com/agent-inject-secret-db: "expense/static/data/mysql"        vault.hashicorp.com/agent-inject-template-db: |          {{ with secret "expense/static/data/mysql" -}}          export MYSQL_ROOT_PASSWORD="{{ .Data.data.db_login_password }}"          {{- end }}      labels:        app: expense-db-mysql    spec:      serviceAccountName: expense-db-mysql      containers:        - name: expense-db-mysql          image: "joatmon08/expense-db:mysql-8"          ports:            - containerPort: 3306          command: ["/bin/bash"]          args: ["-c", "source /vault/secrets/db && /usr/local/bin/docker-entrypoint.sh mysqld"]

With file-based secrets injection, you localize access to the secret to the application deployment. You avoid committing secrets (even encrypted) to version control and let a secrets manager handle the rotation of the secret. On the other hand, you must refactor your application’s Kubernetes manifest to reference the file containing the secret. Your application must also enable a reload capability to use the updated secrets file when Vault rotates the credentials.

»Alternative Approach: Sync as Kubernetes Secrets

If you use GitOps frameworks to deploy your applications and cannot refactor your application to use the file-based approach, you can use the Vault CSI provider to synchronize a Vault secret to a Kubernetes secret. Store your static credentials (such as a repository token) or set up dynamic credentials (such as a database username) in Vault and use the Vault CSI provider to write them to a Kubernetes secret.

HashiCorp Vault CSI Provider retrieves secrets from Vault and syncs a Kubernetes secret

With this approach, you do not need to refactor your application’s Kubernetes manifest. Furthermore, this approach automatically handles any updates you make to secrets in Vault and synchronizes the changes to a Kubernetes secret. However, make sure to control access to the Kubernetes secret with RBAC policies. This prevents unauthorized applications or users from retrieving the unencrypted secret.

To start using the Vault CSI provider, you need the Secrets Store CSI driver installed on your Kubernetes cluster. Follow the instructions in our Learn tutorial to install and configure the Vault CSI provider. You will also need to set up the Kubernetes authentication method in your Vault cluster to allow authentication by Kubernetes service accounts.

»Example: Injecting GitLab Deploy Tokens to Flux

The example described here synchronizes a GitLab deploy token for a private repository to a Kubernetes secret using the Vault CSI provider. Flux can reference the Kubernetes secret. Similarly, you can extend this workflow to synchronize secrets from any Vault secrets engine to any applications that references Kubernetes secrets as files or environment variables. You can manage static secrets like a GitLab deploy token for a private repository in Vault and automatically sync updates to a Kubernetes secret using the Vault CSI provider. Flux needs access to the deploy token in order to synchronize with a private repository. This example uses HashiCorp Terraform to configure GitLab and Vault.

HashiCorp Vault CSI Provider retrieves GitLab credential from Vault and syncs a Kubernetes secret for Flux to use

Use the GitLab provider for Terraform to create a GitLab deploy token for your private GitLab project (repository).

resource "gitlab_deploy_token" "hashicups" {
 depends_on = [gitlab_project.hashicups]
 project    = gitlab_project.hashicups.id
 name       = "Deploy Token for Flux"
 username   = local.flux_username
 
 scopes = ["read_repository", "read_registry"]
}
resource "gitlab_deploy_token" "hashicups" { depends_on = [gitlab_project.hashicups] project    = gitlab_project.hashicups.id name       = "Deploy Token for Flux" username   = local.flux_username  scopes = ["read_repository", "read_registry"]}

Then, store the GitLab username and token as a static secret in Vault’s key-value secrets engine. Pass the token into the password field. Flux needs username and password data to access a private repository.

resource "vault_mount" "flux" {
 path        = "hashicups/flux"
 type        = "kv-v2"
 description = "For hashicups Flux deploy secrets"
}
 
resource "vault_generic_secret" "gitlab" {
 path = "${vault_mount.flux.path}/gitlab"
 
 data_json = jsonencode({
   username = "${gitlab_deploy_token.hashicups.username}"
   password = "${gitlab_deploy_token.hashicups.token}"
 })
}
resource "vault_mount" "flux" { path        = "hashicups/flux" type        = "kv-v2" description = "For hashicups Flux deploy secrets"} resource "vault_generic_secret" "gitlab" { path = "${vault_mount.flux.path}/gitlab"  data_json = jsonencode({   username = "${gitlab_deploy_token.hashicups.username}"   password = "${gitlab_deploy_token.hashicups.token}" })}

Define a Vault role with the Vault provider for Terraform. The Vault role allows Flux’s source-controller service account in the flux-system namespace to retrieve the username and password for the private repository. This ensures that Flux can read the secret but not change it.

data "vault_policy_document" "gitlab" {
 rule {
   path = "hashicups/flux/data/gitlab"
 
   capabilities = [
     "read"
   ]
 
   description = "read GitLab credentials"
 }
}
 
resource "vault_policy" "gitlab" {
 name   = "gitlab"
 policy = data.vault_policy_document.gitlab.hcl
}
 
resource "vault_kubernetes_auth_backend_role" "gitlab" {
 backend                          = vault_auth_backend.kubernetes.path
 role_name                        = "flux"
 bound_service_account_names      = ["source-controller"]
 bound_service_account_namespaces = ["flux-system"]
 token_ttl                        = 3600
 token_policies                   = [vault_policy.gitlab.name]
}
data "vault_policy_document" "gitlab" { rule {   path = "hashicups/flux/data/gitlab"    capabilities = [     "read"   ]    description = "read GitLab credentials" }} resource "vault_policy" "gitlab" { name   = "gitlab" policy = data.vault_policy_document.gitlab.hcl} resource "vault_kubernetes_auth_backend_role" "gitlab" { backend                          = vault_auth_backend.kubernetes.path role_name                        = "flux" bound_service_account_names      = ["source-controller"] bound_service_account_namespaces = ["flux-system"] token_ttl                        = 3600 token_policies                   = [vault_policy.gitlab.name]}

After setting up the static secret in Vault, deploy a SecretProviderClass in the flux-system namespace to allow the Vault CSI provider to retrieve it from Vault. Unlike the previous example for SecretProviderClass, this definition includes the secretObjects field. secretObjects writes the username and password to a Kubernetes secret called gitlab-credentials with type Opaque. Our example uses the flux Vault role to retrieve the GitLab repository’s username and deploy token.

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
 name: gitlab-credentials
 namespace: flux-system
spec:
 provider: vault
 secretObjects:
   - secretName: gitlab-credentials
     type: Opaque
     data:
       - objectName: gitlab-username
         key: username
       - objectName: gitlab-password
         key: password
 parameters:
   roleName: 'flux'
   vaultAddress: 'http://vault:8200'
   objects: |
     - objectName: "gitlab-username"
       secretPath: "hashicups/flux/data/gitlab"
       secretKey: "username"
     - objectName: "gitlab-password"
       secretPath: "hashicups/flux/data/gitlab"
       secretKey: "password"
apiVersion: secrets-store.csi.x-k8s.io/v1kind: SecretProviderClassmetadata: name: gitlab-credentials namespace: flux-systemspec: provider: vault secretObjects:   - secretName: gitlab-credentials     type: Opaque     data:       - objectName: gitlab-username         key: username       - objectName: gitlab-password         key: password parameters:   roleName: 'flux'   vaultAddress: 'http://vault:8200'   objects: |     - objectName: "gitlab-username"       secretPath: "hashicups/flux/data/gitlab"       secretKey: "username"     - objectName: "gitlab-password"       secretPath: "hashicups/flux/data/gitlab"       secretKey: "password"

However, the Secrets Store CSI driver does not create the secret until a deployment creates a volume mount. Create a deployment that mounts the Secrets Store CSI volume. The deployment creates a Kubernetes secret called gitlab-credentials when it mounts the volume. Make sure the deployment uses the same serviceAccountName (source-controller) and namespace (flux-system) configured in the Vault role.

apiVersion: apps/v1
kind: Deployment
metadata:
 name: secrets-store-inline
 namespace: flux-system
 labels:
   app: secrets-store-inline
spec:
 replicas: 1
 selector:
   matchLabels:
     app: secrets-store-inline
 template:
   metadata:
     labels:
       app: secrets-store-inline
   spec:
     serviceAccountName: source-controller
     containers:
       - name: busybox
         image: k8s.gcr.io/e2e-test-images/busybox:1.29
         command:
         - "/bin/sleep"
         - "10000"
         volumeMounts:
         - name: secrets-store
           mountPath: "/mnt/secrets-store"
           readOnly: true
     volumes:
       - name: secrets-store
         csi:
           driver: secrets-store.csi.k8s.io
           readOnly: true
           volumeAttributes:
             secretProviderClass: "gitlab-credentials"
apiVersion: apps/v1kind: Deploymentmetadata: name: secrets-store-inline namespace: flux-system labels:   app: secrets-store-inlinespec: replicas: 1 selector:   matchLabels:     app: secrets-store-inline template:   metadata:     labels:       app: secrets-store-inline   spec:     serviceAccountName: source-controller     containers:       - name: busybox         image: k8s.gcr.io/e2e-test-images/busybox:1.29         command:         - "/bin/sleep"         - "10000"         volumeMounts:         - name: secrets-store           mountPath: "/mnt/secrets-store"           readOnly: true     volumes:       - name: secrets-store         csi:           driver: secrets-store.csi.k8s.io           readOnly: true           volumeAttributes:             secretProviderClass: "gitlab-credentials"

The deployment can use any container image compliant with your organization and use limited resources. In general, it helps to create a separate deployment that initializes and continuously synchronizes the secret. This means tools and applications can reference the Kubernetes secret as an abstraction between Vault and Flux. Flux does not have to read the secret directly from Vault.

Check if the Secrets Store CSI driver created a Kubernetes secret called gitlab-credentials in the flux-system namespace. It should have two data entries.

$ kubectl get secrets gitlab-credentials -n flux-system
 
NAME                                  TYPE                                  DATA   AGE
gitlab-credentials                    Opaque                                2      4h14m
$ kubectl get secrets gitlab-credentials -n flux-system NAME                                  TYPE                                  DATA   AGEgitlab-credentials                    Opaque                                2      4h14m

Use this private repository username and token to set up a GitRepository in Flux. Update secretRef to reference the gitlab-credentials Kubernetes secret.

apiVersion: source.toolkit.fluxcd.io/v1beta1
kind: GitRepository
metadata:
 name: hashicups
 namespace: flux-system
spec:
 interval: 1m0s
 ref:
   branch: main
 url: https://gitlab.com/joatmon08/hashicups.git
 secretRef:
   name: gitlab-credentials
apiVersion: source.toolkit.fluxcd.io/v1beta1kind: GitRepositorymetadata: name: hashicups namespace: flux-systemspec: interval: 1m0s ref:   branch: main url: https://gitlab.com/joatmon08/hashicups.git secretRef:   name: gitlab-credentials

Any Kustomization referencing the GitRepository source should successfully log into the GitLab private repository and deploy Kubernetes resources.

You can apply the Flux GitRepository example to retrieve secrets from any Vault secrets engine for any application. Your applications and object manifests can keep using references to Kubernetes secrets and benefit from Vault managing the secrets. Setting up the Vault CSI driver with synchronization to Kubernetes secrets minimizes the disruption of migrating your application’s secrets to a secrets manager. Later, you can gradually refactor the application to use file-based secrets injection.

»Conclusion

When possible, use a file-based approach to inject secrets from Vault into Kubernetes deployments. You can use the Vault Agent sidecar injector or CSI provider to read secrets from Vault. Both of them write the secret to a file on a container volume, which removes the need to use Kubernetes RBAC to secure access to the secret and handles dynamic credentials.

When should you consider synching a Vault secret as Kubernetes secret with the Vault CSI provider? Do it when you need to use Kubernetes secrets for compatibility with a GitOps tool like Flux, or if you just want to minimize the impact of refactoring applications. By implementing a background deployment to mount the CSI volume, you allow the controller to continuously read secrets from Vault and update a Kubernetes secret. Secure access to the Kubernetes secret and Secrets Store CSI driver objects with Kubernetes RBAC.

If you would like to review the configuration in this blog post, check out the code repository for this post. For additional information on deploying Vault on Kubernetes, review our Learn tutorial. Get started on file-based secrets injection with tutorials on the Vault agent sidecar injector and CSI provider. Learn more about Flux here to deploy your applications in a GitOps manner.

Questions about this post? Add them to the community forum!

Sign up for the latest HashiCorp news