A detailed comparison of two HashiCorp-supported methods for HashiCorp Vault and Kubernetes integration.
In this post, I will explore two different methods for integrating HashiCorp Vault with Kubernetes:
I’ll provide practical guidance for both methods to help you understand and choose the best method for your use case.
This post is not intended to be a product documentation or step-by-step implementation guide. It is for DevOps practitioners familiar with HashiCorp Vault and Kubernetes who also have a basic understanding of secrets-management concepts.
The Vault Sidecar Agent Injector leverages the sidecar pattern to alter pod specifications to include a Vault Agent container that renders Vault secrets to a shared memory volume. By rendering secrets to a shared volume, containers within the pod can consume Vault secrets without being Vault-aware. The injector is a Kubernetes mutating webhook controller. The controller intercepts pod events and applies mutations to the pod if annotations exist within the request. This functionality is provided by the vault-k8s project and can be automatically installed and configured using the Vault Helm chart.
The Vault CSI provider allows pods to consume Vault secrets by using ephemeral CSI Secrets Store volumes. At a high level, the CSI Secrets Store driver allows users to create SecretProviderClass
objects. These objects define which secret provider to use and what secrets to retrieve. When pods requesting CSI volumes are created, the CSI Secrets Store driver sends the request to the Vault CSI provider if the provider is vault
. The Vault CSI provider then uses the specified SecretProviderClass
and the pod’s service account to retrieve the secrets from Vault and mount them into the pod’s CSI volume. Note that the secret is retrieved from Vault and populated to the CSI secrets store volume during the ContainerCreation
phase. This means that pods will be blocked from starting until the secrets have been read from Vault and written to the volume.
There are some similarities and differences between the two solutions that you should consider when designing and implementing your secrets management strategy in Kubernetes environments. Common Design Considerations
Keeping these design considerations in mind, let’s go over some of the similarities and differences between the two integration solutions.
Both CSI and sidecar solutions:
Simplify retrieving different types of secrets stored in Vault and expose them to the target pod running on Kubernetes without it being aware of the not-so-trivial Vault processes. It’s important to note that there is no need to make any changes in the application logic or code in order for it to use these solutions, making it easier to migrate brownfield applications into Kubernetes. Developers working on greenfield applications can leverage the Vault SDKs to directly integrate with Vault.
Support all types of Vault secrets engines. This means that you can leverage an extensive set of secret types, ranging from static key-value secrets to dynamically generated database credentials and TLS certs with customized TTL.
Leverage the application’s Kubernetes pod service account token as “Secret Zero” to authenticate with Vault via the Kubernetes auth method. This means that there is no need to manage yet another separate identity to identify the application pods when authenticating to Vault.
Vault’s Kubernetes auth workflow
Here’s how the two solutions are different:
The Sidecar Agent Injector solution is composed of two elements:
In contrast, the Vault CSI Driver is deployed as a daemonset on every node in the Kubernetes cluster and uses the Secret Provider Class specified and the pod’s service account to retrieve the secrets from Vault, and mount them into the pod’s CSI volume.
The Sidecar Agent Injector supports all Vault auto-auth methods. The Sidecar CSI driver supports only Vault’s Kubernetes auth method.
The sidecar container that is launched with every application pod uses Vault Agent, which provides a powerful set of capabilities such as auto-auth, templating, and caching. The CSI driver does not use the Vault Agent and therefore lacks these functionalities.
The Vault CSI driver supports rendering Vault secrets into both Kubernetes secrets and environment variables. Sidecar Injector Service does not support rendering secrets into Kubernetes secrets — however there are ways to use agent templating to render secrets into environment variables.
The CSI driver uses hostPath
to mount ephemeral volumes into the pods, which some container platforms (e.g. OpenShift) disable by default. On the other hand, Sidecar Agent Service uses in-memory tmpfs volumes.
The table below provides a high-level comparison of the two solutions:
*achieved through Agent templating
On the surface, Kubernetes native secrets might seem similar to the two approaches presented above, but there are major differences between them:
Kubernetes is not a secrets management solution. It does have native support for secrets, but that is quite different from an enterprise secrets management solution. Kubernetes secrets are scoped to the cluster only and many applications will have some services running outside Kubernetes or in different Kubernetes clusters. Therefore, considering the secret scope as part of the design process is critical. Having these applications use Kubernetes secrets from outside a Kubernetes environment will be cumbersome and introduce authentication and authorization challenges.
Kubernetes secrets are static in nature. You can define secrets by using kubectl or the Kubernetes API but once they are defined they are stored in etcd and presented to pods only during pod creation. This can create scenarios where secrets get stale, outdated, or expired, requiring additional workflows to update and rotate the secrets and then re-deploying the application to use the new version of the secrets. This can add complexity and waste time. So make sure you consider any requirement for secret freshness, updates, and rotation as part of your design process.
The security model of secret access management is tied to the Kubernetes RBAC model. This can be challenging to adopt for users who are not familiar with Kubernetes. Adopting a platform-agnostic security governance model can enable you to adopt workflows for applications regardless of how and where they are running.
Designing for secrets management in Kubernetes is no easy task. There are multiple approaches each with its own set of pros and cons. I highly recommend exploring the options presented in this blog post to understand their internals and decide on the best option for your use case.
The HashiCorp Vault ecosystem continues to grow with the addition of 25 new integrations this past quarter.
Here’s how to use HashiCorp Boundary to provide identity-based remote access and credential management for Kubernetes clusters.
Before we ring in the new year, here’s a look back at some of the most important moments in 2022 for HashiCorp.