A detailed comparison of three HashiCorp-supported methods for HashiCorp Vault and Kubernetes integration.
In this post, I will explore three different methods for integrating HashiCorp Vault with Kubernetes:
I’ll provide practical guidance for each method to help you understand and choose the best method for your use case.
This post is not intended to be a product documentation or step-by-step implementation guide. It is for DevOps practitioners familiar with HashiCorp Vault and Kubernetes who also have a basic understanding of secrets-management concepts.
The Vault Sidecar Agent Injector leverages the sidecar pattern to alter pod specifications to include a Vault Agent container that renders Vault secrets to a shared memory volume. By rendering secrets to a shared volume, containers within the pod can consume Vault secrets without being Vault-aware. The injector is a Kubernetes mutating webhook controller. The controller intercepts pod events and applies mutations to the pod if annotations exist within the request. This functionality is provided by the vault-k8s project and can be automatically installed and configured using the Vault Helm chart.
The Vault CSI provider allows pods to consume Vault secrets by using ephemeral CSI Secrets Store volumes. At a high level, the CSI Secrets Store driver allows users to create
SecretProviderClass objects. These objects define which secret provider to use and what secrets to retrieve. When pods requesting CSI volumes are created, the CSI Secrets Store driver sends the request to the Vault CSI provider if the provider is
vault. The Vault CSI provider then uses the specified
SecretProviderClass and the pod’s service account to retrieve the secrets from Vault and mount them into the pod’s CSI volume. Note that the secret is retrieved from Vault and populated to the CSI secrets store volume during the
ContainerCreation phase. This means that pods will be blocked from starting until the secrets have been read from Vault and written to the volume.
The Vault Secrets Operator is a new integration method that implements a Kubernetes Secrets Operator with a set of CRDs responsible for synchronizing Vault secrets to Kubernetes Secrets natively. The Operator supports synchronizing the full lifecycle of secrets management including static, dynamic, and PKI-based secrets from one or more Vault server instances. The Operator is also capable of managing secret rotation and performing post rotation actions including notifying an application directly via a rolling update of a Deployment or by triggering a rolling update.
There are some similarities and differences between the three solutions that you should consider when designing and implementing your secrets management strategy in Kubernetes environments.
Keeping these design considerations in mind, let’s go over some of the similarities and differences between the three integration solutions.
Vault Operator, CSI, and sidecar solutions:
Simplify retrieving different types of secrets stored in Vault and expose them to the target pod running on Kubernetes without it being aware of the not-so-trivial Vault processes. It’s important to note that there is no need to make any changes in the application logic or code in order for it to use these solutions, making it easier to migrate brownfield applications into Kubernetes. Developers working on greenfield applications can leverage the Vault SDKs to directly integrate with Vault.
Support all types of Vault secrets engines. This means that you can leverage an extensive set of secret types, ranging from static key-value secrets to dynamically generated database credentials and TLS certs with customized TTL.
Leverage the application’s Kubernetes pod service account token as “Secret Zero” to authenticate with Vault via the Kubernetes auth method. This means that there is no need to manage yet another separate identity to identify the application pods when authenticating to Vault.
Vault’s Kubernetes auth workflow
Here’s how the three solutions are different:
hostPathto mount ephemeral volumes into the pods, which some container platforms (e.g. OpenShift) disable by default. On the other hand, Sidecar Agent Service uses in-memory tmpfs volumes.
The table below provides a high-level comparison of the three solutions:
*achieved through Agent templating
On the surface, Kubernetes native secrets might seem similar to the three approaches presented above, but there are major differences between them:
Kubernetes is not a secrets management solution. It does have native support for secrets, but that is quite different from an enterprise secrets management solution. Kubernetes secrets are scoped to the cluster only and many applications will have some services running outside Kubernetes or in different Kubernetes clusters. Therefore, considering the secret scope as part of the design process is critical. Having these applications use Kubernetes secrets from outside a Kubernetes environment will be cumbersome and introduce authentication and authorization challenges.
Kubernetes secrets are static in nature. You can define secrets by using kubectl or the Kubernetes API but once they are defined they are stored in etcd and presented to pods only during pod creation. This can create scenarios where secrets get stale, outdated, or expired, requiring additional workflows to update and rotate the secrets and then re-deploying the application to use the new version of the secrets. This can add complexity and waste time. So make sure you consider any requirement for secret freshness, updates, and rotation as part of your design process.
The security model of secret access management is tied to the Kubernetes RBAC model. This can be challenging to adopt for users who are not familiar with Kubernetes. Adopting a platform-agnostic security governance model can enable you to adopt workflows for applications regardless of how and where they are running.
Designing for secrets management in Kubernetes is no easy task. There are multiple approaches each with its own set of pros and cons. I highly recommend exploring the options presented in this blog post to understand their internals and decide on the best option for your use case.
A recap of HashiCorp infrastructure and security news and developments on AWS from the past year, from self-service provisioning to fighting secrets sprawl and more.
Vault benchmark is an open source tool that tests the performance of HashiCorp Vault auth methods and secrets engines.
If you’re attending AWS re:Invent in Las Vegas, Nov. 27 - Dec. 1, visit us for breakout sessions, expert talks, and product demos to learn how to accelerate your adoption of a cloud operating model.