A Kubernetes User's Guide to HashiCorp Nomad Secret Management

Both Kubernetes and HashiCorp Nomad are mature orchestrators used for managing the lifecycle of containerized applications. We recently created a comparison and mapping of concepts between Kubernetes and HashiCorp Nomad in another guide titled: A Kubernetes User's Guide to HashiCorp Nomad. If you’re familiar with Kubernetes but not with Nomad, that guide will provide helpful context for this guide.

This guide is a continuation of the earlier guide focusing on the topic of secrets management; a major competency required for cloud security. A secret, in this context, means any sensitive information that you want to tightly control access to, such as API keys, passwords, OAuth tokens, certificates, and SSH keys.

Be sure to check out our Kubernetes to Nomad Cheat Sheet as well.

First, we will compare the native secrets management functionality of Kubernetes to HashiCorp Vault. Then we will outline how Vault integrates with Kubernetes and Nomad.

»Kubernetes Secrets

In Kubernetes, a Secret is an object that is used to store sensitive information as unencrypted base64-encoded strings. It can be mounted as a file on containers, made available as environment variables in the Pod, or retrieved when the image is pulled for the Pod.

Secrets are stored in etcd, the central key-value store for Kubernetes cluster data. This storage approach allows users to create one Secret that can then be referenced by any number of Pods.

Kubernetes secrets are static as they are generated in preparation for an application by users, not dynamically when the Pod initializes, and remain the same for long periods of time. While Kubernetes’s native capabilities enable a simple mechanism for users to store and retrieve sensitive data, its limitations could cause serious issues when deploying containers at scale in production.

»Limitations

First, Kubernetes secrets are base64 encoded, not encrypted. By default, they are available to anyone with API access or anyone with access to etcd running in the control plane.

More importantly, there is no lease or time-to-live with these secrets. They never expire and will remain in place until deleted even if the application using the secret has been removed or deleted from production.

For these reasons above, most Kuberentes users turn to external solutions to help them achieve better security. Among many secret management tools, HashiCorp Vault has the broadest adoption across many companies and industries.

»HashiCorp Vault

HashiCorp Vault provides a centralized secrets repository that encrypts data by default during transit and at rest. Vault provides options for static secrets, dynamic credentials, and has many plugins for extending core functionality. In addition, Vault has built-in mechanisms for secret renewal, rotation, and revocation, along with logging, monitoring, and telemetry support. The goal is to give you a centralized place to manage secrets using a common workflow across your applications.

Vault is built around a few core concepts.

»Core Concepts

Secrets: Vault stores, generates, or encrypts secrets via secret engines. Vault can generate dynamic secrets on-demand. With dynamic secrets, users delegate the responsibility for creating and managing the lifecycle of a secret to Vault.

Policies: Vault uses policies to manage and safeguard access and secret distribution to applications and infrastructure. Security teams write Vault policies to define which secrets and operations an authenticated user or machine can perform.

Authentication: Before a user or machine can interact with Vault, it must authenticate against an authentication backend. Once authenticated, a Vault token is returned to the user/machine with any attached policies that grant the token a set of permissions.

Encryption: Vault was built with the belief that everything should be encrypted at all times if possible. So it uses ciphertext wrapping to encrypt all data at rest and in-transit. This minimizes the exposure of secrets and sensitive information.

»Kubernetes and Vault Integrations

Vault can be integrated to store and manage access to secrets for Kubernetes in a number of ways. When integrated, Vault will broker access to secrets and the system instead of having Kubernetes use its native capabilities that allow Pods to use shared static credentials for things like database access.

»Authentication with Vault

Kubernetes has its own implementation of authentication and authorization. In Kubernetes, a Role contains a set of permissions and it can be granted to a human User that interacts with Kuberentes clusters or a Service Account, which provides an identity for Pods running in the cluster. Users and Pods can use those identities (service account credentials) as a mechanism to authenticate to the API and issue requests.

Vault comes with a pluggable authentication backend to allow information to be verified by external authentication providers, such as GitHub, LDAP, IAM, etc. Vault supports the Kubernetes authentication method and the actual authentication is delegated to Kuberentes. Users need to map the identity data in Kubernetes to a set of Vault policies.

Pods can use a Kuberentes service account token to authenticate to Vault. Vault verifies this token against Kubernetes' TokenReviewer API. If the token is valid, Vault maps the Pod’s identity to a Vault policy using the mapping configured previously. Vault returns an internally managed Vault Token to the Pod, attaching the correct policies assigned.

»Dynamic Secrets with Vault

Vault can generate dynamic secrets on-demand and manage the lifecycle of a dynamic secret for users. When the frontend Pod needs to access a backend database, the Pod will ask Vault. Vault, acting as a broker, will connect to the database and generate a set of unique credentials with a limited scope and lifetime. Vault then grants one of the credentials to this Pod. Vault will automatically rotate the credentials and revoke the access when they are no longer needed.

»Vault Agent Injector and Annotation

From an implementation perspective, the authentication and secret retrieval described above can be automated by Vault Agent injector and Pod annotations. The injector is a Kubernetes Mutation Webhook Controller, which can be automatically installed and configured using the official Vault Helm chart.

The controller intercepts events related to creating or updating Pods. Users add secret-related annotations into the Pod spec. When the injector parses an event and finds annotations in the Pod spec, the injector will modify the deployment spec to include Vault agent containers into the Pod.

The Vault agent containers can render secrets to a shared memory volume in the Pod using Vault Agent Templates. Users can define what secrets they want and how to render them with templating configurations. In this way, secrets are automatically available for the containers in the same Pod and the containers are not required to be aware of Vault's existence.

»Nomad and Vault Integration

Unlike Kubernetes, Nomad is designed to focus solely on cluster management and scheduling. It makes no attempt to provide its own native secrets management capabilities and delegates that, along with role-based-access control, to Vault or whatever external authentication providers you choose.

Similar to Kubernetes, Nomad doesn’t run Vault for you. Users will need to configure and install Vault separately. However,Nomad’s native integration with Vault allows users to take full advantage of Vault’s capabilities in secret management and enables simplified operations for users. Nomad server and client agents coordinate with Vault to derive a Vault token, extract the policies attached to it, make the token available to the task, and handle the token’s renewal.

Enabling Vault integration with Nomad only requires configuration for the location of the Vault service and a Vault token in a Nomad server’s configuration file. The policy attached to the token must enable the Nomad server to perform a basic set of operations for Vault integration.

In terms of authentication, Nomad delegates the responsibility to Vault. Because Vault supports a number of authentication backends, Nomad can leverage an organization's existing workflows where users or application platforms authenticate with a pre-existing provider (LDAP, Okta, Amazon IAM, etc) to obtain Nomad tokens. This saves you from implementing a new authentication method, its roles, and mechanisms (i.e. Namespaces and Service Accounts) as required by Kubernetes.

Using the same example as before, a frontend application (task) wants to access a backend database. In the Nomad job spec, which defines the deployment of the frontend application, users just need to add two additional blocks:

Vault stanza: The Vault policy access-db allows it to retrieve credentials to access the database. Nomad will automatically retrieve a Vault token for the task and handle token renewal for the task.

Template stanza: Similar to a Vault agent template in Kubernetes, the template stanza integrates with Vault to render dynamic secrets into configuration files. The same underlying tool being used is Consul template. But unlike Kubernetes, you don’t need additional Vault agent containers to run alongside the frontend application. In this example, the database credential that is provided by Vault will be encoded as a JSON string for the app to consume.

»Conclusion and Resources

The engineers who work on Vault and Kubernetes have created a number of excellent methods for integrating the technologies for a number of different architectures and workflows. However, the process is even simpler for Vault and Nomad.

Nomad's focus on only orchestration and open integration for other capabilities allows you to easily add Vault or another secrets management solution into your deployment workflow. The fact that Nomad and Vault are developed by the same company with common principles gives extra depth and simplicity to their integration, as illustrated above. Additionally, Nomad and Vault's flexibility allow you to manage jobs and their secrets across a heterogeneous environment given Nomad’s focus on diverse workload orchestration vs Kubernetes’ container-focused approach.

To learn more about how to integrate Nomad or Kubernetes with Vault, read our tutorials on HashiCorp Learn:

• Integrate Nomad with Vault
• Integrate Kubernetes with Vault