Nomad secrets consumption patterns: Vault integration
Learn how to consume application secrets in HashiCorp Nomad using HashiCorp Vault.
Nomad is great at orchestrations, but how do you ensure that your workloads can securely consume secrets? A Nomad-native option is Nomad variables which you can explore in a previous blog post. However, mature organizations ought to be using a centralized secrets management platform, so Nomad should harness that platform rather than its native secrets management, which is meant for onramping and smaller scale organizations.
This blog post will look at using Nomad’s native integration with HashiCorp Vault to securely centralize and consume secrets.
»Why Vault?
Vault is a secrets management platform responsible for storing static long-lived secrets, brokering short-lived ephemeral identities and credentials, and providing a full suite of cryptographic services such as encryption, decryption, cryptographic signing, signature verification, and HMACing data. These capabilities make Vault a great fit for providing centralized secrets to Nomad jobs.
The native integration between Nomad and Vault enables Nomad jobs to retrieve secrets from Vault at runtime without exposing them in the jobspec or application code.
The above diagram shows the end-to-end workflow of how the integration works.
»Configuring Vault
To use this integration, there are some steps that need to be performed on Vault:
1. Create a Vault policy for Nomad to enable it to create and revoke child tokens. The policy below will give Nomad the required permissions in Vault:
# Allow creating tokens under "nomad-cluster" token role. The token role name
# should be updated if "nomad-cluster" is not used.
path "auth/token/create/nomad-cluster" {
capabilities = ["update"]
}
# Allow looking up "nomad-cluster" token role. The token role name should be
# updated if "nomad-cluster" is not used.
path "auth/token/roles/nomad-cluster" {
capabilities = ["read"]
}
# Allow looking up the token passed to Nomad to validate # the token has the
# proper capabilities. This is provided by the "default" policy.
path "auth/token/lookup-self" {
capabilities = ["read"]
}
# Allow looking up incoming tokens to validate they have permissions to access
# the tokens they are requesting. This is only required if
# `allow_unauthenticated` is set to false.
path "auth/token/lookup" {
capabilities = ["update"]
}
# Allow revoking tokens that should no longer exist. This allows revoking
# tokens for dead tasks.
path "auth/token/revoke-accessor" {
capabilities = ["update"]
}
# Allow checking the capabilities of our own token. This is used to validate the
# token upon startup.
path "sys/capabilities-self" {
capabilities = ["update"]
}
# Allow our own token to be renewed.
path "auth/token/renew-self" {
capabilities = ["update"]
}
This can be written to Vault using the following command:
vault policy write nomad-server nomad-server-policy.hcl
2. Create a role in Vault for the policy created above. Nomad will use this role. Here’s an example configuration:
{
"allowed_policies": "hashicups-api",
"token_explicit_max_ttl": 0,
"name": "nomad-cluster",
"orphan": true,
"token_period": 259200,
"renewable": true
}
This role will only allow tokens to be created with the hashicups-api policy attached. To allow all policies to be created, change the allowed_policies configuration to disallowed_policies and explicitly set which policies should not be allowed.
Write this role config to Vault using the following command:
vault write /auth/token/roles/nomad-cluster @nomad-cluster-role.json
3. Create the hashicups-api policy that the Nomad job will need to get the HashiCups API key from Vault:
path "secret/data/hashicups-api" {
capabilities = ["read"]
}
Write this policy to Vault using the following command:
vault policy write hashicups-api hashicups-api-policy.hcl
4. Write the HashiCups API token to a KVv2 secret in Vault.
vault kv put /secret/hashicups-api api-key=the_api_key_for_hashicups \
principal-id=a_random_string_identifying_an_entity
5. Create an orphan token for the Nomad server:
vault token create -policy nomad-server -period 72h -orphan
This step can be substituted with Nomad workload identity.
»Configuring Nomad
In your Nomad server configuration file, add the following stanza to enable the Vault integration:
vault {
enabled = true
address = "$VAULT_ADDR"
task_token_ttl = "1h"
create_from_role = "nomad-cluster"
token = "$VAULT_TOKEN"
}
Ensure the $VAULT_ADDR and $VAULT_TOKEN environment variables are set. These values can be hard-coded, but the token would be exposed in plaintext, so this is not recommended.
Once this has been added, start the server in dev mode:
sudo nomad agent -dev \
-config=nomad.hcl \
-bind 0.0.0.0 \
-network-interface='{{ GetDefaultInterfaces | attr "name" }}'
or start normally:
sudo nomad agent -server \
-config=nomad.hcl
If you are not using dev mode, all Nomad clients will need the following stanza added to their configuration files:
vault {
enabled = true
address = "$VAULT_ADDR"
}
For more details on Nomad agent configuration, refer to the documentation.
»Consuming Vault secrets in Nomad jobs
This tutorial will use a demo application similar to the one used in Nomad secrets consumption patterns: Nomad variables. With some small tweaks to the UI, this prior application can be used to demonstrate Vault secrets consumption in a Nomad job.
The jobspec below will deploy the application to Nomad and configure it to use the secrets we have stored in Vault:
job "example" {
datacenters = ["dc1"]
type = "service"
group "example" {
count = 1
network {
port "vs" {
static = 14646
}
}
task "demo" {
driver = "docker"
template {
destination = "${NOMAD_SECRETS_DIR}/env.vars"
env = true
change_mode = "restart"
data = <<EOF
SECRET_ONE = "{{with secret "secret/data/hashicups-api"}}{{index .Data.data "api-key"}}{{end}}"
SECRET_TWO = "{{with secret "secret/data/hashicups-api"}}{{index .Data.data "principal-id"}}{{end}}"
EOF
}
config {
image = "devopsrob/nomad-vault-integration:1.1.0"
ports = ["vs"]
}
resources {
cpu = 500
memory = 256
}
}
}
}
The reference to the secrets uses the index in this case because the keys in the key-value pairs of the secret include hyphens -. If the keys do not include hyphens, the secret could be referenced like this: {{.Data.data.principal-id}}.
For more information on using the template stanza to get secrets from Vault at runtime, see the Vault integration documentation for more details.
When this job is running, you should be able to access the UI on http://localhost:14646.
»Summary
This post has walked through the steps required to enable the Vault integration and consume secrets in Nomad jobs. This pattern is a good fit for those already using Vault as a centralized secrets store. It also works well for applications that do not need to be Vault-aware.
This tutorial’s example uses a token to authenticate Nomad to Vault; however, for those that are not able to use a token due to internal security policies, this can be done using Nomad's workload identity. This consumption pattern works for both static secrets stored in the KV secrets engine and dynamic secrets from the various platform-specific secrets engines.
Sign up for the latest HashiCorp news
More blog posts like this one
Solving secret zero with Vault and OpenShift Virtualization
Explore how you can use Red Hat OpenShift Virtualization and HashiCorp Vault to solve the secret zero problem for your virtualized infrastructure
Managing Ansible Automation Platform (AAP) credentials at scale with Vault
Learn how to automate SSH certificate retrieval and management through AAP, using Vault to issue signed SSH certificates on demand.
HCP Terraform introduces Hold Your Own Key (HYOK)
HCP Terraform customers can now gain greater control over access to secrets within Terraform artifacts such as state and plan files with Hold Your Own Key (HYOK).