Skip to main content
HashiConf More sessions have been added to the conference agenda. Buy your pass and plan your schedule. Register

Nomad secrets consumption patterns: Vault integration

Learn how to consume application secrets in HashiCorp Nomad using HashiCorp Vault.

Nomad is great at orchestrations, but how do you ensure that your workloads can securely consume secrets? A Nomad-native option is Nomad variables which you can explore in a previous blog post. However, mature organizations ought to be using a centralized secrets management platform, so Nomad should harness that platform rather than its native secrets management, which is meant for onramping and smaller scale organizations.

This blog post will look at using Nomad’s native integration with HashiCorp Vault to securely centralize and consume secrets.

»Why Vault?

Vault is a secrets management platform responsible for storing static long-lived secrets, brokering short-lived ephemeral identities and credentials, and providing a full suite of cryptographic services such as encryption, decryption, cryptographic signing, signature verification, and HMACing data. These capabilities make Vault a great fit for providing centralized secrets to Nomad jobs.

The native integration between Nomad and Vault enables Nomad jobs to retrieve secrets from Vault at runtime without exposing them in the jobspec or application code.

Nomad Vault integration

The above diagram shows the end-to-end workflow of how the integration works.

»Configuring Vault

To use this integration, there are some steps that need to be performed on Vault:

1. Create a Vault policy for Nomad to enable it to create and revoke child tokens. The policy below will give Nomad the required permissions in Vault:

# Allow creating tokens under "nomad-cluster" token role. The token role name
# should be updated if "nomad-cluster" is not used.
path "auth/token/create/nomad-cluster" {
  capabilities = ["update"]
}
 
# Allow looking up "nomad-cluster" token role. The token role name should be
# updated if "nomad-cluster" is not used.
path "auth/token/roles/nomad-cluster" {
  capabilities = ["read"]
}
 
# Allow looking up the token passed to Nomad to validate # the token has the
# proper capabilities. This is provided by the "default" policy.
path "auth/token/lookup-self" {
  capabilities = ["read"]
}
 
# Allow looking up incoming tokens to validate they have permissions to access
# the tokens they are requesting. This is only required if
# `allow_unauthenticated` is set to false.
path "auth/token/lookup" {
  capabilities = ["update"]
}
 
# Allow revoking tokens that should no longer exist. This allows revoking
# tokens for dead tasks.
path "auth/token/revoke-accessor" {
  capabilities = ["update"]
}
 
# Allow checking the capabilities of our own token. This is used to validate the
# token upon startup.
path "sys/capabilities-self" {
  capabilities = ["update"]
}
 
# Allow our own token to be renewed.
path "auth/token/renew-self" {
  capabilities = ["update"]
}
 

This can be written to Vault using the following command:

vault policy write nomad-server nomad-server-policy.hcl

2. Create a role in Vault for the policy created above. Nomad will use this role. Here’s an example configuration:

{
  "allowed_policies": "hashicups-api",
  "token_explicit_max_ttl": 0,
  "name": "nomad-cluster",
  "orphan": true,
  "token_period": 259200,
  "renewable": true
}

This role will only allow tokens to be created with the hashicups-api policy attached. To allow all policies to be created, change the allowed_policies configuration to disallowed_policies and explicitly set which policies should not be allowed.

Write this role config to Vault using the following command:

vault write /auth/token/roles/nomad-cluster @nomad-cluster-role.json

3. Create the hashicups-api policy that the Nomad job will need to get the HashiCups API key from Vault:

path "secret/data/hashicups-api" {
  capabilities = ["read"]
}

Write this policy to Vault using the following command:

vault policy write hashicups-api hashicups-api-policy.hcl

4. Write the HashiCups API token to a KVv2 secret in Vault.

vault kv put /secret/hashicups-api api-key=the_api_key_for_hashicups \
  principal-id=a_random_string_identifying_an_entity

5. Create an orphan token for the Nomad server:

vault token create -policy nomad-server -period 72h -orphan

This step can be substituted with Nomad workload identity.

»Configuring Nomad

In your Nomad server configuration file, add the following stanza to enable the Vault integration:

vault {
  enabled = true
  address = "$VAULT_ADDR"
  task_token_ttl = "1h"
  create_from_role = "nomad-cluster"
  token = "$VAULT_TOKEN"
}

Ensure the $VAULT_ADDR and $VAULT_TOKEN environment variables are set. These values can be hard-coded, but the token would be exposed in plaintext, so this is not recommended.

Once this has been added, start the server in dev mode:

sudo nomad agent -dev \
     -config=nomad.hcl \
     -bind 0.0.0.0 \
     -network-interface='{{ GetDefaultInterfaces | attr "name" }}'

or start normally:

sudo nomad agent -server \
     -config=nomad.hcl

If you are not using dev mode, all Nomad clients will need the following stanza added to their configuration files:

vault {
  enabled = true
  address = "$VAULT_ADDR"
}

For more details on Nomad agent configuration, refer to the documentation.

»Consuming Vault secrets in Nomad jobs

This tutorial will use a demo application similar to the one used in Nomad secrets consumption patterns: Nomad variables. With some small tweaks to the UI, this prior application can be used to demonstrate Vault secrets consumption in a Nomad job.

The jobspec below will deploy the application to Nomad and configure it to use the secrets we have stored in Vault:

job "example" {
 
  datacenters = ["dc1"]
  type        = "service"
 
  group "example" {
 
    count = 1
 
    network {
      port "vs" {
        static = 14646
      }
    }
 
task "demo" {
     driver = "docker"
 
     template {
       destination = "${NOMAD_SECRETS_DIR}/env.vars"
       env         = true
       change_mode = "restart"
       data        = <<EOF
         SECRET_ONE = "{{with secret "secret/data/hashicups-api"}}{{index .Data.data "api-key"}}{{end}}"
         SECRET_TWO  = "{{with secret "secret/data/hashicups-api"}}{{index .Data.data "principal-id"}}{{end}}"
         EOF
     }
 
     config {
       image          = "devopsrob/nomad-vault-integration:1.1.0"
       ports          = ["vs"]
     }
 
     resources {
       cpu    = 500
       memory = 256
     }
   }
 }
}

The reference to the secrets uses the index in this case because the keys in the key-value pairs of the secret include hyphens -. If the keys do not include hyphens, the secret could be referenced like this: {{.Data.data.principal-id}}.

For more information on using the template stanza to get secrets from Vault at runtime, see the Vault integration documentation for more details.

When this job is running, you should be able to access the UI on http://localhost:14646.

»Summary

This post has walked through the steps required to enable the Vault integration and consume secrets in Nomad jobs. This pattern is a good fit for those already using Vault as a centralized secrets store. It also works well for applications that do not need to be Vault-aware.

This tutorial’s example uses a token to authenticate Nomad to Vault; however, for those that are not able to use a token due to internal security policies, this can be done using Nomad's workload identity. This consumption pattern works for both static secrets stored in the KV secrets engine and dynamic secrets from the various platform-specific secrets engines.


Sign up for the latest HashiCorp news

By submitting this form, you acknowledge and agree that HashiCorp will process your personal information in accordance with the Privacy Policy.