Chris van Meer is a HashiCorp Ambassador.

If you are running HashiCorp Nomad, chances are you also are dealing with secrets. Nomad can create and inject variables into your jobspecs at render time, but this can be limiting if you need to use those same secrets elsewhere from within your organization.

Here is where Vault comes into play. Since Nomad 0.5, users have been able to integrate Vault with Nomad to allow the Nomad servers to read and consume secrets from Vault and render them into Nomad jobspecs.

This post gives you an overview of the earlier integration method (which is now deprecated) and shows you a newer method for Nomad+Vault integration: Setting up workload identity using a manual or automated workflow. The automated workflow will use Terraform.

»The previous Vault integration

Setting up the old Vault integration with Nomad (the integration that’s been available since Nomad 0.5) is pretty straightforward, though sometimes there is friction in this process. It starts by creating a specific Vault token for the Nomad servers to include in their configuration.

You could either go with the “allowed policies” list approach, as shown here in a Vault token role…

…or you could go with the “disallowed policies” list approach, as shown here:

Note: If you’re using the allowed_policies, Nomad tasks may only request Vault policies that are in that list. If disallowed_policies is used, Nomad tasks may request any policy that is not in the disallowed_policies list. Generally, it is easier to use the denylist approach and just use the disallowed_policies list. Though that might not be your safest choice.

Then you would create the necessary token with the command below:

Since Nomad clients do not have to connect to Vault, the generated token would be placed on the Nomad servers within a Vault stanza alongside the name of the Vault token role in that same stanza. An example stanza is shown below:

Nomad then automatically renews its Vault token using the Vault token role and you have a working Vault integration. With this, you can now use your Vault secrets (whether static or dynamic) in your Nomad workloads.

You would use a vault block in your Nomad jobspec to allow Nomad to consume Vault secrets that are covered by the list of policies that is supplied. Here’s what that would look like with an example MongoDB policy:

This integration has several challenges related to provisioning, global cluster policies, and token vulnerability, which is why it was sunsetted starting in Nomad 1.10.

»The Vault + Nomad workload identity integration

Starting in Nomad 1.7, a new feature called Nomad workload identities was introduced. Clients can use a task's workload identity to authenticate to Vault and obtain a token specific to the task. When using Nomad workload identities, you no longer need to actively pass in a Vault token for Nomad to be able to consume Vault secrets. This makes provisioning Nomad way easier.

Workload identity allows Vault to verify Nomad-generated workload tokens using OIDC-style JWT tokens, signed by Nomad. Vault uses a JWT auth method to validate these tokens and issue scoped Vault tokens to tasks. You would need a Vault version that is equal or higher than 1.13 to be able to use this.

On the Nomad side, there are a few things to change in the server and client config files, plus a small change in the vault stanza within the Nomad jobspec where you will shift from using a policy argument to a role argument.

This tutorial walks you through enabling workload identity step-by-step through the Vault CLI first, so it makes more sense when we automate this.

»Vault

To enable workload identity, we start in Vault where you’ll enable the JWT auth method by entering this simple command:

Next, you’ll configure some basic settings for this auth method where you’ll specify the key set, its supported algorithms, and a default role. This will allow Nomad to consume secrets that are allowed by the Vault ACL policy, which is specified in the default role. The default role is used if there is no role argument defined in our Nomad’s vault stanza.

Here is the default role you will configure for this tutorial:

What this does is map some data to what Nomad sends out so that you can differentiate between certain levels (in this case the namespace, the job ID, and the task name), and you also specify the Vault ACL token that comes with this role

Again, this role acts as a generic way for Nomad to consume secrets where the Nomad jobspec does not have a specific role defined. So in a real use case, you would want to create a Vault ACL policy that either uses some form of identity injection and/or some secrets that should be available to all Nomad jobspecs that have no Vault role defined.

Below you will see an example using a combination of the two.

This example policy allows read capabilities on secrets that reside in the apps/data/<job_id>/ path. If, for instance, your Nomad job ID is traefik, then — without having to specify a Vault role in your Nomad jobspec — the Nomad task would be able to read all secrets in the path of apps/data/traefik/.

That might suit your needs but still be too generic for your workload. Then you have the option to create additional Vault roles to specify the needs of your workload. For example we will create a Vault role for a workload called mongodb-prod, which will only grant that specific workload to access secrets scoped by the token policies of that specific role.

Here we have a role that looks like the nomad-workloads role but has some extra settings in bound_claims. That section specifies that Vault will check on the given nomad_job_id and if that equals mondodb-prod and the nomad_namespace equals default, then allow the Nomad task to connect to Vault with an auto-generated token that only has the pol-mongodb-prod Vault ACL policy attached to it.

»Nomad

For Nomad, you just have to ensure that certain arguments are set in your Nomad configuration, both on Nomad servers and on Nomad clients. Templating this and provisioning it to the servers is much easier with workload identity compared to the sunsetted method for Vault integration.

»Nomad server

Ensure that the following arguments are set in your configuration file. By default, this would be /etc/nomad.d/nomad.hcl, but this could vary in your setup.

»Nomad client

For the clients, ensure that the following arguments are set in your configuration file:

After that, make sure you have reloaded or restarted your Nomad service and you’re good to go.

The only thing left to do is to amend our Nomad jobspecs. For workloads that can use the default role nomad-workloads, you would only have to instantiate the vault stanza within your jobspec like so:

vault {}

Think back on the mongodb-prod jobspec mentioned earlier. That has a specific Vault role and the only thing needed to migrate from the legacy Vault integration to workload identity is to change…

…to:

…and you are done. Now tasks coming from this specific nomad_job_id will be able to consume secrets that are allowed by the pol-mongodb-prod Vault ACL policy.

»Infrastructure as code

With all of this set, now it’s time to introduce Terraform into this workflow. You will use Terraform to automate the Vault parts of the workflow. If you want to change the configuration files of the Nomad servers and clients, you would probably hand that off to a configuration management tool like Ansible.

First, you’ll ensure you’re using the Terraform version of your choice plus the Vault provider version of your choice. Here’s the configuration for this tutorial:

Next, configure the Vault provider and point it in the right direction:

Next up are local values. For the ease of reading, these are composed below in several locals blocks. In real life, you would probably combine them into one locals block.

The first block defines the generic side information:

The second locals block is called nomad_special_roles, where you will specify a role name, a Nomad job ID, and a list of policies to be specified in the Vault role(s), which will be managed by Terraform and the actual policy content:

The third locals block transforms the data from local.nomad_special_roles:

The third locals block does the following.

1. It loops over each role in nomad_special_roles
2. For each policy inside a role, it:
   • Includes the policy only if it has a content field (i.e. policies with actual ACL rules)
3. For each such policy, it constructs a map where:
   • The key is the policy name (e.g. "pol-mongodb-prod")
   • The value is an object containing:
     • job_id: The job associated with the role (e.g. "mongodb-prod")
     • content: The ACL rules of the policy

And for our automation to work correctly, we need a more flattened version of the locals block mentioned above. Here is that version:

This block will:

1. Iterate over each role in local.special_role_policies
2. Iterate over each policy in each role
3. Construct a single-level map where
   1. The key is the policy name (e.g. "pol-mongodb-prod")
   2. The value is the corresponding object with job_id and content.

Note: the ... (three dots after the ]) is a Terraform splat operator for argument expansion. If you omit the ..., Terraform would try to merge a single list argument, which would throw an error. The ... expands the list of maps into individual map arguments for merge().

That’s it for the local values part. Next, you’ll call for some resources. First, ensure your JWT auth backend is set up:

Then set up your default backend role:

And the policy to go along with that:

Now you’ll start looping over our nomad_special_roles local value.

Then set up your Vault ACL policies if the content attribute is supplied:

Finally, run terraform apply. Now you have a dynamic way of managing the Vault side of the Nomad workload identity. Whenever you need another role, just add it to the list and let Terraform manage this for you.

As with every Terraform configuration, you can always further parameterize every argument, that is up to you at your discretion. The code in this tutorial was written to be as readable as possible.

»Recap

With this tutorial, you’ve learned how to make use of Vault for your Nomad secrets. Of course, your specific environment or use case may differ from the examples, but the core principles and best practices displayed are generic and can be applied among a broad variety of scenarios. This will both optimize your workflow by using automation and reduce your secrets-related security risks by replacing methods that are less safe and error prone.

Explore the Nomad and Vault documentation for more information about these products.