Nomad secrets consumption patterns: Nomad variables

Nomad is an excellent choice for scheduling your workloads, but what if your workloads need to consume secrets to perform their function? This post will explore Nomad variables — one of four different secret consumption patterns in Nomad — and explain when to use it so that secrets don’t leak into job files.

»Nomad variables

The Nomad variables feature is built on top of a configuration store that encrypts all stored values by default. These values can be accessed by all jobs in the scope of the variable. A Nomad variable is a collection of one or more key/value pairs that should be made available to a job.

»Namespaces and scopes

Nomad namespaces are a way to separate jobs from each other, and from other users of Nomad clusters. Every Nomad variable belongs to a namespace, which means rogue jobs or jobs from other namespaces cannot access them. To further secure sensitive values, Nomad ACLs can be used to provide granular access controls.

Access to variables can also be limited when multiple jobs exist in the same namespace. For example, a namespace could have job A and job B, but only job A should have access to variable A.

The nature of how Nomad variables are configured makes it easy to achieve this. Each Nomad variable address is path-based and can be written for a specific job, group, task, or all jobs in the namespace. The diagram below shows how the variable address specifies which tasks, task groups, and jobs have access to each variable.

The table below provides more details about the above example and illustrates the access patterns to the Nomad variables above in a table format.

Here is a variable address path example that makes its variable accessible to a task called example-task, within the example-group task group, which is within the example-job job:

nomad/jobs/example-job/example-group/example-task

If the example-group task group has multiple tasks, only example-task will be in scope to consume the variable. Should multiple tasks within the example-group task group be required to consume a variable, the variable address would be:

nomad/jobs/example-job/example-group

Should all jobs in the namespace be required to consume a variable, the address would be:

nomad/jobs

»Creating Nomad variables

As mentioned before, Nomad variables each belong to a namespace, so the first step is to create one. The code snippet below is a namespace specification saved to a file called example-namespace.hcl:

This file is a high level specification for a namespace called example. To apply this to Nomad, use the namespace apply command:

nomad namespace apply example-namespace.hcl

Now that the namespace is created, the Nomad variable can be created within it. There are a few ways to do this through the CLI or API:

1. Using inline configuration values (Best practice)
2. Using a Nomad variable specification file

This command will create a variable for all jobs in the example namespace:

nomad var put -force \
    -namespace example \
    nomad/jobs \
    secret_author=devopsrob \
    role=developer

This is the recommended method; however, if you require a config driven approach, the code below is an example specification file that creates the same variable (spec.nv.hcl):

To apply this, use the var put command below:

nomad var put spec.nv.hcl

While the specification file is a great fit for general purpose configuration values and works well in GitOps workflows, secrets being stored in plaintext in a file is not recommended.

»Consuming Nomad variables

Now that the variable has been created, a job file can now be written to consume it. The common way to consume these variables is to take the twelve-factor app approach, which is to use environment variables.

We have a demo application that reads some environment variables and outputs them to a web UI. The full application code can be viewed here but the main section of the code that matters for this demo is shown below:

This function reads two environment variables, SECRET_ONE and SECRET_TWO. The application has been packaged into a Docker image.

The Nomad job file below can be used to deploy this application and read the secrets from Nomad variables as environment variables.

The magic happens within the template stanza of the job file. The definition specifies that the output should be rendered as environment variables in an env.vars file in the Nomad secrets directory. This is a built-in directory that Nomad exposes to jobs via the $NOMAD_SECRETS_DIR runtime environment variable.

The data argument in the template stanza references the Nomad variable and assigns the value of each specified key to an environment variable. In this example, SECRET_ONE is assigned the value of the secret_author key/value pair within the variable.

The change mode is set to restart, which means if any changes are detected in the Nomad variable, the job will be restarted so it can obtain any new configuration values.

The running job should display the page below at http://localhost:14646 with confetti appearing every time the Nomad variable values change:

»Securing Nomad variables encryption

Although Nomad variables are encrypted by default, the Key Encryption Key (KEK) is stored on the Nomad server in plaintext. This presents an extra attack vector that must be mitigated for production environments.

Nomad supports KEK wrapping via external KMS systems, like HashiCorp Vault, Azure Key Vault, AWS KMS, and GCP Cloud KMS.

This workflow uses specialist external systems to further wrap and unwrap the root encryption key to perform cryptographic operations. Not only would this add a layer of protection for Nomad variables, it would also enhance the protection of other operations that rely on the keyring, such as signing JWTs for Nomad workload identities and OIDC client assertions.

»Summary

Nomad variables can be a simple yet powerful way to store small amounts of application configuration code, including secrets. Authoring applications to consume configuration and secrets is easily achieved using the twelve-factor app principles.

Part 2 of this blog series will look at how Vault can be integrated into Nomad and how to consume its secrets.