Two approaches to injecting variability into your Nomad batch job template without having to modify the template in the future.
By default, HashiCorp Nomad prevents duplicate batch jobs from executing. This is by design because duplicate job submissions could result in unnecessary work. From Nomad’s perspective, an unchanged job qualifies as a duplicate batch job.
However, there are times when a duplicate batch job or an unchanged job may be the correct approach. One example is a batch job that executes a calculation and outputs the results. In this scenario, it is likely that there is no need to change the Nomad job specification definition. Running the command nomad run
for the specific job would be the desired behavior. But, due to Nomad’s default behavior, this would result in the batch job placement failing.
To get around this default behavior, you can use a couple of techniques to inject variation in ways that don't require you to alter the job’s content. This blog presents two approaches to injecting variability into your Nomad batch job template without having to modify the template in the future.
The meta
block of a Nomad job specification allows for user-defined arbitrary key-value pairs. By using HCL2 functions and the meta
block, you can inject variation into a batch job without having to alter the job specification template. You can use the UUID function to inject variation and thus ensure the job is unique every time you run the command nomad run
.
To see how it works, create a file called uuid.nomad
and copy the content below into it. This batch job runs the Hello World
Docker example. Note how the meta
block is setting a key-value pair and using the uuidv4()
function:
Start a local Nomad server by issuing the command nomad agent -dev
:
Ensure the Nomad server is up and running. Then navigate to the directory where you created the file uuid.nomad
and issue the command nomad run uuid.nomad
. This will submit the batch job to Nomad:
Check the status of the job allocation by using the nomad alloc status
command:
The output indicates a successful job with an exit code 0
. Submit the job again though the command nomad run uuid.nomad
:
The job ran again and bypassed the default behavior due to having a different uuid
value. You can verify that the job ran twice through the Nomad UI by looking at the jobs overview, as shown here:
You can see in the Recent Allocations view that the two jobs ran successfully.
You can achieve the same behavior of injecting variability by utilizing the meta
block in a job specification and a variable.
Start by creating a file named variable.nomad
and copy the content below into the file. This batch does the exact same thing as the uuid.nomad
file, except this code snippet is using variables:
Go ahead and submit the batch job by running the command nomad run -var run_index=1 variable.nomad
:
Check the status of the job with the nomad alloc status
command:
The output reveals that the job completed successfully. If you were to submit the job again through the command nomad run -var run_index=1 variable.nomad
, the job allocation would have failed as the index value provided is the same as the previously submitted batch job. The screenshot below was taken after three submissions of the same batch job were submitted:
Three evaluations were conducted but only one batch job was allocated, the first one:
In order for Nomad to accept the job, you need to provide a unique value. Go ahead and change the index to a value of 2 and issue the command nomad run -var run_index=2 variable.nomad
:
This submission is accepted because it contains a unique value, an index value of 2. You can confirm the allocation was successful by visiting the Nomad UI or by running the command nomad alloc status
:
This post shared two approaches to injecting variability into your Nomad batch job template without having to modify the template in the future. There are many more Nomad tutorials available on the HashiCorp Learn Platform, where you can expand your Nomad knowledge and skills. Here are a few tutorials worth checking out this summer that will help you power up your Nomad skills.
Learn about the internals of Nomad's evaluation broker and how we recently reduced scheduler loads by 90% during rapid cluster changes.
HashiCorp Nomad 1.5, now GA, improves management of security and access with single sign-on and OIDC support, and adds dynamic node metadata, policy management in the UI, and more.
Before we ring in the new year, here’s a look back at some of the most important moments in 2022 for HashiCorp.