Backend batch processing with Nomad
How to move an application from the traditional processing jobs out of a queue on multiple VMs to scheduling them as container jobs in Nomad.
At Graymeta, we index large filesystems by extracting metadata and leveraging machine learning APIs to process files. Wanting to benefit from the security of running our indexing process in a container, reduce our code’s complexity that was scheduling jobs to run on specific nodes, consolidate infrastructure by treating all back-end batch processing machines as a pool of resources to scale up and down as a single unit, and several other reasons, we began looking for an easy to operate container orchestration layer. For us, it has to be dead simple to operate as we have to be able to run it in the cloud and behind corporate firewalls in on-premises deployments. We chose Nomad as it was the best fit for our requirements.
In this talk, Jason Hancock, Operations Engineer at Graymeta, will dive into the how of moving our application from the traditional processing jobs out of a queue on multiple VMs to processing the same jobs out of the same queue as before, but scheduling them as container jobs in Nomad. It will also touch on our local development environments that each developer runs on their laptop where we run Nomad inside a Docker container and use the raw_exec driver to mimic what we’re doing in production. As a final point, I will discuss the increased portability and flexibility that moving to Nomad has afforded us, specifically in terms of being able to abstract the container scheduling and use a variety of different schedulers in different environments (for example, ECS in EC2).