Running HashiCorp with HashiCorp

Reducing time, effort, and mental overhead

Many of our workflows and processes have been around as long as the company — and in some cases, longer than even some of our own products. Out of necessity and expedience for serving evolving customer needs, we opted for established third-party tools to support everyday operations. But as our company grew rapidly and the volume and complexity of our customers’ needs evolved, the team recognized the benefits of making a clean break from third-party tools and consolidating to the HashiCorp solutions that large organizations around the world rely on instead.

With the HashiCorp stack — Terraform, Consul, Nomad, and Vault specifically — the team has a comprehensive end-to-end toolset that enables them to stand up infrastructure, launch and connect microservices, and manage secrets across both on-premises and AWS cloud environments. Each part natively interoperates with the other parts, making it a much smoother, integrated, efficient workflow.

“Replacing our third-party tools with the HashiCorp stack helped us standardize the entire environment to give us better control and agility across the board,” says JP Toto, engineering manager at HashiCorp. “After a methodical initial deployment and configuration, we now have the ability to automate a range of previously manual secrets-related operations while also monitoring our various server clusters in real-time to quickly respond to and resolve potential issues without interfering with our users’ experiences.”

Slow and steady wins the race

The team took a deliberate and modular approach to their transition away from third-party tools, letting them learn best practices as they went along. They started migrating extant systems with fewer business-critical applications to run using the HashiCorp stack bit by bit, allowing them to gain operational experience in production.

The transition started modestly, deploying Vault to handle authentication for the internal stack instances. Then they expanded to enable easy scale-out and scale-in instances without the time-consuming manual credential bootstrapping required by the former secrets management solution.

“Vault gives us the flexibility and fine-grained secrets management control to grant temporary credentials for somebody to audit a database or system, automate secrets updates, or manage batches of secrets however we want or need,” Toto says. “For example, if secrets are inadvertently leaked somehow, we know they’re just going to expire anyway and we can invalidate all of them in a batch, which brings us much closer to the zero trust environment we’ve been aiming to create for years.”

After a successful run with Vault, the team determined it had also worked out a decent pattern for managing the underlying infrastructure and configuration using Terraform and Packer. They leveraged that knowledge to relatively quickly stand up Consul and Nomad clusters alongside Vault.

“We started with simple recurring background tasks in Nomad, doing things like retrieving application credentials from Vault and then expanded that into low-priority and internally facing web applications,” Hogan explains. “It allowed us to gain experience enabling HTTP ingress for Nomad jobs as well as monitoring and operating it day-to-day that we could apply when we started deploying higher-value, more important web applications later on.”

Hogan says that the runtime environment created with Nomad or Consul makes it easier than ever to run experiments or pre-production services at any time because if parts of the clusters break, they’ll come back online without interrupting service. It gives the team peace of mind that they can add new services in an environment the security team has validated for them.

“The fact we can have our Nomad or Consul clusters sit for two weeks and not worry about them breaking because they come back online without interrupting services, means we can have Nomad and Consul available to fulfill any sort of runtime networking,” he explains. “Now, we can ship a change, have it automatically staged, then deploy and redeploy every instance in the entire stack with all of our applications migrating seamlessly across nodes, without any downtime.”

A new standard for today and tomorrow

Adopting the HashiCorp stack has fundamentally reshaped the way our internal teams operate. Moving to using HashiCorp’s own products replaced the hodgepodge of third-party solutions, while offering a greater number of features and functionality. At the same time, the move has significantly reduced the amount of time and effort — and cost — of onboarding new teams and their workloads, since they’re already familiar with the tools used in customer deployments.

With the HashiCorp stack, the team also has more granular control over every aspect of the hosted environment. Already using Terraform to create infrastructure, Toto says that it took just six months for the group to go from having no direct usage of Consul, Nomad, or Vault to the full HashiCorp stack being their default choice of deployment platform. More importantly, he says that even with just two engineers involved at any given time, the HashiCorp stack has enabled the team to make material improvements to their workflows that have boosted productivity, efficiency, and overall skill development.

“Using the HashiCorp stack gives us a consistent deployment platform that allows for a more dedicated focus on improving developer and operator experiences,” he says. “We’ve virtually eliminated manual management of entire classes of credentials like AWS access keys. We’ve replaced other manual infrastructure management with automated scaling in Nomad for horizontal node scaling and dynamic application sizing. With Consul, we automatically connect the microservices we’ve deployed.”

Overall, both Toto and Hogan agree that opting to use the HashiCorp stack has significantly increased the team’s work capacity. It also delivers a better engineering experience due to its greater flexibility and a lack of undesirable compromises they were forced to make in the past. “I’ve been in lots of other situations where a transition like this would have been a pipe dream that wasn’t achievable in reality,” says Hogan. “As an engineer first, it’s been one of the most joyful things to work on.”