We are excited to announce the general availability of the Integrated Storage backend for Vault with support for production workloads. Integrated Storage exists as a purely Vault internal storage option and eliminates the need to manage a separate storage backend. Integrated Storage inherits a number of the benefits from the Consul storage backend and improves the user experience.
In this blog, we will discuss the capabilities of Integrated Storage, as well as its differences from Consul as a storage backend. We will also discuss the factors affecting the decision of whether or not to migrate to Integrated Storage (if using Consul), and provide references to relevant resources such as the new references architecture guide.
Integrated Storage is a Vault internal storage option that leverages the Raft consensus protocol to persist data to disk.
Here are a few feature highlights of Integrated Storage:
Integrated Storage eliminates the need to set-up, manage, and monitor a third-party storage system such as Consul, resulting in operational simplicity as well as lower infrastructure cost. When using Integrated Storage, troubleshooting Vault becomes much easier because there is only one system to investigate, whereas when running Vault in addition to an external backend storage system like Consul, you would have to debug two systems and possibly the network in between.
When comparing Integrated Storage to the Consul storage backend, Integrated Storage provides better network performance because there is no additional network hop to Consul. There are differences in how system resources are consumed too. For example, Integrated Storage writes updates to the disk which allows Vault's dataset to not be bound by the amount of RAM on the host, whereas Consul loads the entire data set into RAM. With Integrated Storage, data is on-disk and bound by disk I/O (SSDs are recommended), which results in an extra disk write compared to Consul. Due to these differences, each storage option has its own reference architectures, suggested system requirements (machine specifications, SSDs, network requirements, etc.), performance characteristics, and data inspection methods.
As you can see, the Integrated Storage backend offers many improvements over a non-integrated backend. Still, there are likely to be some operational and performance differences between Integrated Storage and your current backend. Given that, we highly recommend that if you are running in production today, and interested in migrating your storage backend, you create a test environment and explore the Integrated Storage backend with a workload similar to your production environment. The best way to gauge performance is to benchmark in your own environment using your workloads.
We have created Learning Guides to assist you here:
As already mentioned, Integrated Storage is an additional storage option made available in Vault 1.4. However, we continue to support Consul as a storage backend in production for our Vault Enterprise users. The decision on whether to migrate from Consul, or another existing storage backend, to the Integrated Storage backend is up to you and your operational requirements. In order to make this decision, it is important to understand the differences between using the Integrated Storage backend versus using an external storage backend. We recommend you start with the Preflight Checklist.
Once you familiarize yourself with all the information in the preflight checklist, should you choose to migrate from Consul to Integrated Storage, please review the Storage Migration Guide for Consul to Integrated Storage, that provides migration steps using the vault operator migrate CLI command.
To learn more about Integrated Storage, please visit the Integrated Storage documentation, or these helpful learn guides:
Also, if you enjoy playing around with this type of stuff, maybe you’d be interested in working at HashiCorp too since we’re hiring!
Use Vault-generated dynamic credentials to provision infrastructure. Learn how to inject secrets into your Terraform configuration using the Vault provider.
Get Terraform, Packer, Vault, Consul, and Nomad up and running even faster on macOS with our new official Homebrew Tap.
Learn about a Vault SlackBot made by DigitalOnUs.