Today we announce Vault — a tool for securely managing secrets and encrypting data in-transit. From storing credentials and API keys to encrypting passwords for user signups, Vault is meant to be a solution for all secret management needs.
A modern system requires access to a multitude of secrets: credentials for databases, API keys for external services, credentials for service-oriented architecture communication, etc. Understanding who is accessing what secrets is already very difficult and often platform-specific. Adding on key rolling, secure storage, and detailed audit logs is almost impossible without a custom solution. Vault solves all of these problems.
Vault is already being deployed in very large infrastructures. We are excited for the future of Vault, and what we have for you today is just the beginning of what we believe is an incredible tool.
Read on to learn more.
Vault is one giant leap forward for practical security in a cloud environment.Rob Witoff, Director at Coinbase
Vault is the most feature-rich initial release of any of our open source projects. Vault has basic features you'd expect from a secret manager, but also has some novel features that we believe will modernize the space and extend the boundary for operational excellence.
We'll go into detail about some of the major features later in this post, but first we'll enumerate all the major features of Vault:
Secure secret storage: Arbitrary key/value secrets can be stored in Vault. These secrets are encrypted prior to being written to persistent storage, so gaining access to the raw storage isn't enough to access your secrets. Vault can write to disk, Consul, and more.
Dynamic secrets: Vault can generate secrets on-demand for some systems, such as AWS or SQL databases. For example, when an application needs to access an S3 bucket, it asks Vault for credentials, and Vault will generate an AWS keypair with valid permissions on demand. After creating these dynamic secrets, Vault will also automatically revoke them after the lease is up.
Leasing and renewal: All secrets in Vault have a lease associated with it. At the end of the lease, Vault will automatically revoke that secret. Clients are able to renew leases via built-in APIs.
$ vault read secret/foo Key Value lease_id secret/foo/9c5f3cf1-1239-0160-4311-d6544fd1018c lease_duration 2592000 value bar
$ vault delete secret/foo Success! Deleted 'secret/foo'
Note: This and all other features of Vault are also possible through the complete HTTP API as well.
When you read a secret, Vault returns the data you wrote along with leasing information. Leasing is a core, critical part of Vault. Every secret in Vault must have a lease. The lease tells a consumer of a secret that the secret is guaranteed to remain valid for that lease period, but past the lease period, there is no guarantee the secret will still function. This forces clients to renew the lease periodically, where Vault can record this in the audit log and can use this opportunity to potentially deny lease renewal as well.
When writing secrets manually, revocation of secrets past the leasing period is also manual. However, Vault also supports dynamic secrets that have automatic revocation and are covered next.
In the modern world of API-driven everything, many systems also support programmatic creation of access credentials. Vault takes advantage of this support through a feature called dynamic secrets: secrets that are generated on-demand, and also support automatic revocation.
For Vault 0.1, Vault supports dynamically generating AWS, SQL, and Consul credentials.
The power of this feature is immense: credentials to access these systems no longer ever need to be written to disk. They can be accessed from Vault, stored in-memory, and automatically revoked when they're no longer used. If a specific application instance is compromised, that single key can be revoked, rather than some more global set of credentials.
The most revolutionary place we're seeing this feature used already is for SQL credentials. With dynamic SQL credentials, every application that needs database access is generated a new SQL user. For systems such as PostgreSQL, Vault even sets the VALID UNTIL field so PostgreSQL itself will revoke the user automatically.
Here is what reading a dynamic secret looks like:
$ vault read postgresql/creds/production Key Value lease_id postgresql/creds/production/8ade2cde-5081-e3b7-af1a-3b9fb070df66 lease_duration 3600 password 56b43bc3-b285-4803-abdf-662d6a105bd0 username vault-root-1430141210-1847
It looks just like reading a normal secret! The difference is that the returned username and password didn't exist prior to reading it, and in 3600 seconds (the lease duration shown), Vault will automatically delete that SQL user unless the lease is renewed.
Remember, all of this can be done from the API as well, and in the case of an application it probably would use the API so that the secret never has to be written to disk or copied anywhere else.
You can see from above that reading and writing secrets from Vault is easy and even enjoyable. But more important than this usability is ensuring that these secrets are secure. We took security very seriously and use industry best practices to secure Vault.
The data stored with Vault is encrypted using 256-bit AES in GCM mode with a randomly generated nonce. This encryption happens in-memory prior to ever being sent to the backend storage driver. This way the storage never sees the unencrypted value.
The key used to encrypt the data is also encrypted using 256-bit AES in GCM mode. This is known as the master key. The encrypted encryption key is stored in the backend storage. The master key is then split using Shamir's Secret Sharing. Shamir's Secret Sharing ensures that no single person (including Vault) has the ability to decrypt the data. To decrypt the data, a threshold number of keys (by default three, but configurable) are required to unseal the Vault. These three keys are expected to be with three different individuals.
With the Vault unsealed, all API calls are done with HTTP over TLS. While it is possible to disable TLS, it requires explicitly opting into it both on the server as well as all clients.
All API calls to Vault require an identity obtained through authentication. This identity is mapped to various metadata. The identity and metadata are logged with every audit log entry. For example, if you use GitHub for authentication, the GitHub username and organization of the user is in the audit logs. If you use certificate authentication, the fingerprint of the client certificate is logged.
Within Vault, data is split into multiple backends. For example, when
you write data to
secret/foo, it is communicating with a different
secret backend than when you read a PostgreSQL credential from
postgresql/creds. Each backend is given a restricted view to the backend
data. The backend at
secret/foo can never access the data at
postgresql/creds, for example. This isn't just an ACL; the backends
themselves simply do not have a way to address data from other backends.
This ensures that even within Vault there is protection against malicious
In addition to all of the above, Vault was in closed beta for over 6 weeks with over 50 individuals and corporations. We reached out specifically to security teams at companies as well as individuals with an interest/background in security to help review Vault prior to release. We've gotten basic approval from multiple companies and are confident that Vault has a strong foundation as we move forward.
At HashiCorp, we build solutions to DevOps problems that are technically sound and are a joy to use. We don't take shortcuts with the technologies we choose, and just as importantly we don't take shortcuts in the experience of using and operating our tools. As a result, HashiCorp-made tools are stable, scalable, and easy to use and operate.
Vault is the sixth such tool we've built. We've also built Vagrant, Packer, Serf, Consul, and Terraform. Vault works great with these other tools, but doesn't require any of them. We have plans to integrate Vault more closely into some of our other tools as well, such as automatically retrieving AWS credentials for Packer via Vault.