Using HashiCorp's Vault with Chef

Using HashiCorp's Vault with Chef

Feb 02 2016 Seth Vargo

One common challenge organizations face when integrating Vault by HashiCorp in their infrastructure is how to fetch secrets from Vault using a configuration management tool. This blog post details a few techniques for retrieving secrets from Vault using Chef, but the topics can be broadly applied to any configuration management software such as Puppet or Ansible.


For each of the following techniques, we will look at two different secret-retrieval scenarios with Vault. The first scenario involves retrieving a static secret from the Vault generic secret backend. Here is some sample data from the generic secret backend:

$ vault read secret/vpn Key Value lease_duration 2592000 password b@c0n username sethvargo

Vault's generic secret backend can be thought of as an encrypted key-value store. The vault read command above shows how the data can be structured. We can reference these named pairs when retrieving the secret. Generic secrets do not expire, but they can be revoked at any time.

The second scenario involves retrieving dynamic PostgreSQL credentials from Vault using the Vault PostgreSQL secret backend. Vault is configured with the root credentials to Postgres and then generates unique, leased credentials using those root credentials. Once configured, we can generate new credential sets like this:

$ vault read postgresql/creds/readonly Key Value lease_id postgresql/creds/readonly/c888a097-b0e2-26a8-b306-fc7c84b98f07 lease_duration 3600 password 34205e88-0de1-68b7-6267-72d8e32c5d3d username root-1430162075-7887

When we read this path, Vault will dynamically generate a new set of credentials for connecting to Postgres using the given SQL. The returned credentials have an associated lease, and it is the application's responsibility to renew the credentials before the lease expires or Vault will revoke them. The recommended pattern for retrieving and renewing secrets with Vault is as follows:

HashiCorp Vault secret acquisition and renewal flow

Method 1: Reading Secrets at Runtime

Our recommended approach to use Vault with any configuration manage tool is to move the secret retrieval and renewal into a runtime process instead of a build time process. This is especially useful if you are using Chef and Packer to build artifacts or if you do not run Chef on a regular interval. In this case, traditional configuration management is used to install a tool that manages interaction with Vault at runtime – Consul Template.

Consul Template is a single static binary that can retrieve data from Consul, Vault, or both. Despite unfortunate naming, Consul Template does not require you are using Consul. Provide Consul Template with an input template, output destination, and an optional arbitrary command to execute when the template contents are changed. Prepare for a little bit of "meta", because we are going to use a Chef template to render a Consul Template template that will start as a runtime process to render our application's configuration.

HashiCorp Vault with Consul Template and Chef

Chef's responsibility:

  • Install Consul Template binary

  • Add Consul Template upstart configuration

Consul Template's responsibility:

  • Retrieve secrets from Vault

  • Write secrets to disk and manage application configuration lifecycle

  • Handle secret lease renewal

This post will only include relevant snippets, but a complete recipe and supporting files is available as a GitHub gist for using Vault with Chef. The Chef recipe downloads Consul Template from the HashiCorp releases service, unzips and extracts it into /usr/local/bin, creates the "dot-d" directory for our Consul Template configurations, and configures upstart to manage the service.

Now that Consul Template is installed and running as a service, we need to create our Consul Template template which will render the application configuration. Since this is a fictitious application, let's assume our application accepts a configuration file like this:

[config] username = "..." password = "..."

We need to populate the values in ... with secrets from Vault. Here is how we do that using Consul Template's templating language

template "/etc/vpn/config" do source "my-vpn-config.erb" owner "vpn" group "vpn" mode "0600" sensitive true variables( :username =>[:username], :password =>[:password], ) end

In this example, the Chef run queries the Vault server for the value at "secret/vpn" and passes the values at "username" and "password" as variables into the template.


  • Very "Chef-like"

  • Minimal barrier to entry

  • Works very similar to data bags or encrypted data bags


template "/opt/my-app/config" do source "my-app.conf" owner "my-app" group "my-app" mode "0600" sensitive true variables( :db_username =>[:username], :db_password =>[:password], ) end

Assuming you are properly authenticated to Vault with permissions to read from that path, the Chef run will complete successfully. However, Chef does not have a mechanism to renew these secrets. Unlike the generic secret backend, the dynamic backends hold a lease that must be renewed before the TTL expires. If the lease for the secret is not renewed before the lease expires, Vault will revoke the secret. Even if the Vault is offline, the credentials were created using Postgres' VALID UNTIL clause, so the database will also revoke the credentials on our behalf without a renewal. This is a difficult bug to try and diagnose, because it will only occur when you have not run Chef in the scheduled interval before the lease expires. If you are not running Chef on a regular interval (maybe you are using Chef with Packer to build AMIs or Docker containers), the secrets will likely expire before you launch the application. This is why we consider secrets to be "runtime" configuration, not "build time" configuration.

Even if you execute Chef on a regular interval, each Chef run will create a new set of credentials. This means the application will receive a fresh set of Postgres credentials on each iteration. Not only will this create many unused secret entries in Vault and Postgres, but it could also complicate diagnosis via Vault's audit log in the event of a compromise. While these entries will be cleaned up by Vault, the application will continuously receive a new set of credentials and Chef will restart the application.

Method 3: Custom Resource and Provider

vault_secret "postgresql/creds/readonly" do notifies :create, "template[/config]", :immediately end

template "/config" do source "my-app.conf" owner "my-app" group "my-app" mode "0600" sensitive true variables lazy { { :username => node.run_state["postgresql/creds/readonly"].data[:username], :password => node.run_state["postgresql/creds/readonly"].data[:password], } } action :nothing end

First, you will notice there is a new resource, vault_secret, that accepts a path as the name argument. This resource notifies our template to create (or update if it already exists). The template remains relatively unchanged save for addition of the action :nothing and a slight change to the way secrets are retrieved. Of note is the use of lazy - this is important because the node's run state is not populated until after the secret is retrieved. The full LWRP code can be found in this Gist.

This custom resource fires notifications when a read or renew of a Vault secret occurs. A read should only occur if the secret did not already exist or if the secret lease renewal failed. In both of these scenarios, the template needs to be re-rendered. However, instead of persisting the entire secret into the node object, this LWRP pushes the secret management responsibility into the notified resource. In the case of our template, once the secrets are written to the configuration file, they are no longer needed by Chef. As such, we do not need to store the raw secret.

The LWRP uses the node's run_state to store temporary data for the duration of this Chef run. The secrets only persist in memory and are never stored on the node object that is persisted back to a Chef Server. Because the data is not stored on the run state until after this LWRP executes, we need to use Chef's lazy when setting the template variables.

Lastly, you may have noticed that we are no longer relying on the recipe to provide the path to the lease_id. The LWRP does accepts a property called destination, but the default value is the same as path. If specified, the destination field determines the path where the lease is stored on the node object and path on the node's run state where secrets are stored in memory.


  • Still very Chef-like

  • Does not store secrets on the node object

  • Flexible and extensible to multiple patterns

  • Can notify multiple resources if they share the same secrets


  • Requires custom resource

  • Requires Chef Client run on an interval less than the lowest lease

Method 4: Directly Integrate into Application

The last approach to using Vault with Chef is to bypass configuration management entirely, and add support for Vault directly into your application. Instead of relying on your configuration management tool or Consul Template to communicate with Vault, your application will communicate directly with Vault. This means your application needs to be responsible for retrieving secrets and renewing leases. Since this technique is dependent upon the application's needs, an example is not provided.

Your browser is out-of-date!

Update your browser to view this website correctly. Update my browser now