HashiCorp Vault lost secrets recovery, explained

Secret recovery is a new feature that was recently introduced in Vault Enterprise 1.20 that enables customers to delegate recovery of specific secrets from a cluster snapshot without impacting the Vault cluster. At launch, this feature is available in the KV v1 secrets engine with support in more secrets engines planned.

This blog post will explain:

• The problem space that led to this feature
• How it works
• The choices we made in its development

»The problem

Before secret recovery, when a secret was modified or lost, the entire cluster snapshot had to be restored to recover it. This could cause degradation elsewhere in the cluster.

Restoring from the entire cluster snapshot also had the potential to overwrite legitimate changes that had been made since the snapshot was taken. For example, if a secret was modified in error 3 minutes ago, and the cluster was restored from a cluster snapshot taken 10 minutes ago, any secrets that were legitimately modified in the 10 minute time period could be lost.

This cluster restore process also ran into bottlenecks because only Vault operators with sufficiently privileged permissions could remediate secret loss or modification for all users of their cluster.

»How Vault secret recovery works

Secret recovery uses an existing feature — the cluster snapshot — in a new way to allow granular secret recovery permissions, which can be delegated to different areas of Vault using the flexible Vault policy system.

Rather than recovering the cluster from the cluster snapshot, a snapshot is loaded into Vault. This can either be done manually or by using the pre-existing external storage configured in the automatic snapshot config. For example, if your automatic snapshot config is sending the snapshot to AWS S3, anyone with the update snapshot-load permission on the relevant snapshot path can load the snapshot from S3 into Vault and make it available for recovery functions by other users.

The initial iteration of the secret recovery feature does not allow users to configure the automatic snapshot to make it automatically available for recovery. However, if you are interested in this capability as part of the secret recovery feature, please reach out to your account representative.

»Additional visibility features

In addition to providing snapshot loading to support recovery, the secret recovery feature has a few other capabilities that will benefit customers.

Once a snapshot is loaded, users can “read” and “list” items in the snapshot by using the snapshot ID.

This provides a level of visibility into the snapshot that previously required restoring the snapshot to a second cluster and manually assessing it.

»Current secrets engine support and plans for future expansion

While the initial launch of secret recovery only encompasses the KV version 1 secrets engine, we intend to launch additional secrets engine recovery capabilities and are actively soliciting feedback on future expansion opportunities for secret recovery. If you have any feedback, please reach out to your account representative.

While we are considering most secrets engines as candidates for secret recovery, our goal for secret recovery is to provide a general, delegatable recovery function that will apply across high usage secrets engines.

The exception to this is the KV version 2 secrets engine, which already provides soft deletes of key value data. This provides the same capabilities (as well as some other benefits), but is limited to the KV v2 secrets engine. We generally recommend that customers upgrade to the KV v2 secrets engine.

»Path-based secret recovery vs. namespace snapshot and recovery

We considered implementing a namespace snapshot and restore mechanism. This would essentially be taking a “lower level” snapshot that encompasses all the information in the namespace and does not include information elsewhere in the cluster. However, there were several drawbacks to this approach:

1. Just as with the cluster snapshot restoration, legitimate changes might be lost, albeit at a smaller scale.
2. Just as with the cluster snapshot restoration, there might be degradation that impacted other users of the namespace, albeit at a smaller scale since it would be localized to the namespace rather than the whole cluster.

Additionally, the situations many customers described to us encompassed a need to recover only a single secret. In this case, recovering an entire namespace would be overly broad even without the additional detriments. Recovering a single secret is also a more efficient use of computing resources.

While cluster snapshot restoration and the hypothetical namespace snapshot restoration both require high level permissions, secret recovery permissions can be provided granularly based on the specific secrets a user interacts with. This aligns with least-privilege principles and allows the delegation of recovery permissions to end users, which reduces the burden on Vault operators.

We are continuing to assess namespace-based recovery mechanisms, including namespace soft delete and namespace snapshot and recovery. If this use case would be of interest to you, please reach out to your account representative — we would love to hear from you.

»Terraform Vault provider recommended for secret recovery configuration

All features discussed in this blog are configurable through the Terraform Vault provider. We encourage all customers to codify their Vault configurations using the Vault provider so they can benefit from code-based change control processes, easily deploy at scale across environments with reproducible deployments, and apply their organization’s coding quality practices.

»Get started with Vault Enterprise 1.20

To learn more about the improvements and new features released in Vault Enterprise 1.20, read our blog: Vault Enterprise 1.20: SCEP, usage reporting, cloud secret imports.

You can find additional resources here:

• Read the Vault 1.20 upgrade guide
• Sign up for a free trial of Vault Enterprise or HCP Vault Dedicated
• Get hands-on with Vault tutorials