This is a guest post by Matthew Lapworth, Senior Application Security Engineer at New Relic. New Relic is a leading digital intelligence company, delivering full-stack visibility and analytics with more than 14,000 paid business accounts. The New Relic Digital Intelligence Platform provides actionable insights to drive digital business results. Companies of all sizes trust New Relic to monitor application and infrastructure performance so they can quickly resolve issues, and improve digital customer experiences.
At New Relic, our systems and infrastructure had grown, and we were facing challenges with securely storing and managing credentials. HashiCorp Vault has provided us with a consistent approach to manage secrets and credentials.
Our platform had grown to the point where we integrated with numerous internal and third party systems. Each of our services needed dozens of credentials to function. Credential management became an obstacle, leading to credential storage practices that were inconsistent. The difficulty of managing credentials resulted in serious challenges for teams seeking to collaborate on a single code base.
Changing a credential used by a service involved coordinating with multiple teams, who each had a copy of the changed credential. When a security event occurred, changing one or more of these credentials was necessary. However, due to the infrequent nature of these changes, procedures were not well understood and led to frequent service outages caused by a team not being informed of the change or the inclusion of incomplete features and functionality in the change.
We utilize Vault to manage secrets and credentials and also two other HashiCorp tools for our complete secrets management solution. HashiCorp Consul stores and provides high availability for the secrets management system and HashiCorp Terraform controls the lifecycle of the secrets management cluster.
The initial use case of Vault is to house the numerous credentials used by the services that make up New Relic’s product suite. Each team is assigned a partition in the generic credential backend to store their service’s secrets. This approach allows the teams to decouple rotation of secrets from the deployment of their service and to store their secrets in a tightly access-controlled location.
Our architecture includes a Vault cluster for our integrations and another for our production environments, running in our production AWS account. This VPC is connected to our primary data center via Amazon Direct Connect. We have 5 hosts running Vault, with 5 hosts running Consul as the storage backend. We use a load balancer with health checks to point to the Vault master.
Throughout the process there were several things we learned that will help us going forward:
In the future, we plan to use Vault for continuous credential rotation utilizing the various dynamic secret backends to create short-lived access credentials. This would include issuing certificates from the built-in PKI to services for client authentication and service-to-service encryption in transit. Additionally, we’ll be leveraging the dynamic secret backends more to automate access and revocation to things like AWS, host access via a bastion+SSH, and certificate deployment using the built-in PKI.
Vault has become a core part of our security strategy at New Relic. It, along with a few other services, creates the foundation of infrastructure that abstracts complex topics like credential management, AuthN/AuthZ, and makes the secure thing to do also the easy thing to do.
HashiCorp Vault is a product to secure any infrastructure and any application. Vault provides a consistent workflow to securely store and manage secrets, provide encryption as a service, and granular privilege access management. To learn more about HashiCorp Vault visit https://www.hashicorp.com/products/vault/.
Read this curated list of HashiCorp learning resources to help practitioners and organizations better understand the cloud operating model.
Vault can now integrate with ServiceNow for credential management with the Vault Credential Resolver integration now available in the ServiceNow App Store.
Get a list of the 10 most interesting HashiConf Global sessions for enterprise cloud platform teams. Join us October 19 - 20, 2021.