Enabling the business of medicine
“For years, we’d managed secrets and data security in a proprietary secrets management solution and several purpose-built systems for specialized teams,” says Jeff Byrnes, lead site reliability engineer at athenahealth. “This made it much more difficult to ensure we continued to meet the increasing security needs of our customers given the sensitive nature of the data athenahealth receives. We needed to standardize how we manage secrets to ensure that we can easily and continuously deliver high quality security services to our customers.”
Lack of transparency required X-ray vision
As athenahealth’s customer roster has grown, so have the number of patient and operational records the company is responsible for securely storing. With more than 40% of U.S. patients’ records to protect, it is especially important for athenahealth to remain nimble with its security capabilities.
“A long time ago, we would run hundreds of bare metal servers with VPNs for secure access that we could manually manage ourselves and store secrets how and where we wanted,” says Ganapathysaran Nambirajan, senior engineering manager, platform services at athenahealth. “But when that environment grew to tens of thousands of servers, we needed to evolve how we managed secrets and several different solutions by different teams started to appear. The burden of managing multiple home-grown systems and ensuring all of them met all compliance and security requirements became difficult and overly resource intensive over time for a small team to manage.”
“The fact that everything was so siloed made systems access and governance much more complicated than it needed to be,” says Nambirajan. “Any time one team needed access to systems or records managed by another team, they’d have to request a ticket and wait two or three days for them to resolve it. It was really difficult to track the volume and statuses of those tickets for real-time updates.”
The sheer volume of tickets and disjointed processes for handling and logging them also ran up the cost of the resources needed to support various departmental needs , which elevated the need for athenahealth to look for a viable alternative.
“Our central support team usually acts as the last resort when other teams are having problems resolving an issue, often because of a lack of access or visibility,” Byrnes says. “The number of interruptions during the work day and after-hours requests from our on-call resources continued to increase as the volume of secrets-related problems grew. Eventually we reached an inflection point and wanted to find a unified method for secrets management with self-service capabilities that could save us time, money, and a ton of worry.”
Securing large volumes of patient and platform records
Automating secrets management for greater productivity and efficiency
Reducing secrets-related operating costs
Minimizing service interruptions from mismatched or outdated secrets
A secure and standardized operation
After an extensive search for a simple, intuitive, and reliable solution that could easily integrate with a range of web applications, athenahealth adopted HashiCorp Vault to streamline, standardize, and systematize the company’s secrets management operations.
With some users already familiar with HashiCorp Terraform, athenahealth consolidated thousands of secrets into Vault and distributed them out from there to the company’s various web, database, and applications servers. Centralizing security and certification requirements to have the secrets all in one place enables the team to universally apply general secrets policies and roles at scale automatically, to standardize records management practices, and to minimize service interruptions from outdated or undiscoverable secrets.
“Vault makes it easy to establish overarching secrets policies and give individuals and teams their own self-storage options to meet both general and very specific needs,” Byrnes says. “More importantly, combining Vault’s availability with the provisioning and discovery capabilities of HashiCorp’s other solutions such as HashiCorp Consul, we’ve virtually eliminated our old secrets ticketing system and can resolve availability issues more quickly and confidently.”
In the future, both Byrnes and Nambirajan expect to continue expanding their use of Vault, taking advantage of the dynamic secrets capabilities to further encrypt and decrypt application data with a simple HTTP (TLS) API call as a precursor to an encryption-as-a-service offering.
“Vault has proven to be a great equalizer for us, helping find the balance between ensuring the continued security and protection of our sensitive data and minimizing the amount of time and effort it takes,” Nambirajan says. “It’s been a revelation for simplifying one of the most important and complex areas of our operation and opens a host of new opportunities within our company and for our customers in the future.”
Securely processed more than 300 million requests for secrets a day
Virtualized physical servers to better leverage virtualization stack
Automated secrets policy application for greater efficiency
Eliminated manual ticketing system and data caching layer
Significantly reduced operational costs
athenahealth is using HashiCorp Vault to automate secrets management to securely store the secrets that protect hundreds of millions of patient and business records, improve service availability, and reduce human resources consumption.
Jeff Byrnes Lead Site Reliability Engineer athenahealth
Jeff Byrnes is a lead site reliability engineer for athenahealth, providing technical leadership and coordination on the Core Production SRE team. He is responsible for multiple datacenters with on-premise systems, covering load balancing, web applications, database servers and the systems that stitch them all together. Jeff brings more than a decade of experience in the operations engineering and public cloud space.
Ganapathysaran Nambirajan Senior Engineering Manager, Platform Services athenahealth
Ganapathysaran Nambirajan is one of athenahealth’s senior engineering managers and a member of the company’s Platform Foundational Services, responsible for building a centralized secrets management solution and providing reliable and cost-efficient faxing service. Ganapathysaran is a trusted leader with over 18 years’ experience in delivering software projects and products with particular expertise in health care, investment banking, derivatives and risk analysis, and accounting and costing.
- Bare metal, VMware ESXi Hypervisors
- Oracle Linux, CentOS, RHEL, Windows