athenahealthCustomer Case Study

A New Prescription for Secrets Management

How a global leader in healthcare technology uses HashiCorp Vault to enhance security and keep its partner and customer records in tip-top shape

// Infrastructure Enables Innovation
  • Centralized multiple homegrown systems into a single service
  • 100s of bare metal servers consolidated
  • 300 million requests for secrets a day
  • 160,000+ providers in network
  • 10,000+ customers
  • 40%+ of patients in the U.S. served

athenahealth

Download This Case Study

athenahealth partners with medical organizations across the country to drive clinical and financial results. Their vision is to create a thriving ecosystem that delivers accessible, high-quality, and sustainable healthcare for all, and they are pursuing this through their medical record, revenue cycle, patient engagement, and care coordination service offerings. Their expert teams build modern technology on an open, connected ecosystem, yielding insights that make a difference for their customers and their patients.

By combining Vault’s availability with the provisioning and discovery capabilities of HashiCorp’s other tools, we’ve virtually eliminated our old secrets ticketing system and can resolve availability issues in under 30 minutes instead of four hours.

Jeff Byrnes, lead site reliability engineer, athenahealth

Enabling the business of medicine

Since 1997, athenahealth has been supporting healthcare organizations through its many offerings ranging from electronic health records (EHR) and revenue cycle capabilities through population health, and many more. athenahealth supports more than 10,000 customers of all shapes and sizes, including over 160,000 providers. These customers naturally expect their private information to be kept safe and secure, as outlined in athenahealth’s strict privacy policy. However, managing large volumes of sensitive patient records is not without challenges.

“For years, we’d managed secrets and data security in a proprietary secrets management solution and several purpose-built systems for specialized teams,” says Jeff Byrnes, lead site reliability engineer at athenahealth. “This made it much more difficult to ensure we continued to meet the increasing security needs of our customers given the sensitive nature of the data athenahealth receives. We needed to standardize how we manage secrets to ensure that we can easily and continuously deliver high quality security services to our customers.”

Lack of transparency required X-ray vision

As athenahealth’s customer roster has grown, so have the number of patient and operational records the company is responsible for securely storing. With more than 40% of U.S. patients’ records to protect, it is especially important for athenahealth to remain nimble with its security capabilities.

“A long time ago, we would run hundreds of bare metal servers with VPNs for secure access that we could manually manage ourselves and store secrets how and where we wanted,” says Ganapathysaran Nambirajan, senior engineering manager, platform services at athenahealth. “But when that environment grew to tens of thousands of servers, we needed to evolve how we managed secrets and several different solutions by different teams started to appear. The burden of managing multiple home-grown systems and ensuring all of them met all compliance and security requirements became difficult and overly resource intensive over time for a small team to manage.”

“The fact that everything was so siloed made systems access and governance much more complicated than it needed to be,” says Nambirajan. “Any time one team needed access to systems or records managed by another team, they’d have to request a ticket and wait two or three days for them to resolve it. It was really difficult to track the volume and statuses of those tickets for real-time updates.”

The sheer volume of tickets and disjointed processes for handling and logging them also ran up the cost of the resources needed to support various departmental needs , which elevated the need for athenahealth to look for a viable alternative.

“Our central support team usually acts as the last resort when other teams are having problems resolving an issue, often because of a lack of access or visibility,” Byrnes says. “The number of interruptions during the work day and after-hours requests from our on-call resources continued to increase as the volume of secrets-related problems grew. Eventually we reached an inflection point and wanted to find a unified method for secrets management with self-service capabilities that could save us time, money, and a ton of worry.”

Challenges

  • Securing large volumes of patient and platform records
  • Automating secrets management for greater productivity and efficiency
  • Reducing secrets-related operating costs
  • Minimizing service interruptions from mismatched or outdated secrets

Why Vault?

Vault has proven to be a great equalizer for us, helping find the balance between ensuring the security and protection of our sensitive data and minimizing the amount of time and effort it takes.

Ganapathysaran Nambirajan, senior engineering manager, platform services, athenahealth

A secure and standardized operation

After an extensive search for a simple, intuitive, and reliable solution that could easily integrate with a range of web applications, athenahealth adopted HashiCorp Vault to streamline, standardize, and systematize the company’s secrets management operations.

With some users already familiar with HashiCorp Terraform, athenahealth consolidated thousands of secrets into Vault and distributed them out from there to the company’s various web, database, and applications servers. Centralizing security and certification requirements to have the secrets all in one place enables the team to universally apply general secrets policies and roles at scale automatically, to standardize records management practices, and to minimize service interruptions from outdated or undiscoverable secrets.

“Vault makes it easy to establish overarching secrets policies and give individuals and teams their own self-storage options to meet both general and very specific needs,” Byrnes says. “More importantly, combining Vault’s availability with the provisioning and discovery capabilities of HashiCorp’s other solutions such as HashiCorp Consul, we’ve virtually eliminated our old secrets ticketing system and can resolve availability issues more quickly and confidently.”

In the future, both Byrnes and Nambirajan expect to continue expanding their use of Vault, taking advantage of the dynamic secrets capabilities to further encrypt and decrypt application data with a simple HTTP (TLS) API call as a precursor to an encryption-as-a-service offering.

“Vault has proven to be a great equalizer for us, helping find the balance between ensuring the continued security and protection of our sensitive data and minimizing the amount of time and effort it takes,” Nambirajan says. “It’s been a revelation for simplifying one of the most important and complex areas of our operation and opens a host of new opportunities within our company and for our customers in the future.”

Outcomes

  • Securely processed more than 300 million requests for secrets a day
  • Virtualized physical servers to better leverage virtualization stack
  • Automated secrets policy application for greater efficiency
  • Eliminated manual ticketing system and data caching layer
  • Significantly reduced operational costs

Solution

athenahealth is using HashiCorp Vault to automate secrets management to securely store the secrets that protect hundreds of millions of patient and business records, improve service availability, and reduce human resources consumption.

athenahealth Partners

  • Jeff Byrnes Lead Site Reliability Engineer athenahealth

    Jeff Byrnes is a lead site reliability engineer for athenahealth, providing technical leadership and coordination on the Core Production SRE team. He is responsible for multiple datacenters with on-premise systems, covering load balancing, web applications, database servers and the systems that stitch them all together. Jeff brings more than a decade of experience in the operations engineering and public cloud space.

  • Ganapathysaran Nambirajan Senior Engineering Manager, Platform Services athenahealth

    Ganapathysaran Nambirajan is one of athenahealth’s senior engineering managers and a member of the company’s Platform Foundational Services, responsible for building a centralized secrets management solution and providing reliable and cost-efficient faxing service. Ganapathysaran is a trusted leader with over 18 years’ experience in delivering software projects and products with particular expertise in health care, investment banking, derivatives and risk analysis, and accounting and costing.

Technology Stack

Infrastructure: 
Bare metal, VMware ESXi Hypervisors
Platform: 
Oracle Linux, CentOS, RHEL, Windows