Skip to main content
Case Study

The coldest Monday with a $1 million cloud bill: Terraform to the rescue

How a cryptomining attack led to one startup's overhaul of its cost management systems with the help of Terraform.

Imagine this "cold monday." You're working at a small startup and you wake up to a $1 million+ bill from Google App Engine accrued over the weekend. What was it?

This was the experience at a former startup for the KubeCloud founder. A compromised service key led to a cryptomining attack.

»What was the issue?

While this talk doesn't look at the practices around preventing service key theft (hint: check out secrets management best practices), it does talk about the other guardrails you should have in place to put cloud cost monitoring and limitations in place. These were the issues the speaker established:

  • Poor cleanup processes for long-running resources (see: ephemeral workspaces)

  • Risky complexity due to the usage of many shell scripts

  • No org-wide restrictions on resource count and type (see: Sentinel)

Their first steps included setting billing alerts and converting some of those shell scripts to Terraform code, but the big fix was quotas.

»Quotas and Terraform

The startup started building a large matrix of cloud vendor quotas. But they needed something to manage all these quotas in an automated way.

In this talk you'll see a demo of how Terraform was used as a state engine to manage these quotas. While it's not a typical use case, it worked very effectively, essentially creating an in-house dashboard for cloud cost quota management and visibility.

Here's an example main.tf for this quota managing use case in Terraform:

Main.tf for quota management

More resources like this one

  • 2/3/2023
  • Case Study

Automating Multi-Cloud, Multi-Region Vault for Teams and Landing Zones

  • 1/5/2023
  • Case Study

How Discover Manages 2000+ Terraform Enterprise Workspaces

zero-trust
  • 12/13/2022
  • White Paper

A Field Guide to Zero Trust Security in the Public Sector

  • 9/26/2022
  • Case Study

How Deutsche Bank onboarded to Google Cloud w/ Terraform

HashiCorp uses data collected by cookies and JavaScript libraries to improve your browsing experience, analyze site traffic, and increase the overall performance of our site. By using our website, you’re agreeing to our Privacy Policy and Cookie Policy.

The categories below outline which companies and tools we use for collecting data. To opt out of a category of data collection, set the toggle to “Off” and save your preferences.