How Do I Decompose Monolithic Terraform Configurations?

For more details regarding the specific patterns to break down monolithic Terraform projects, refer to this article and this HashiCorp blog post

Also, don't forget to check out our series on how Terraform Enterprise helps in this area.

Transcript

Sometimes our users approach us at conferences or trade shows and ask us a question, "How do I decompose my monolithic Terraform configurations?"

What do we mean by a monolithic configuration? This is a huge set of Terraform files that build something big. You've thrown everything—from your network to your security rules, your virtual machines, load balancers—into one giant workspace.

We define a workspace as a folder or a repository that contains some Terraform code. And a monolithic workspace, or a monolithic repo, contains a whole lot of Terraform code.

The good and bad of monorepos

The advantage here is it's convenient. It's easy to stand up something because everything's included inside this giant repository. The downside is, if you want to make even the tiniest change to that repository, you could potentially break something. Or you're changing one tiny bit of something, but you have to change the whole repo to make it happen.

This is particularly troublesome if you're storing your code inside of Git or another version control system. Because even the smallest change to that repository can trigger an automated build and all sorts of other dependencies that are tied tightly to that thing that you're trying to change.

Terraform modules solve the monolith problem

Instead of using a giant, monolithic repo, you can convert that repository into what we call a "modular repo," or smaller, bite-size Terraform configs. Terraform supports the concept of modules, or reusable bits of Terraform code that are small and replaceable. This way, if you need to make a change to part of your infrastructure, you just change the module that you're working on, instead of having to worry about changing the whole monolithic repo.

Let's go through an example. Suppose I have an application that includes a few Linux servers, a load balancer, maybe a DNS record, some security rules, networking—all of the bits and pieces that make up your application stack. What I could do is take the networking code out of my Terraform config and move it over into its own module. And that way that module can be reused over and over again inside of my code. But if I need to change it, it lives in its own directory, in its own workspace, and I don't have to change the entire mono repo just to make a small change to my module.

Another advantage of modules is that you can version them. So production might be using version 1 of my module code, whereas dev and staging could be using version 2 or 3, depending on what level of testing you've done and how ready that code is for production.

Getting from mono to modules

You might be asking yourself, "How do I go from this big, monolithic repository to a smaller, module-based repository?" The answer is, you don't have to do it all at once. You can break it down into steps. Take part of your monorepo, and take out the parts that make sense to manage separately, like the network, for example. Or you might manage the load balancer inside of its own module.

Gradually you can break that monorepo down into a bunch of smaller repositories that could be managed by different individuals, or different teams. This makes it much easier to manage your code, because you're not dealing with one gigantic codebase. You have lots of smaller codebases that you can work on independently.