Repeating Yourself: Iteration in HashiCorp Terraform 0.12+

Transcript

My name is Brian Menges, and I'm going to talk on repeating. We're going to cover things like Terraform 0.11 and what used to be iteration, and also Terraform 0.12 and 0.13.

I work for Anaplan as a principal engineer. I have a number of hobbies, including brewing, contributing to open source, and fostering for animal rescue. The usual social media links are where you can find me online, usually with my last name followed by first initial: Twitter, LinkedIn, GitHub.

I always like to work with other fellow engineers and good people. If you're looking for a career or a change in career, or you're new to the market, come check out Anaplan. We do some very interesting things across many clouds, and we have a pretty large infrastructure, so we could always use the help.

We're going to cover some older iteration in Terraform 0.11. We're going to go over what iteration is today with Terraform 0.12 and 0.13, and we're going to describe how that works: numeric indexes versus named indexes, and the strategies that we would use with for and for_each, and recount what we used to do with count and numeric indexes.

Some of the syntax that helps with that are things like "length," "flatten," and "locals." Length, to get you your numbers; flatten, to manipulate your structures; and locals, to help you with generating those variables that you're going to use for manipulation.

Some of the tools and things that I use are terraform console, for seeing what it is that I'm going to work on in terms of the functions that I use; terraform output, to make sure that it displays correctly; and terraform state, which generates an output file.

We can parse and view that, use those in our IDEs, and then investigate what the structures are and how it's coming out.

Then there's terraform jq, which is very useful for parsing JSON and making sure that we're going to get the right kind of output structure.

Iteration with Terraform 0.11

With Terraform 0.11 and a lot of the versions before that, you used an indicator count on your resource items.

It would return a string of that number of how many of that resource you would like to do. You would provide that length of list, and it would iterate over the number of items and then return a number, and then that resource would happen that many number of times.

Also, as a result of that lack of object expressivity, you would have to create other variables like repo private as a map.

And that map would use one of those index items from your list, and then repo private indicates whether or not this is a private or public repo.

We would have the repos on the left side of our map and a true/false on the right side of our map determining whether or not that was a public repo or a private repo when we iterate it through this.

Of course, in order to represent that information, we would have to join those items in what's called "splat notation." In Terraform 0.11 and prior, you would have to do outputs as a 3-field item unless you had numerics, which is what splat notation was. It brought that fourth field in there.

These fields are separated by periods. So github_repository. meant all of the counts, all of the indexes of GitHub repository named repos and .name.

Moving to Terraform 0.12

With Terraform 0.12, we expanded a bunch. HCL2 was introduced, which is a complete refactor of the HCL language. With that, a bunch of the double quoting that you saw in the last example has been reduced significantly.

What we used to know as "interpolation" is now called "function."

Types are actually meaningful, so when we set things like booleans or numbers or strings, they return as that type in Terraform 0.12 and beyond. In Terraform 0.11, it was questionable whether or not you got a real numeric. In fact, most times you would get a string as a return.

Modules can now have very specific provider information. So when you configure your module, you can provide it a specific alias for your module that says, "Use the credentials from this alias, or use the information from the provider from this alias," which you previously couldn't do.

Of course, there are a number of others. This "Introducing Terraform 0.12" link is a fairly large and expansive document. It goes over a lot of the high-level changes.

Then of course the configuration syntax is fairly current and consistent.

With typing, we can say our variable is a number or a string or a boolean.

And we can express those more richly. So we can say that we have a list of numbers or a list of strings or a list of booleans or a set of, a map of, object of, and tuple. Those are our data structure objects constructors, our types, our number, string, and boolean.

We also introduced the concept of null. In Terraform 0.11 and before, if you didn't want something or you wanted something unset, you didn't have a way of saying that it was undefined or to not provide it. You would provide a quote with nothing in between.

Null actually means null. Null is a special type, and it means "undefined" ultimately.

Looping

Let's get a bit into the looping. One of our first constructs is for_each, and we do these on resources. In a for_each of the resource, we can take a structure like a map of objects, or a map itself, or a list of objects, or a list of maps, things like that, and we can iterate over those and say, "From those in each value that I grab, I'm going to grab each value the field name."

I'm going to look for the key name, which is a string, and I'm going to ask, "Is this null?" I'm going to use the key from my structure, or I'm going to use the actual value from my structure.

Then for private, I'm going to look into my structure and say, "Is private there, and is it null?" Then I'm going to assume that it's not a private repository.

Otherwise, I'm going to take the value that I specified from it in my structure. As for for, it's a way to iterate over our objects, but not really for the resource itself. It's for manipulating or indexing through the data structure that we're going to work with.

In this particular case, we're going to create a data structure that combines our teams and our repositories and decide what we're going to do. In the locals definition that you can see, I'm creating what's called the "global team repository variable." It says, "for r in github_repository.repos"—that's our resource and our resource name—we're going to give it an index of r.

Then for the github_team.team, we're going to give a resource of t and we're going to go through each of those objects. And we're going to take a look at our variable teams and see if we have some permissions.

We're going to go over the repository and capture that name. We're going to grab the team ID and the team name.

We're going through all the teams with every repo and seeing if we're going to include this is our new object, because what we're going to do is assign these teams to each repo.

That's what the github_team_repository resource does. It says, "I need a team ID, I need a repository, and I need a permission for that team that I'm going to assign." So that's the data structure that we built.

When we have that structure, there's no left-side index, and this is where we come into for. For x in the local structure that we built, we're going to put something on the left, which is a combining of the team name and the repository, and then we're going to give the whole object as what it points to.

That way, our index for globalteamrepository is going to be the combination of that team name, a colon, and the repository. Then all of the settings for are going to be contained within that map.

In dynamic, this allows us to configure a block of code that doesn't have to exist, or may exist multiple times.

So in github_repository repos, we might have a template repository. And there might be a configuration there on the object that we've configured that says that this is a template.

In this for_each structure, we're going to check if the value template, the key template inside of that structure, is null, and if it's null, we're going to pass it empty, which means that dynamic template as a resource block is completely excluded from this index when we for_each through the var.repos.

If we do find it, we want this resource to only accept 1 template block. We're going to give it a list with 1 index. It doesn't really matter what that index is; we're just going to give it an index.

We put an index number 0 in there to make sure that the list is at least the length of 1. For the content, we're going to reference our data structure or parent structure, not the iterator that we get from template.

We're going to look at the key template and the key template owner, because template is a map itself, and we're going to provide those to the owner argument and the repository argument dynamically.

You can also use dynamic to configure multiple instances of that block. One popular strategy is tagging in AWS.

When you filter for your AMI, you may want to filter on a number of items. We're going to filter in this particular case on 5 filters. That would normally mean that our resource would be configured with 5 filter blocks.

You would have:

filter {

name = "some kind of thing"

values = ["some kind of thing"] 

Instead of specifying that multiple times, dynamic allows us to configure that in a very malleable way, should I modify my filter structure and add more indexes or remove more indexes.

Let's say that we don't filter on state anymore or virtualization type. I could specify that variable with just the name, owner, and route device type items. Then the AWS AMI data resource will have 3 filter blocks computed on it: 1 for the name filter, 1 for the owner alias filter, and 1 for the route device type filter.

It'll do that any number of times, as long as the resource handles it. Your differences are whether or not you're doing multiples of the block or 1 of the block and what strategy you use.

In this dynamic filter state, we use dynamic filter and filter as the object index. We're using the keys and values from this flat map and making sure that we create these items.

Changes in Terraform 0.13

Let's see what we improved with Terraform 0.13. Now, instead of specifying modules multiple times—like a stamp where, if you wanted multiple colors, you would hit a different ink pad with different colors and then stamp it again—we're going to call them more like resources, and we're going to allow ourselves to iterate over them, just like we would any other regular resource that you saw previously.

Also, depends_on can now block on your modules. Before, in Terraform 0.12 and earlier, once your resource computed an output or computed some kind of value or variable item to output, your modules then could technically broadcast that, and your next module that depended on that would be able to fire.

You wound up with situations where people would put in purposeful delays at the end of their modules and rely on some kind of output captured from that in order to make sure that their module fully completes before it goes on to the next module.

Now, you can put depends_on on modules just like you would on resources. And now that whole module has to complete all the resources within it before the next thing can fire. So if you have a dependency relationship between 2 different modules or a module and a resource, that module will fully complete now, whereas before, depends_on was not an allowed option.

And of course, that iteration, now we could for_each for the module. Instead of passing a whole data structure, like what we had described before, into the module and the module has to be built in order to handle that, now we can make those modules more singleton, so that the module is going to do exactly what we say it is.

If we want to iterate that, we do that with our plan. Instead of creating a massive data structure in our plan and passing it into the module, the module parses and handles that data structure and produces the output.

Here's an example. Just like a regular terraform plan for a single resource, we're going to create all the variables that we'll take as inputs and generate our resource. In this particular case, we're going to create a GitHub repository, and we're going to ignore some lifecycle items, but then provide all of our options.

This allows me as a module creator to allow or put in some restrictions like some conditional formatting or conditional objects on top of those items.

Then of course, we brought back the dynamic template, because if I want to create a repository that is off of a template repo, I can provide this item and specify that. Without changing the module itself, I can handle both.

When we want to call this module multiple times, we do our data structure in our plan, and we provide that in a for_each to the module. Now the module becomes the index-iterable object. When we pass that in, we have to tell it, "Each value from my resource that I'm creating, I'm going to pass that allow_merge_commit key from that object and the allow_rebase_merge key object."

Of course, if any of those items are null, it results in nothing being input and allows my module to take a default action, as opposed to me having to provide that. Of course, name = each key, I'm going to force the user to give me a name because I have to name the module something and the resource itself requires that.

Numeric Versus Named Indexes

Now let's compare our numeric versus named indexes. With numeric indexes, you get simply numbers. Of course, you can do that just the same any number of times.

With for_each, you get a referenceable name, for example "first repository," "second repository," "third repository." While structurally in the object these don't make too much difference, it does make a difference when you modify these objects

When we want to come in and, let's say, remove the second repository, which is our index No. 1, we go in, make our change, and remove our first index.

But what we've really done is we've made what used to be index No. 2, otherwise defined as the third repository, and renamed that as "index No. 1." This is going to reshuffle our items

You can see really confusing results, similar to destroy/create or change/destroy.

Of course, your relationships on account resource are very difficult to maintain. If you want resources built on other resources, like we have teams built off of repositories and that sort of membership relationship, you're going to have to do some complicated modular interleaving in between those items.

As an example, we removed our first index. But what Terraform 0.12 and earlier does is compare its state and then find that the diff says index No. 1. It's not that it's removed; it's that it's changed because the list that you provided me is a list of 2, not a list of 3.

So it interprets the second index, index No. 1, as changing to all the values of the third repository. Now we're going to have to change all of our settings.

In GitHub is still our second repository; we're just naming it now our third repository. Then what happens to our third repository? It gets deleted, because our index list is now not 3 long, it's 2 long. It doesn't find that second index and it removes all the content.

Of course, in the GitHub world, that means that the third repository got deleted and not the second repository, and that results in our change/destroy sort of relationship.

Each resource is different, so you'll have to figure that out.

Now when we go to named indexes, we have referenceable names. When I say I want to delete that first index, I'm actually saying, "I want to delete the index called second repository." In this example, I have that relationship back. I have my GitHub teams related to my repositories.

When I go through, I'm only going to delete my second repository, because my object, I only removed that index. Doesn't matter where in the list it is, because on the left-hand side, my index name is second repository.

Because there was a relationship previously where we said that there were teams with that, we delete only the ones related to the second repository. When we used to have 6 objects and that sort of relationship and we wanted to remove 1 repository, we needed to remove the teams associated with that.

Now we remove 4 things from our list.

Here's a code listing or setup, because we can go back and see that there's a very easy and express relationship, especially from my comment.

I'm going to generate an object and assign the team and permission to all repositories, where that item global has my Boolean true. That's the test at the end of my 4 statements inside that inner for loop.

So I could create this object, and if I have other teams that are not global, this won't be included, but the teams that are global will be included.

Syntax Specifics

Going back to length in Terraform 0.11 syntax, we're going to be returning strings when we return these numbers, which is not optimal, but that's the way that the language runs.

Of course, we're going to run on count indexes, and we're going to have to make maps in order to express our complex object listings and things like that.

But in 0.12 and 0.13, we're going to return specific types when we use these items. In 0.12 and 0.13, account returns a number as opposed to a string.

Resources based on length and using your count are going to have numeric iteration, and it's going to be kind of complex to express relationships.

In a flattened scenario, it removes all of the empty outer constructs like empty arrays or arrays of only your object and compresses them down so you have a simplified data structure.

When you work with complex data structures, you're definitely going to want to use a lot of the tools, like terraform console, a lot. You're going to want to look at the state in outputs and look at the relationships between these items.

Sometimes you're going to have to work with your loops and iterators, your for calls. And you may need to give 2 indexes, things like for r, i in github_repository.repos, or i becomes the left-hand side of that relationship and r becomes the left-hand side.

Reviewing Terraform Tools

Let's cover the tools again. Terraform is your built-in binary, so console, output, and state are free commands that you have in order to look through and go through your objects. In console, you can do your function math. If it's local, you're going to have to make a couple of the steps.

For things like count.index, you're going to have to provide the actual index of your resource in order to see the data object and what goes out there. In terraform output, you have your full output and what you want to pass on.

State contains a lot of the same information, but also the resource objects and fields and other things that you can probably call and go through and look at.

Now, terraform jq is extremely useful, especially if you output or tell Terraform to output as a JSON and your state file is in JSON. You can go through and do filters in jq to make sure that you are getting the right content in the right places.

And of course, most useful as always is running and operating tests. For a lot of the code that I've shown as an example, I will provide Terraform 0.13-compatible code, and that'll be up on my personal GitHub repository when I release.

You'll also see Terratests in there that will show that my object is getting the right result.

Of course, there are some wonderful resources. The learn.hashicorp.com site is constantly revamped. It's probably the best place to go for educational material. I highly recommend going through some of the tutorials.

Specifically, there are some upgrade gotchas on Terraform 0.13, if you're coming from 0.12. If you're fresh and starting, fantastic, but you still might want to take a look at them because there are some better strategies that we have learned since Terraform 0.12 that will be included in there.

As always, citing Terratest, Terraform's doc, and all the syntax, and then your module registry, which is great for browsing both providers and provider documentation. And there's my GitHub resource link.

I'd like to thank you for spending the last couple of minutes with me. Hopefully, you found this informative.