HashiCorp Terraform 0.12 Preview: For and For-Each
This is the third post of the series highlighting new features in Terraform 0.12.
Tip: As of Terraform 0.13
for_each
now works with modules as well as resoruces. Follow ourfor_each
tutorial to get started.
As part of the lead up to the release of Terraform 0.12, we are publishing a series of feature preview blog posts. The post this week is on the new iteration features: for
expressions and for_each
.
A common problem in Terraform configurations for versions 0.11 and earlier is dealing with situations where the number of values or resources is decided by a dynamic expression rather than a fixed count. The general problem of iteration is a big one to solve and Terraform 0.12 introduces a few different features to improve these capabilities, namely for
expressions and for_each
blocks. We also discuss further enhancements that will come later.
» For Expressions for List and Map Transformations
When working with lists and maps, it is common to need to apply a filter or transformation to each element in the collection and produce a new collection. Prior to Terraform 0.12, Terraform only had limited support for such operations via a few tailored interpolation functions, such as formatlist.
Terraform 0.12 introduces a new construct called a for
expression, which allows the construction of a list or map by transforming and filtering elements in another list or map. The example below shows this in use:
# Configuration for Terraform 0.12
variable "vpc_id" {
description = "ID for the AWS VPC where a security group is to be created."
}
variable "subnet_numbers" {
description = "List of 8-bit numbers of subnets of base_cidr_block that should be granted access."
default = [1, 2, 3]
}
data "aws_vpc" "example" {
id = var.vpc_id
}
resource "aws_security_group" "example" {
name = "friendly_subnets"
description = "Allows access from friendly subnets"
vpc_id = var.vpc_id
ingress {
from_port = 0
to_port = 0
protocol = -1
# For each number in subnet_numbers, extend the CIDR prefix of the
# requested VPC to produce a subnet CIDR prefix.
# For the default value of subnet_numbers above and a VPC CIDR prefix
# of 10.1.0.0/16, this would produce:
# ["10.1.1.0/24", "10.1.2.0/24", "10.1.3.0/24"]
cidr_blocks = [
for num in var.subnet_numbers:
cidrsubnet(data.aws_vpc.example.cidr_block, 8, num)
]
}
}
When a for
expression is wrapped in square brackets ([
and ]
) as shown above, the result is a list. A for
expression wrapped in braces ({
and }
) produces a map in a similar way. For example:
# Configuration for Terraform 0.12
output "instance_private_ip_addresses" {
# Result is a map from instance id to private IP address, such as:
# {"i-1234" = "192.168.1.2", "i-5678" = "192.168.1.5"}
value = {
for instance in aws_instance.example:
instance.id => instance.private_ip
}
}
The optional if
clause can be used also to filter the input collection so that the result contains only a subset of input items:
# Configuration for Terraform 0.12
output "instance_public_ip_addresses" {
value = {
for instance in aws_instance.example:
instance.id => instance.public
if instance.associate_public_ip_address
}
}
Finally, the map form of a for
expression has a grouping mode where the map key is used to group items together into a list for each distinct key. This mode is activated by placing an ellipsis (...
) after the value expression:
# Configuration for Terraform 0.12
output "instances_by_availability_zone" {
# Result is a map from availability zone to instance ids, such as:
# {"us-east-1a": ["i-1234", "i-5678"]}
value = {
for instance in aws_instance.example:
instance.availability_zone => instance.id...
}
}
In the the example above we show that a resource with count set now also behaves as a list, allowing all of the instances of aws_instance.example
to be iterated over to produce a grouping by availability zone.
These new expressions can be used to generate a value for any argument that expects a list or map expression. And in Terraform 0.12, this includes any location that accepts a list or map, including module inputs.
» Dynamic Nested Blocks
Several resource types use nested configuration blocks to define repeatable portions of their configuration. Terraform 0.12 introduces a new construct for dynamically constructing a collection of nested configuration blocks.
For example, the aws_autoscaling_group
resource type uses nested blocks to declare tags that may or may not be propagated to any created EC2 instances. The example below shows the Terraform 0.11 and earlier syntax:
# Configuration for Terraform 0.11 and earlier
resource "aws_autoscaling_group" "example" {
# ...
tag {
key = "Name"
value = "example-asg-name"
propagate_at_launch = false
}
tag {
key = "Component"
value = "user-service"
propagate_at_launch = true
}
tag {
key = "Environment"
value = "production"
propagate_at_launch = true
}
}
Because these nested blocks are validated statically, it was previously difficult or impossible to implement more dynamic behaviors. Some users found ways to exploit some implementation details to trick Terraform into partially supporting dynamic generation of blocks, but these workarounds were unreliable because Terraform makes assumptions about nested blocks that do not hold for arbitrary expressions.
Terraform 0.12 introduces a special new dynamic block construct to support these dynamic configuration use-cases in a first-class way. The same example converted to Terraform 0.12:
# Configuration for Terraform 0.12
locals {
standard_tags = {
Component = "user-service"
Environment = "production"
}
}
resource "aws_autoscaling_group" "example" {
# ...
tag {
key = "Name"
value = "example-asg-name"
propagate_at_launch = false
}
dynamic "tag" {
for_each = local.standard_tags
content {
key = tag.key
value = tag.value
propagate_at_launch = true
}
}
}
A dynamic "tag"
block behaves as if a separate tag
block were written for each element of the list or map given in the for_each
argument. Because dynamic
is itself given as a nested block, all of the same syntax constructs can be used within its content block that would normally be valid in a literal tag block, and both static and dynamic tag blocks can be mixed as shown above. This enables arbitrarily complex behaviors as necessary.
Since the for_each
argument accepts any list or map expression, this feature can be combined with for
expressions as described above to create nested blocks based on arbitrary transformations of other list and map values:
# Configuration for Terraform v0.12
variable "subnets" {
default = [
{
name = "a"
number = 1
},
{
name = "b"
number = 2
},
{
name = "c"
number = 3
},
]
}
locals {
base_cidr_block = "10.0.0.0/16"
}
resource "azurerm_virtual_network" "example" {
name = "example-network"
resource_group_name = azurerm_resource_group.test.name
address_space = [local.base_cidr_block]
location = "West US"
dynamic "subnet" {
for_each = [for s in subnets: {
name = s.name
prefix = cidrsubnet(local.base_cidr_block, 4, s.number)
}]
content {
name = subnet.value.name
address_prefix = subnet.value.prefix
}
}
}
Terraform is able to validate the arguments inside the content
block in the same way as it would validate a static block, even if the value of the for_each
expression is not yet known. Thus this approach allows more problems to be caught at plan time, before any real actions have been taken.
We still recommend that you avoid writing overly-abstract, dynamic configurations as a general rule. These dynamic features can be useful when creating reusable modules, but over-use of dynamic behavior will hurt readability and maintainability. Explicit is better than implicit, and direct is better than indirect.
» Resource for_each
The dynamic block construct described previously includes the idea of iterating over a list or map using the for_each
argument, which is intended as a more intuitive and useful way to create dynamic nested blocks compared to the count argument on resources.
During the development of Terraform 0.12 we've also laid the groundwork for supporting for_each
directly inside a resource
or data
block as a more convenient way to create a resource instance for each element in a list or map. Unfortunately we will not be able to fully complete this feature for the Terraform 0.12 initial release, but we plan to include this in a subsequent release to make it easier to dynamically construct multiple resource instances of the same type based on elements of a given map. The example below shows future Terraform syntax, not included in the initial 0.12 release:
# Planned for a later release, after v0.12.0
variable "subnet_numbers" {
description = "Map from availability zone to the number that should be used for each availability zone's subnet"
default = {
"eu-west-1a" = 1
"eu-west-1b" = 2
"eu-west-1c" = 3
}
}
resource "aws_vpc" "example" {
# ...
}
resource "aws_subnet" "example" {
for_each = var.subnet_numbers
vpc_id = aws_vpc.example.id
availability_zone = each.key
cidr_block = cidrsubnet(aws_vpc.example.cidr_block, 8, each.value)
}
The new object each
, with attributes each.key
and each.value
, will allow access to the key and value of each element of the for_each
expression, in a similar way as count.index
for the count argument.
When the for_each
argument value is a map, Terraform will identify each instance by the string key of the map element rather than by a numeric index, which will avoid many limitations with the current pattern of using count to iterate over a list where items may be added and removed from the middle of that list, changing the subsequent indices.
While we will not be able to complete this feature in time for the initial Terraform 0.12 release, Terraform will consider the argument name for_each
and the expression each
to be reserved and not usable by resource types. That means that this feature can be completed in a subsequent release without additional breaking changes.
» Module count and for_each
For a long time, users have wished to be able to use the count
meta-argument within module blocks, allowing multiple instances of the same module to be created more easily.
Again, we have been laying the groundwork for this during Terraform 0.12 development and expect to complete this work in a later release. Along with count
, module blocks will also accept the new for_each
argument described for resources above, with similar results.
This feature is particularly complicated to implement within Terraform's existing architecture, so some more work will certainly be required before we can support this. To avoid further breaking changes in later releases, 0.12 will reserve the module input variable names count
and for_each
in preparation for the completion of this feature.
» Upgrade Guide
The new for
and for_each
functionality introduces new reserved words into existing resources and modules. We've verified that this doesn't introduce any breaking changes to the resources provided by official providers. User-created modules that use count
or for_each
will need to be updated.
The future functionality (beyond Terraform 0.12) described in this post for resource and module iteration will not introduce any further breaking changes since we are reserving all the necessary keywords in the Terraform 0.12 release.
The existing count
functionality remains working as before, but many edge cases have been resolved by the underlying architecture changes to support arbitrary for_each
.
» Next Steps
This was part 3 of the blog post series previewing Terraform 0.12.
for
expressions and for_each
will be released in Terraform 0.12 (except those use cases specifically noted), coming later this summer. To learn more about how to upgrade to Terraform 0.12, read the upgrade instructions which will be continuously updated as we get closer to releasing Terraform 0.12. If you have any feedback or concerns about these changes, please communicate with the Terraform team via the public mailing list.
Sign up for the latest HashiCorp news
More blog posts like this one
Terraform delivers launch-day support for Amazon S3 Tables, EKS Hybrid Nodes, and more at re:Invent
The Terraform provider for AWS now enables users to manage a variety of new services just announced at re:Invent.
HashiCorp at re:Invent 2024: Infrastructure Lifecycle Management with AWS
A recap of HashiCorp infrastructure news and developments on AWS from the past year, from a new provider launch to simplifying infrastructure provisioning and more.
Simplify policy adoption in Terraform with pre-written Sentinel policies for AWS
HashiCorp introduces a new pre-written policy library co-developed with AWS, aiming to reduce the barrier of adoption for policy as code infrastructure workflows.