Using Terraform dynamic provider credentials in your AWS landing zones

We often hear from customers about the steps they took to start a new project or migration in Amazon Web Services (AWS). Landing zones are a popular pattern that many customers use. For AWS environments, that often means using AWS Control Tower Account Factory for Terraform (AFT), which sets up a Terraform Cloud workspace and configures the AWS credentials.

AWS Control Tower provides an easy way to set up and govern secure, multi-account AWS environments. Using AWS Control Tower, you can set up a new AWS account with a baseline security features such as audit logging and controls/guardrails. AFT extends AWS Control Tower functionality using a Terraform pipeline to provision and customize AWS Control Tower managed accounts. Using AFT, you can apply account-specific customizations or global customizations to all accounts.

Terraform Cloud dynamic provider credentials are temporary credentials that are automatically generated for each run by HashiCorp Vault or the cloud vendor you’re using. Terraform Cloud kicks off that process by using an OpenID Connect protocol (OIDC) token. This process allows Terraform Cloud to use more secure, short-lived credentials to authenticate with AWS, and assign granular roles for each plan and apply.

Using customizations via AFT, this post demonstrates how to set up a new Terraform Cloud workspace with dynamic provider credentials pre-configured. This new workspace will then accompany every new AWS account provisioned via AFT.

»Architecture overview

It’s best to use AFT customization only for guardrails and account governance tools, such as shared VPCs, identity and access management (IAM) roles for break-glass, and Amazon S3 Block Public Access. You should also provide application-level configuration in a separate Terraform Cloud organization. Application-level configuration can be the infrastructure that runs the application (Amazon EC2, ECS, or EKS) or the data store (Amazon S3 bucket, RDS database, etc.). To satisfy both of these recommendations, this tutorial uses two separate Terraform Cloud organizations:

The Infrastructure organization: Used exclusively by the infrastructure team to deploy guardrails that enforce security, compliance, governance, and best practices, such as service control policies (SCPs), IAM policies, approved Amazon machine images (AMIs), encryption, logging, and tagging standards.

The Application organization: Used by application teams to deploy application resources, such as EC2 instances, RDS databases, Amazon Load Balancers (ALBs), and other application-specific resources.

The diagram below illustrates how to use AFT to build and configure both Terraform Cloud organizations:

1. In the Infrastructure organization, AFT creates Terraform workspaces for account and global customizations. The account customization workspace is meant for account-specific guardrails and other account-level governance. The global customization workspace is meant for generic customizations that apply to all accounts.
2. Using AFT’s account customization, create an IAM role, IAM permissions, and a Terraform Cloud OIDC identity provider in the target AWS account.
3. Use the same AFT account customization to authenticate to the Application organization and create an Application workspace.
4. AFT account customization also configures the Application workspace with the required environment variables to enable dynamic provider credentials.
5. With dynamic provider credentials configured on the Application workspace, the workspace users can run the Terraform workflow (plan/apply) without having to set up the AWS credentials. Terraform Cloud will automatically assume a role that uses temporary credentials to access the target AWS account.
6. The Application workspace user can connect the Application workspace with their VCS of choice (GitHub, GitLab, BitBucket, etc.) to enable a GitOps workflow.

Using this architecture, every new AWS account provisioned will include the guardrails and governance you configure in AFT and a dedicated Terraform Cloud Application workspace that automatically generates dynamic provider credentials for each piece of AWS infrastructure that you provision.

»Prerequisites

Before you can adopt this architecture, there are several prerequisites that you must complete:

• AWS Control Tower enabled and AFT deployed with Terraform Cloud backend. For more information, refer to the module example and the tutorial.
• Two Terraform cloud organizations, labeled “Infrastructure” and “Application”. You can use different organization names, but they must be separate.
• A Terraform Cloud user token with permissions to create workspaces in the Application organization.

»Using the Terraform Cloud/Enterprise provider in AFT

To create a new Application workspace in the Application organization, AFT account customization uses the Terraform Cloud/Enterprise (TFE) provider. This provider accepts a token argument in the provider configuration as shown below:

provider "tfe" {
 hostname = "app.terraform.io"
 token    = var.token
}

To securely retrieve this token during the plan and apply stages, we use the data source aws_secretsmanager_secret_version from the AWS provider and store the Terraform Cloud token in an AWS Secrets Manager secret. (Note: The secret must be stored with the name “/tfc/token”.) Optionally, you can also use Vault. Here is an example of the provider configuration with a token stored in AWS Secrets Manager:

To implement this in AFT account customization, we create a new customization name called SANDBOX with the following folder structure:

├── aft-account-customizations
│   └── SANDBOX
│   	├── api_helpers
│   	│   ├── post-api-helpers.sh
│   	│   ├── pre-api-helpers.sh
│   	│   └── python
│   	│   	└── requirements.txt
│   	└── terraform
│       	├── aft-providers.jinja
│       	├── backend.jinja
│       	└── tfe.tf

We use the tfe_workspace resource to set up the Application workspace. A snippet of the tfe.tf file is shown below. Each application workspace is configured with a unique name using the format: {aws_account_id}-app-workspace.

To avoid hardcoding the Application workspace organization name, we use the data source aws_ssm_parameter. The value for this SSM parameter is populated during account requests using custom fields. AFT custom fields are used to capture key-value metadata that deploy as SSM parameters in the vended account under the path “/aft/account-request/custom-fields/”:

Note from the example above: The data source aws_secretsmanager_secret_version uses provider alias aws.aft-mgt. This is intentional, because the Terraform Cloud token is stored as an AWS Secrets Manager secret in the AFT management account. This means you need to modify the AFT account customization’s aft-providers.jinja to include the new alias as shown here:

So far, you have configured the AFT account customizations to automatically configure the Application workspace. Next, set up the dynamic provider credentials for this workspace.

»Configure dynamic provider credentials

We briefly touched on dynamic provider credentials earlier, and if you need to learn more, check out this tutorial: Authenticate Providers with Dynamic Credentials.

To enable dynamic provider credentials in AWS, configure Terraform Cloud as an IAM OIDC identity provider (IdP) and add an IAM role with a trust policy to the Terraform Cloud as an IdP. To implement this in AFT account customizations, add a new iam.tf file to the SANDBOX customization:

├── aft-account-customizations
│   └── SANDBOX
│   	├── api_helpers
│   	│   ├── post-api-helpers.sh
│   	│   ├── pre-api-helpers.sh
│   	│   └── python
│   	│   	└── requirements.txt
│   	└── terraform
│       	├── aft-providers.jinja
│       	├── backend.jinja
│       	├── iam.tf <- configure OIDC and IAM role
│       	└── tfe.tf

Read our terraform-dynamic-credentials-setup-examples repository as an example. Here is the snippet for OIDC IdP setup:

Optionally, you can harden this setup by modifying the subject claim condition in the assume_role_policy. There are a number of other ways you can fine-tune your guardrails for this setup, such as limiting permissions to particular projects or just terraform plan or terraform apply. But note that the IAM role should also have enough privileges to be able to deploy the infrastructure in AWS, otherwise the plan or apply will fail.

One benefit of this setup is that you don't need static secret keys. All you need is the IAM role. You can also configure the optional environment variable to use different IAM roles for plan and apply.

Next, you’re going to automate this as part of AFT account customization. Use the tfe_variable to inject the environment variables into the Application workspace as shown here:

»Using the customization

To use the AFT account customizations, specify the customization name SANDBOX when calling the aft-account-request module. Specify the target Application organization in a custom field with the attribute app_org. AFT applies the Account customization on new account requests and subsequent runs of the AWS CodePipeline.

To run this on an existing account, update the relevant aft-account-request module with the new customization name to re-trigger the CodePipeline. When finished, AFT creates a new Application workspace with the format {aws_account_id}-app-workspace in the Application organization. As mentioned earlier, the lifecycle of this Application workspace should be managed outside of AFT, for example, by connecting the Application workspace with a Git repository to enable a VCS-driven workflow:

»An important consideration

When using AFT account customizations to create a new workspace, the lifecycle of the workspace follows the lifecycle of the account customizations name. For example, when switching from SANDBOX to another customization name, AFT may delete the workspace because it was not declared as a resource in the other customizations. Consider using the lifecycle meta argument prevent_destroy as a safety mechanism.

»Next steps

By leveraging AWS Control Tower Account Factory for Terraform (AFT) and Terraform Cloud dynamic credentials using the solution described in this blog post, organizations can automate the process of setting up new AWS environments at scale. Automating this process makes it easier to securely deploy AWS application resources using dynamic, short-lived credentials.

You can find the configuration from the example earlier in this GitHub repository. There are many other opportunities to leverage AFT to customize the Application workspace. For example, you can group the workspace to a Terraform Cloud project, configure team access, connect the Application workspace to a VCS, apply variable sets, use aSentinel policy set, configure run tasks, and much more. We have provided additional samples in the repository and we encourage you to test and come up with new ideas.