consulterraform

Configuring Azure Application Gateway with Consul-Terraform-Sync

Synchronize HCP Consul services as backend address pools for Microsoft Azure Application Gateway using Consul-Terraform-Sync (CTS).

Imagine that you update a virtual machine and register a new service on it with HashiCorp Consul. How can you automatically add the new service in Consul as backend address pools to a load balancer? Ideally, you want to keep your load balancer configuration in version control and continuously deliver IP address changes to the configuration. The combination of these two concepts offers a GitOps workflow for configuring the load balancer.

This post will show you how to automate backend address pools for a Microsoft Azure Application Gateway using a HashiCorp Terraform module compatible with HashiCorp Consul-Terraform-Sync (CTS). Consul-Terraform-Sync connects to the service catalog of HashiCorp Cloud Platform (HCP) Consul and monitors for service-level changes. When a change is detected, it runs a Terraform module in response to an event and automates the target resource of your choice, offering a GitOps workflow for managing an application gateway. Consul-Terraform-Sync behaves as a controller, responding to changes to Consul’s service catalog and running infrastructure as code.

Consul service catalog changes to API and web services will start a Consul-Terraform-sync task, which creates backend address pools in Azure Application Gateway

This example uses a Terraform module to register two services on virtual machines (named “web” and “api”) to HCP Consul. You’ll use Consul-Terraform-Sync to recognize the updates to the services, run a task that calls the Terraform module, and add the services as backend address pools to Azure Application Gateway.

»Set up a Consul Cluster and Register Services

Before running CTS, make sure you create a Consul cluster and register services to its catalog. Use this example to create an HCP Consul cluster on Azure. Check that CTS has access to the HCP Consul endpoint, as you will need to configure a backend to Consul.

Consul-Terraform-Sync can automate services registered by service mesh or service discovery. The example sets up two services on Azure virtual machines, named “web” and “api”. The two services register themselves to Consul. Review the example’s documentation to set up your environment.

»Configure Consul-Terraform-Sync

After configuring a Consul cluster and registering services, you can define a Consul-Terraform-Sync configuration file to synchronize service changes from Consul to an Azure Application Gateway. The configuration file downloads Terraform and sets up a CTS task that synchronizes the web and API services. The example Terraform code generates a CTS configuration based on the Consul address from Azure and services in the examples/cts-config-path.hcl file.

Running the example generates a set of CTS configuration files named “cts-config-basic.hcl” and “cts-config-path.hcl”. Both configurations set the working directory to the sync-tasks folder and expose CTS on port 8558.

working_dir = "sync-tasks"
port        = 8558
working_dir = "sync-tasks"
port        = 8558

Then, the configuration defines a buffer period. You must set a buffer period to a minimum of one minute for the application gateway. CTS will detect all changes to services and re-run the Terraform module in response. A buffer period helps account for flapping services that can cause rapid changes to the entire gateway.

buffer_period {
 enabled = true
 min     = "60s"
 max     = "240s"
}
buffer_period {
 enabled = true
 min     = "60s"
 max     = "240s"
}

The CTS configuration uses the HCP Consul public endpoint and its management ACL token. Alternatively, use a more secure ACL token with read access to Consul’s service catalog.

consul {
 address = "HCP Consul Public Address"
 token   = "HCP Consul Token"
}
consul {
 address = "HCP Consul Public Address"
 token   = "HCP Consul Token"
}

The example also defines the Terraform driver, which downloads the appropriate version of Terraform and the AzureRM provider. The parameter log = true prints Terraform output to CTS logs. This CTS configuration uses a local backend for Terraform state and the terraform_provider stanza to configure the AzureRM provider. The provider will retrieve Azure credentials from environment variables.

driver "terraform" {
 log     = true
 version = "1.1.9"
 
 backend "local" {}
 
 required_providers {
   azurerm = {
     source  = "hashicorp/azurerm"
     version = "~> 2.90"
   }
 }
}
 
terraform_provider "azurerm" {
 features {}
}
driver "terraform" {
 log     = true
 version = "1.1.9"
 
 backend "local" {}
 
 required_providers {
   azurerm = {
     source  = "hashicorp/azurerm"
     version = "~> 2.90"
   }
 }
}
 
terraform_provider "azurerm" {
 features {}
}

You can configure CTS with other supported Terraform backends. Some Terraform providers require additional configuration in their provider blocks, which you define in the terraform_provider stanza in CTS.

The CTS configuration file then defines the services to synchronize between Consul and Terraform. The Terraform module for Azure Application Gateway allows you to configure path-based or basic routing using the cts_user_defined_meta field.

The example configuration sets up paths for path-based routing to API and web services as /api and /web, respectively.

service {
 name = "api"
 cts_user_defined_meta = {
   path = "/api/*"
 }
}
 
service {
 name = "web"
 cts_user_defined_meta = {
   path = "/web/*"
 }
}
service {
 name = "api"
 cts_user_defined_meta = {
   path = "/api/*"
 }
}
 
service {
 name = "web"
 cts_user_defined_meta = {
   path = "/web/*"
 }
}

Alternatively, you can update cts_user_defined_meta to use basic routing by setting the hostname or hostnames field. Review the module documentation for additional metadata you can customize for basic or path-based routing, such as probes.

The last part of the configuration defines a CTS task, which runs a Terraform module. The example imports the AzureRM provider, the module in its local folder, and a file with additional Terraform variables. The task synchronizes API and web services.

task {
 name           = "testing"
 description    = "Example task with two services and path-based routing"
 providers      = ["azurerm"]
 module         = "joatmon08/application-gateway-nia/azurerm"
 version        = "0.0.1"
 services       = ["api", "web"]
 variable_files = ["cts-example-path.tfvars"]
}
task {
 name           = "testing"
 description    = "Example task with two services and path-based routing"
 providers      = ["azurerm"]
 module         = "joatmon08/application-gateway-nia/azurerm"
 version        = "0.0.1"
 services       = ["api", "web"]
 variable_files = ["cts-example-path.tfvars"]
}

Use a Terraform variable file to pass information about the Azure resource group, location, public IP, service subnet, and port of the application gateway. The example sets enable_path_based_routing to true for path-based routing. If you prefer basic routing, you can enable basic routing in the module by setting enable_path_based_routing to false.

name                            = "nia-testing"
azurerm_resource_group_name     = "testing"
azurerm_resource_group_location = "eastus"
azurerm_public_ip_id            = "<Public IP ID>"
azurerm_service_subnet_id       = "<Subnet ID>"
private_ip_address_allocation   = "Dynamic"
 
enable_path_based_routing = true
 
frontend_port = 80
sku_name      = "Standard_Small"
sku_tier      = "Standard"
name                            = "nia-testing"
azurerm_resource_group_name     = "testing"
azurerm_resource_group_location = "eastus"
azurerm_public_ip_id            = "<Public IP ID>"
azurerm_service_subnet_id       = "<Subnet ID>"
private_ip_address_allocation   = "Dynamic"
 
enable_path_based_routing = true
 
frontend_port = 80
sku_name      = "Standard_Small"
sku_tier      = "Standard"

You’ll want to separately create Azure resources like resource group, public IP, and service subnet with Terraform resources and pass their identifiers to CTS. Staging these resources before running CTS reduces the run time of the CTS task.

»Automate Azure Application Gateway

The example runs CTS locally to create and configure an Azure Application Gateway for the web and API services. Set Azure credentials as environment variables so Terraform can authenticate to Azure.

$ export ARM_CLIENT_SECRET="" && \
  export ARM_CLIENT_ID="" && \
  export ARM_TENANT_ID="" && \
  export ARM_SUBSCRIPTION_ID="" && \
$ export ARM_CLIENT_SECRET="" && \
  export ARM_CLIENT_ID="" && \
  export ARM_TENANT_ID="" && \
  export ARM_SUBSCRIPTION_ID="" && \

Make sure to set the working directory for the demo to the examples/ folder in the module repository.

$ ls -1
 
cts-config-basic.hcl
cts-config-path.hcl
cts-example-basic.tfvars
cts-example-path.tfvars
setup
$ ls -1
 
cts-config-basic.hcl
cts-config-path.hcl
cts-example-basic.tfvars
cts-example-path.tfvars
setup

Start CTS with the cts-config-path.hcl configuration file to create an application gateway with path-based routing for the web and API services. Running the module with path-based routing dynamically generates a request routing rule, HTTP listener, and URL path map for routes.

$ consul-terraform-sync -config-file cts-config-path.hcl
 
2022-07-11T14:09:17.472-0400 [INFO]  cli: v0.6.0 (7f0d7ba)
 
## omitted for clarity
 
2022/07/11 14:09:20 [INFO] running Terraform command:/terraform-azurerm-application-gateway-nia/examples/terraform init -no-color -force-copy -input=false -backend=true -get=true -upgrade=false
Initializing modules...
 
## Omitted for clarity
 
Initializing the backend...
 
## omitted for clarity
 
Terraform has been successfully initialized!
 
## omitted for clarity
 
2022/07/11 14:09:27 [INFO] running Terraform command:/terraform-azurerm-application-gateway-nia/examples/terraform apply -no-color -auto-approve -input=false -lock=true -parallelism=10 -refresh=true
 
Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create
 
Terraform will perform the following actions:
 
  # module.testing.azurerm_application_gateway.service will be created
 
## omitted for clarity
 
Plan: 1 to add, 0 to change, 0 to destroy.
 
## omitted for clarity
 
module.testing.azurerm_application_gateway.service: Creation complete after 13m55s [id=/subscriptions/SUBSCRIPTION_ID/resourceGroups/testing/providers/Microsoft.Network/applicationGateways/nia-testing]
 
Apply complete! Resources: 1 added, 0 changed, 0 destroyed.
2022-02-01T15:01:24.812-0500 [INFO]  ctrl: task completed: task_name=testing
$ consul-terraform-sync -config-file cts-config-path.hcl
 
2022-07-11T14:09:17.472-0400 [INFO]  cli: v0.6.0 (7f0d7ba)
 
## omitted for clarity
 
2022/07/11 14:09:20 [INFO] running Terraform command:/terraform-azurerm-application-gateway-nia/examples/terraform init -no-color -force-copy -input=false -backend=true -get=true -upgrade=false
Initializing modules...
 
## Omitted for clarity
 
Initializing the backend...
 
## omitted for clarity
 
Terraform has been successfully initialized!
 
## omitted for clarity
 
2022/07/11 14:09:27 [INFO] running Terraform command:/terraform-azurerm-application-gateway-nia/examples/terraform apply -no-color -auto-approve -input=false -lock=true -parallelism=10 -refresh=true
 
Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create
 
Terraform will perform the following actions:
 
  # module.testing.azurerm_application_gateway.service will be created
 
## omitted for clarity
 
Plan: 1 to add, 0 to change, 0 to destroy.
 
## omitted for clarity
 
module.testing.azurerm_application_gateway.service: Creation complete after 13m55s [id=/subscriptions/SUBSCRIPTION_ID/resourceGroups/testing/providers/Microsoft.Network/applicationGateways/nia-testing]
 
Apply complete! Resources: 1 added, 0 changed, 0 destroyed.
2022-02-01T15:01:24.812-0500 [INFO]  ctrl: task completed: task_name=testing

After running CTS, access the services using the load balancer’s frontend public IP address. Make an API call to the web service at the /web path. Its response includes the IP address of the web service.

$ curl -H 'Content-Type:application/json' http://<load balancer public IP>/web/                                 
{
  "name": "web",
  "uri": "/web/",
  "type": "HTTP",
  "ip_addresses": [
    "10.0.2.4"
  ],
  "start_time": "2022-02-01T21:59:47.028325",
  "end_time": "2022-02-01T21:59:47.028435",
  "duration": "109.599µs",
  "body": "Hello from Web",
  "code": 200
}
$ curl -H 'Content-Type:application/json' http://<load balancer public IP>/web/                                 
{
  "name": "web",
  "uri": "/web/",
  "type": "HTTP",
  "ip_addresses": [
    "10.0.2.4"
  ],
  "start_time": "2022-02-01T21:59:47.028325",
  "end_time": "2022-02-01T21:59:47.028435",
  "duration": "109.599µs",
  "body": "Hello from Web",
  "code": 200
}

Consul-Terraform-Sync runs as a daemon, monitoring for changes to the services in Consul’s catalog. For example, uncomment the second half of setup/example/api.tf to create a second instance for the API service. CTS detects the IP address of the new virtual machine and runs the Terraform module. Terraform adds the IP address (10.0.2.6) to the API service’s backend address pool.

2022-02-01T16:44:37.995-0500 [DEBUG] driver.terraform: change detected for task: task_name=testing
2022-02-01T16:44:38.017-0500 [INFO]  ctrl: executing task: task_name=testing
2022/02/01 16:44:38 [INFO] running Terraform command: /terraform-azurerm-application-gateway-nia/examples/terraform apply -no-color -auto-approve -input=false -lock=true -parallelism=10 -refresh=true
module.testing.azurerm_application_gateway.service: Refreshing state... [id=/subscriptions/SUBSCRIPTION_ID/resourceGroups/testing/providers/Microsoft.Network/applicationGateways/nia-testing]
 
Terraform used the selected providers to generate the following execution plan. Resource actions are
indicated with the following symbols:
  ~ update in-place
 
Terraform will perform the following actions:
 
  # module.testing.azurerm_application_gateway.service will be updated in-place
  ~ resource "azurerm_application_gateway" "service" {
        id                                = "/subscriptions/SUBSCRIPTION_ID/resourceGroups/testing/providers/Microsoft.Network/applicationGateways/nia-testing"
        name                              = "nia-testing"
        tags                              = {}
        # (7 unchanged attributes hidden)
 
      ~ backend_address_pool {
            id           = "/subscriptions/SUBSCRIPTION_ID/resourceGroups/testing/providers/Microsoft.Network/applicationGateways/nia-testing/backendAddressPools/api"
          ~ ip_addresses = [
              + "10.0.2.6",
                "10.0.2.5",
            ]
            name         = "api"
            # (1 unchanged attribute hidden)
        }
    }
 
Plan: 0 to add, 1 to change, 0 to destroy.
 
## omitted for clarity
 
module.testing.azurerm_application_gateway.service: Modifications complete after 9m43s [id=/subscriptions/SUBSCRIPTION_ID/resourceGroups/testing/providers/Microsoft.Network/applicationGateways/nia-testing]
 
Apply complete! Resources: 0 added, 1 changed, 0 destroyed.
2022-02-01T16:54:34.703-0500 [INFO]  ctrl: task completed: task_name=testing
2022-02-01T16:44:37.995-0500 [DEBUG] driver.terraform: change detected for task: task_name=testing
2022-02-01T16:44:38.017-0500 [INFO]  ctrl: executing task: task_name=testing
2022/02/01 16:44:38 [INFO] running Terraform command: /terraform-azurerm-application-gateway-nia/examples/terraform apply -no-color -auto-approve -input=false -lock=true -parallelism=10 -refresh=true
module.testing.azurerm_application_gateway.service: Refreshing state... [id=/subscriptions/SUBSCRIPTION_ID/resourceGroups/testing/providers/Microsoft.Network/applicationGateways/nia-testing]
 
Terraform used the selected providers to generate the following execution plan. Resource actions are
indicated with the following symbols:
  ~ update in-place
 
Terraform will perform the following actions:
 
  # module.testing.azurerm_application_gateway.service will be updated in-place
  ~ resource "azurerm_application_gateway" "service" {
        id                                = "/subscriptions/SUBSCRIPTION_ID/resourceGroups/testing/providers/Microsoft.Network/applicationGateways/nia-testing"
        name                              = "nia-testing"
        tags                              = {}
        # (7 unchanged attributes hidden)
 
      ~ backend_address_pool {
            id           = "/subscriptions/SUBSCRIPTION_ID/resourceGroups/testing/providers/Microsoft.Network/applicationGateways/nia-testing/backendAddressPools/api"
          ~ ip_addresses = [
              + "10.0.2.6",
                "10.0.2.5",
            ]
            name         = "api"
            # (1 unchanged attribute hidden)
        }
    }
 
Plan: 0 to add, 1 to change, 0 to destroy.
 
## omitted for clarity
 
module.testing.azurerm_application_gateway.service: Modifications complete after 9m43s [id=/subscriptions/SUBSCRIPTION_ID/resourceGroups/testing/providers/Microsoft.Network/applicationGateways/nia-testing]
 
Apply complete! Resources: 0 added, 1 changed, 0 destroyed.
2022-02-01T16:54:34.703-0500 [INFO]  ctrl: task completed: task_name=testing

CTS updates the API service’s backend pool with a second target. The load balancer will round-robin the request between the two API service instances.

Azure Application Gateway shows two targets in its backend pool for the API service and one target for the web service

Each time you create or update web and API service instances, CTS will respond to changes in Consul and re-run the Terraform module.

»Conclusion

Running Consul-Terraform-Sync automates the synchronization of service changes in Consul open source or Enterprise to Azure Application Gateway. Your application gateway dynamically responds to new services and service instances, which removes the need to manually update IP addresses and configurations. By combining a declarative Terraform module and a continuously running CTS controller, you implement a GitOps workflow to update and continuously deliver changes to your Azure networking resources.

For additional information on configuring Consul-Terraform-Sync, review our tutorials on HashiCorp Learn. To secure the Terraform state generated by CTS, run it with Terraform Cloud or Enterprise for secure state management. For a production-ready Consul cluster, try HCP Consul on Azure.

Questions about this post? Post it on our community forum!

Sign up for the latest HashiCorp news