Using Consul’s transparent proxy on virtual machines

Configure Consul’s transparent proxy on virtual machines to find and connect to services in the service mesh with DNS.

Transparent proxy helps secure services in a service mesh by forcing traffic through a proxy like Envoy to prevent unauthorized direct access to the application. It allows a service to resolve and connect to another service using DNS instead of through proxy upstreams listening on localhost. In HashiCorp Consul, a service can connect to an upstream service using transparent proxy and Consul’s DNS as long as an intention allows traffic between the services.

This post shows you how to configure Consul’s transparent proxy on virtual machines (VMs). Your operating system must support iptables in order for transparent proxy to restrict traffic on the VM. Furthermore, you must run Consul 1.11.1 or later to use transparent proxy. For a full example, check out my Terraform configuration that uses a user data script to set up transparent proxy. While the example uses Terraform to set up Consul with transparent proxy through user data, you can adapt its scripts for your own VM configuration.

»Configure Consul client

Before your service can use transparent proxy, set up a Consul client on your virtual machine with service mesh and DNS forwarding enabled. Make sure Consul runs under the consul Linux user. You will need its user ID to make it easier to exclude Consul-related traffic from transparent proxy redirection.

If you have access control lists (ACLs) enabled, create a Consul ACL token for the node identity. This token includes a policy to write nodes and read services in Consul.

consul acl token create -node-identity=${node_name}

The Consul client configuration sets both agent and default tokens to register the virtual machine and resolve services with Consul for service discovery.

{
  "server": false,
  "acl": {
    "tokens": {
      "agent": "${CONSUL_HTTP_TOKEN}",
      "default": "${CONSUL_HTTP_TOKEN}"
    }
  },
  "connect": {
    "enabled": true
  },
  "ports": {
    "grpc": 8502
  }
}

In order to use Consul service discovery on the virtual machine, forward the virtual machine’s DNS to Consul. When you use Consul service discovery, you can configure your service to look up another service in the mesh using its virtual service name. Set up DNS forwarding for your DNS server. For example, add Consul to the systemd-resolved configuration file and restart the process to forward any DNS requests to the service mesh to Consul.

[Resolve]
DNS=127.0.0.1:8600
Domains=~consul

Start the Consul client. The virtual machine should register as a node in Consul. Before registering the service and starting the proxy, create a user for the proxy named envoy. Consul needs this user ID in order to exclude traffic to and from the proxy.

»Register the service

Define a service to add it to the mesh and enable transparent proxy. The service definition sets the service’s name, port, and proxy mode.

{
 "service": {
   "connect": {
     "sidecar_service": {
       "proxy": {
         "mode": "transparent"
       }
     }
   },
   "name": "${service_name}",
   "port": ${service_port}
 }
}

Consul assigns a virtual IP in the 240.0.0.0/4 range to a service with transparent proxy enabled. This allows downstream services to connect to upstream services using a unique IP assigned by the service mesh. You can register the service with the Consul API or add the service definition to the Consul configuration directory on your VM. Adding the service definition to the configuration directory registers the service each time the Consul client restarts.

»Redirect traffic to proxies

After installing and configuring Consul and Envoy, redirect traffic to and from the virtual machine using transparent proxy. If you have a mesh configuration entry defined, check if TransparentProxy.MeshDestinationsOnly is true. This setting will deny all inbound and outbound requests to the VM once you redirect traffic. As a result, make sure you install the application and its dependencies before redirecting traffic since transparent proxy will deny requests to download packages by default.

Issue the consul connect redirect-traffic command to enable transparent proxy on the VM for the named service. The command should include the user ID for consul and envoy in order for transparent proxy to exclude traffic from Consul and Envoy from redirection.

consul connect redirect-traffic \
   -proxy-id="${service_name}-sidecar-proxy" \
   -proxy-uid="$(id --user envoy)" \
   -exclude-uid="$(id --user consul)"

Once you issue the command, any connection you have to the VM (including SSH) will disconnect. The command adds iptables rules that will deny existing inbound traffic unless you have specified exclusions. If you need to exclude inbound or outbound traffic from specific ports or IP addresses, append additional arguments specifying exclusions. For example, you may need to exclude inbound traffic to the SSH port for login:

consul connect redirect-traffic \
   -proxy-id="${service_name}-sidecar-proxy" \
   -proxy-uid="$(id --user envoy)" \
   -exclude-uid="$(id --user consul)" \
   -exclude-inbound-port=22

Next, install and run Envoy on the VM. Issue the consul connect envoy command for Consul to run Envoy. If you have ACLs enabled, make sure that you create a service token and pass it to the command using the -token argument.

consul connect envoy -sidecar-for=${service_name} -token=${CONSUL_HTTP_TOKEN}

To automate transparent proxy configuration, you can add a script enabling traffic redirection to the proxy’s system file before you start the proxy. After stopping the proxy, you may want a script to clean up iptables. Use a systemd unit file to use the Consul ACL token from an environment file and redirect traffic before starting the proxy:

[Unit]
Description=Consul Envoy
After=syslog.target network.target consul.service
Wants=consul.service
 
ConditionFileIsExecutable=<script for redirecting traffic>
ConditionFileIsExecutable=<script for cleaning up iptables>
 
[Service]
Type=simple
User=envoy
Group=envoy
EnvironmentFile=/etc/consul.d/consul.env
ExecStartPre=+<script for redirecting traffic> ${service_name}
ExecStart=/usr/bin/consul connect envoy -sidecar-for=${service_name}
ExecStopPost=+<script for cleaning up iptables>
Restart=on-failure
RestartSec=5
WorkingDirectory=/etc/consul.d/
 
[Install]
WantedBy=multi-user.target

Review the example for scripts to set up traffic redirection and clean up iptables. After starting the proxy, the service can now connect to another service in the mesh.

»Connecting services

If you have Consul ACLs enabled, you have a service mesh that denies all traffic between services by default. Make sure you have an intention authorizing communication between your source and destination services. The example uses Terraform to define an intention for the API service that allows inbound traffic from the web service:

resource "consul_config_entry" "service_intentions" {
 name = "api"
 kind = "service-intentions"
 
 config_json = jsonencode({
   Sources = [
     {
       Action     = "allow"
       Name       = "web"
       Namespace  = "default"
       Partition  = "default"
     }
   ]
 })
}

Once you define an intention, log into your VM for a downstream service. A service can resolve the in-mesh IP of an upstream service using a DNS hostname in the format of ${service_name}.virtual.consul. For example, the VM hosting the web service resolves the API service’s virtual IP:

$ nslookup api.virtual.consul
Server:         127.0.0.53
Address:        127.0.0.53#53
 
Non-authoritative answer:
Name:   api.virtual.consul
Address: 240.0.0.2

The intention allows the web service to connect to the API service:

$ curl --head http://api.virtual.consul
HTTP/1.1 200 OK
Date: Thu, 16 Mar 2023 19:50:23 GMT
Content-Length: 243
Content-Type: text/plain; charset=utf-8
Connection: close

»Next steps

Once you try transparent proxy on a VM, check out my Terraform module to automate the setup of transparent proxy. The module takes variables and generates cloud-init user data you can pass to your VM to configure a Consul client, proxy, and application on startup. For more on transparent proxy, including its use for Kubernetes services, review the Enable transparent proxy mode documentation.


Sign up for the latest HashiCorp news

By submitting this form, you acknowledge and agree that HashiCorp will process your personal information in accordance with the Privacy Policy.