Consul 1.10 can transparently intercept and redirect traffic to sidecar proxies in the service mesh to allow applications to be deployed without modification.
Transparent proxy, a new feature in Consul 1.10, provides an easier onboarding experience for deploying applications with Consul service mesh on Kubernetes. With transparent proxy, traffic is automatically redirected from a service within the application to the service's sidecar proxy. Previously, application owners needed to make slight modifications to their applications to explicitly dial the sidecar proxy deployed alongside the application. This enabled these applications to communicate to other services within the mesh.
Transparent proxy ensures that both inbound and outbound application traffic is directed through the proxy by capturing and redirecting traffic to the Envoy sidecar, which then proxies the connection to its appropriate destination. This eliminates the need to modify an application to communicate through a proxy and removes the need for defining application upstreams, allowing the user to quickly realize the benefits of using a service mesh.
Utilizing transparent proxy with a securely configured service mesh ensures that application communication is encrypted. When properly enabled, mTLS is enforced and authorization policies, such as service intentions, are used to control access to services and access to the types of requests allowed.
Before transparent proxying, the upstreams for service to service communication needed to be explicitly declared via annotations, for example,
consul.hashicorp.com/connect-service-upstreams: api:1234. These annotations would configure the local Envoy proxy with a listener at
localhost:1234 that forwards traffic to the
api service. The service would then have to explicitly use
localhost:1234 whenever addressing the
In Kubernetes, services commonly communicate to each other using hostnames resolvable by Kubernetes DNS (KubeDNS). A Kubernetes service named
web can talk to a Kubernetes service named
api using the hostname
api.svc.cluster.local. To enable this, each service is given a unique, virtual IP that’s routable within the Kubernetes cluster called a clusterIP, that is tied to the Kubernetes Service.
With Consul 1.10, services using transparent proxy utilize the Kubernetes DNS hostnames will now have their traffic routed natively within the mesh. Let’s walk through the proposed datapath in an example:
web is making a request to Service
api, using the KubeDNS address.
------ api.svc.cluster.local ---> Service
web: In the application’s request, the DNS lookup for
api.svc.cluster.local returns 10.20.30.40, so the application connects to 10.20.30.40.
web Pod: Iptables rules in the Pod sees outbound traffic from the application, and redirects it to the Envoy sidecar’s port for outbound traffic. (See Consul Connect Redirect Traffic for more details)
web Envoy: The Envoy sidecar configuration for service
web has the following
filter_chain on its outbound listener to match the outbound traffic based on the packet’s destination IP, to tell it which Envoy cluster (Service
api’s Envoy cluster) to direct the traffic to. This will send the traffic to a Pod for Service
"address_prefix": "10.244.0.38", # Service api Pod IP
"address_prefix": "10.97.110.136", # Service api cluster IP
"filters": [ # If traffic matches above IPs, this filter will run
# This filter sends traffic to the Envoy cluster for service api
api Pod: Iptables rules in the Pod see inbound traffic for service
api and redirect it to the Envoy sidecar’s port for inbound traffic.
api: Receives the request from Service
web. The request was successfully transparently proxied securely through the mesh.
For virtual machines, the
consul connect redirect-traffic command can be utilized to configure traffic redirection through an inbound and outbound listener on the Envoy proxy. In addition, to communicate with upstream service instances on virtual machines, a service could either connect to the IP addresses of other service instances directly (via the DialedDirectly in the Service Defaults configuration entry) or utilize a pre-configured virtual IP configured as a
virtual tagged addresses on the sidecar service registration, which will allow Consul to load balance traffic across the available upstream service instances.
Transparent Proxy is enabled by default within Consul on Kubernetes starting with Consul 1.10 and Consul Helm 0.32.0. Users can optionally control Transparent Proxy globally via the Helm stanza below:
In addition, users can opt-in services to utilize transparent proxy on a per-service basis via the Kubernetes annotation
consul.hashicorp.com/transparent-proxy: true. If transparent proxy is not enabled globally, then only the
connect-inject annotation will need to be enabled on the pod to utilize transparent proxy.
- protocol: TCP
# ‘consul.hashicorp.com/transparent-proxy’: ‘true’ - only required if
# Transparent Proxy is disabled as a global setting
# This name will be the service name in Consul.
- name: static-server
- -text="hello world"
- containerPort: 8080
# If ACLs are enabled, the serviceAccountName must match the Consul service name.
After a service has been deployed, you can use an intention to define which upstream service it should communicate to. A
consul.hashicorp.com/connect-service-upstreams annotation is no longer required to define the upstream service, as the upstream service is now inferred directly from the Consul Service Intention:
- name: static-client
By default, Consul’s transparent proxy allows services to connect to any external service, and Envoy will act as a pass-through proxy for services outside the mesh. This allows users to install Consul and enable transparent proxying without breaking connectivity to external services. Using transparent proxy in conjunction with terminating gateways and service intentions, traffic can also be forced to egress through terminating gateways, ensuring to secure communication with services outside the network.
Alternatively, if third-party external service communication is not allowed due to internal policy, users can also disable this pass-through mode on the entire Consul datacenter using a
mesh custom resource definition (CRD), which was also introduced in Consul 1.10. Users can now utilize a
meshDestinationsOnly option to configure sidecar proxies operating in transparent mode to proxy traffic to only services registered within the mesh.
Headless services in Kubernetes represent services that do not have an allocated ClusterIP set within a Kubernetes service resource. The use case for headless services arises when a user needs to communicate with all individual pods in a service (such as stateful set database deployment) directly as opposed to load balancing traffic over a single IP. A user can now define whether a specific service should be dialed directly via the Service Defaults CRD, or configure this for all services within a Consul namespace via the Proxy Defaults CRD.
Transparent proxy for Consul also provides features that allow for exempting additional traffic from redirection. One example of utilizing transparent proxy with traffic redirection exemption involves deploying ingress controllers on Kubernetes and using sidecars to facilitate direct communication with a service mesh. Previously, ingress controllers required a direct integration with Consul to communicate securely with services in the mesh.
Consul now supports the following annotations to define the types of traffic that should be exempted from traffic redirection:
consul.hashicorp.com/transparent-proxy-exclude-inbound-ports: Provides the ability to exclude a list of ports for inbound traffic that the service exposes from redirection
consul.hashicorp.com/transparent-proxy-exclude-outbound-ports: Provides the ability to exclude a list of ports for outbound traffic that the service exposes from redirection.
consul.hashicorp.com/transparent-proxy-exclude-outbound-cidrs: Provides the ability to exclude a CIDR that the service communicates with for outbound requests from redirection.
consul.hashicorp.com/transparent-proxy-overwrite-probes (default: true): Provides the ability to overwrite HTTP health probes of a container to point them to Envoy. This allows Pods that have health probes — such as liveness and readiness probes — to still function even if traffic redirection is enabled.
Transparent proxy enables applications to be more easily deployed into the service mesh, and provides better controls to manage communication with applications inside or outside the service mesh.
For more information about transparent proxy within Consul, please visit our documentation for Transparent Proxy. Learn more about the Consul 1.10 release in the Consul 1.10 announcement blog. To get started with Consul 1.10, follow the Consul Kubernetes Deployment Guide on HashiCorp Learn for installing Consul on Kubernetes, or the Install Consul guide for installing Consul on VMs.
Learn the installation and verification workflow for any Linux distribution that does not include HashiCorp software in its package repository.
Integrating HCP Consul Central with Backstage can enhance your organization’s infrastructure-management capabilities. Here’s how to do it.
Managing multiple clusters of HashiCorp tools can be complicated. Target CLI eases the burden by using context profiles to easily switch between different clusters and environments.