Products & Technology

Transparent Proxy on Consul Service Mesh

Consul 1.10 can transparently intercept and redirect traffic to sidecar proxies in the service mesh to allow applications to be deployed without modification.

Jul 01 2021David Yu, Blake Covarrubias

Transparent proxy, a new feature in Consul 1.10, provides an easier onboarding experience for deploying applications with Consul service mesh on Kubernetes. With transparent proxy, traffic is automatically redirected from a service within the application to the service's sidecar proxy. Previously, application owners needed to make slight modifications to their applications to explicitly dial the sidecar proxy deployed alongside the application. This enabled these applications to communicate to other services within the mesh.

Transparent proxy ensures that both inbound and outbound application traffic is directed through the proxy by capturing and redirecting traffic to the Envoy sidecar, which then proxies the connection to its appropriate destination. This eliminates the need to modify an application to communicate through a proxy and removes the need for defining application upstreams, allowing the user to quickly realize the benefits of using a service mesh.

»Overview

Utilizing transparent proxy with a securely configured service mesh ensures that application communication is encrypted. When properly enabled, mTLS is enforced and authorization policies, such as service intentions, are used to control access to services and access to the types of requests allowed.

Before transparent proxying, the upstreams for service to service communication needed to be explicitly declared via annotations, for example, consul.hashicorp.com/connect-service-upstreams: api:1234. These annotations would configure the local Envoy proxy with a listener at localhost:1234 that forwards traffic to the api service. The service would then have to explicitly use localhost:1234 whenever addressing the api service.

In Kubernetes, services commonly communicate to each other using hostnames resolvable by Kubernetes DNS (KubeDNS). A Kubernetes service named web can talk to a Kubernetes service named api using the hostname api.svc.cluster.local. To enable this, each service is given a unique, virtual IP that’s routable within the Kubernetes cluster called a clusterIP, that is tied to the Kubernetes Service.

With Consul 1.10, services using transparent proxy utilize the Kubernetes DNS hostnames will now have their traffic routed natively within the mesh. Let’s walk through the proposed datapath in an example:

Service web is making a request to Service api, using the KubeDNS address.
- Service web ------ api.svc.cluster.local ---> Service api
Service web: In the application’s request, the DNS lookup for api.svc.cluster.local returns 10.20.30.40, so the application connects to 10.20.30.40.
Service web Pod: Iptables rules in the Pod sees outbound traffic from the application, and redirects it to the Envoy sidecar’s port for outbound traffic. (See Consul Connect Redirect Traffic for more details)
Service web Envoy: The Envoy sidecar configuration for service web has the following filter_chain on its outbound listener to match the outbound traffic based on the packet’s destination IP, to tell it which Envoy cluster (Service api’s Envoy cluster) to direct the traffic to. This will send the traffic to a Pod for Service api.

    {     "filter_chain_match": {      "prefix_ranges": [       {        "address_prefix": "10.244.0.38", # Service api Pod IP        "prefix_len": 32       },       {        "address_prefix": "10.97.110.136", # Service api cluster IP        "prefix_len": 32       }      ]     },     "filters": [ # If traffic matches above IPs, this filter will run      {       "name": "envoy.filters.network.tcp_proxy",       "typed_config": {        "@type": "type.googleapis.com/envoy.extensions.filters.network.tcp_proxy.v3.TcpProxy",        "stat_prefix": "upstream.api.default.dc1",	# This filter sends traffic to the Envoy cluster for service api        "cluster": "api.default.dc1.internal.83d7967b-7b3f-4637-7dfb-dab835242cb4.consul"       }      }     ] },

Service api Pod: Iptables rules in the Pod see inbound traffic for service api and redirect it to the Envoy sidecar’s port for inbound traffic.
Service api: Receives the request from Service web. The request was successfully transparently proxied securely through the mesh.

For virtual machines, the consul connect redirect-traffic command can be utilized to configure traffic redirection through an inbound and outbound listener on the Envoy proxy. In addition, to communicate with upstream service instances on virtual machines, a service could either connect to the IP addresses of other service instances directly (via the DialedDirectly in the Service Defaults configuration entry) or utilize a pre-configured virtual IP configured as a virtual tagged addresses on the sidecar service registration, which will allow Consul to load balance traffic across the available upstream service instances.

»Configuration

Transparent Proxy is enabled by default within Consul on Kubernetes starting with Consul 1.10 and Consul Helm 0.32.0. Users can optionally control Transparent Proxy globally via the Helm stanza below:

connectInject:  transparentProxy:    defaultEnabled: true

In addition, users can opt-in services to utilize transparent proxy on a per-service basis via the Kubernetes annotation consul.hashicorp.com/transparent-proxy: true. If transparent proxy is not enabled globally, then only the connect-inject annotation will need to be enabled on the pod to utilize transparent proxy.

apiVersion: v1kind: Servicemetadata:  name: static-serverspec:  selector:    app: static-server  ports:    - protocol: TCP      port: 80      targetPort: 8080---apiVersion: v1kind: ServiceAccountmetadata:  name: static-server---apiVersion: apps/v1kind: Deploymentmetadata:  name: static-serverspec:  replicas: 1  selector:    matchLabels:      app: static-server  template:    metadata:      name: static-server      labels:        app: static-server      annotations:        'consul.hashicorp.com/connect-inject': 'true'        # ‘consul.hashicorp.com/transparent-proxy’: ‘true’ - only required if         #     Transparent Proxy is disabled as a global setting    spec:      containers:        # This name will be the service name in Consul.        - name: static-server          image: hashicorp/http-echo:latest          args:            - -text="hello world"            - -listen=:8080          ports:            - containerPort: 8080              name: http        # If ACLs are enabled, the serviceAccountName must match the Consul service name.      serviceAccountName: static-server

After a service has been deployed, you can use an intention to define which upstream service it should communicate to. A consul.hashicorp.com/connect-service-upstreams annotation is no longer required to define the upstream service, as the upstream service is now inferred directly from the Consul Service Intention:

apiVersion: consul.hashicorp.com/v1alpha1kind: ServiceIntentionsmetadata: name: client-to-serverspec: destination:   name: static-server sources:   - name: static-client     action: allow

»Controlling Communication to Third-Party Services

By default, Consul’s transparent proxy allows services to connect to any external service, and Envoy will act as a pass-through proxy for services outside the mesh. This allows users to install Consul and enable transparent proxying without breaking connectivity to external services. Using transparent proxy in conjunction with terminating gateways and service intentions, traffic can also be forced to egress through terminating gateways, ensuring to secure communication with services outside the network.

Alternatively, if third-party external service communication is not allowed due to internal policy, users can also disable this pass-through mode on the entire Consul datacenter using a mesh custom resource definition (CRD), which was also introduced in Consul 1.10. Users can now utilize a meshDestinationsOnly option to configure sidecar proxies operating in transparent mode to proxy traffic to only services registered within the mesh.

apiVersion: consul.hashicorp.com/v1alpha1 kind: Mesh metadata:   name: mesh spec:   transparentProxy:     meshDestinationsOnly: true

»Headless Services

Headless services in Kubernetes represent services that do not have an allocated ClusterIP set within a Kubernetes service resource. The use case for headless services arises when a user needs to communicate with all individual pods in a service (such as stateful set database deployment) directly as opposed to load balancing traffic over a single IP. A user can now define whether a specific service should be dialed directly via the Service Defaults CRD, or configure this for all services within a Consul namespace via the Proxy Defaults CRD.

apiVersion: consul.hashicorp.com/v1alpha1 kind: ServiceDefaults metadata:   name: counting   namespace: product spec:   transparentProxy:      dialedDirectly: true

»Traffic Exclusion and Overwriting Health Probes

Transparent proxy for Consul also provides features that allow for exempting additional traffic from redirection. One example of utilizing transparent proxy with traffic redirection exemption involves deploying ingress controllers on Kubernetes and using sidecars to facilitate direct communication with a service mesh. Previously, ingress controllers required a direct integration with Consul to communicate securely with services in the mesh.

Consul now supports the following annotations to define the types of traffic that should be exempted from traffic redirection:

consul.hashicorp.com/transparent-proxy-exclude-inbound-ports: Provides the ability to exclude a list of ports for inbound traffic that the service exposes from redirection
consul.hashicorp.com/transparent-proxy-exclude-outbound-ports: Provides the ability to exclude a list of ports for outbound traffic that the service exposes from redirection.
consul.hashicorp.com/transparent-proxy-exclude-outbound-cidrs: Provides the ability to exclude a CIDR that the service communicates with for outbound requests from redirection.
consul.hashicorp.com/transparent-proxy-overwrite-probes (default: true): Provides the ability to overwrite HTTP health probes of a container to point them to Envoy. This allows Pods that have health probes — such as liveness and readiness probes — to still function even if traffic redirection is enabled.

»Conclusion

Transparent proxy enables applications to be more easily deployed into the service mesh, and provides better controls to manage communication with applications inside or outside the service mesh.

For more information about transparent proxy within Consul, please visit our documentation for Transparent Proxy. Learn more about the Consul 1.10 release in the Consul 1.10 announcement blog. To get started with Consul 1.10, follow the Consul Kubernetes Deployment Guide on HashiCorp Learn for installing Consul on Kubernetes, or the Install Consul guide for installing Consul on VMs.

Kubernetes