Consul 1.9 introduces new service mesh visualization tools that help users troubleshoot their mesh and view key metrics.
Configuring and managing a service mesh is complex, involving a series of steps to set up proxies and upstreams correctly. HashiCorp Consul users have relied on error logs and command line output to be able to troubleshoot the Consul service mesh, but Consul 1.9 introduces a new approach. Our new topology diagram helps users visualize and troubleshoot mesh connections and key metrics. In this blog post, we'll walk through an example that illustrates how to use the new topology diagram feature to effectively debug and visualize a service mesh.
To set up Consul as a service mesh, start by enabling connect
on Consul servers. Then, add a connect
stanza to each service. This tells Consul to configure a sidecar proxy for this service. Next, start the proxies for each service. Then add upstreams to each service to identify each service’s dependencies.
Finally, write intentions between services that should be allowed to connect. In production, your default policy should be set to deny
, so you should need to explicitly set intentions for each connection you want to allow.
Now that you‘ve configured your service mesh, how can you tell if you’ve set it up properly? In Consul 1.9.0, we’re introducing a new topological diagram that will help you visualize the connections in your service mesh.
In the Consul UI, from the services page, navigate to a service. In this example, we’ll use the app
service. As long as you have connect
enabled, you’ll be able to see a Topology tab, which will show which services can connect to app
(downstreams), and which services app
can connect to (upstreams).
This visualization is great for understanding the authorized connections between services and can also be helpful for troubleshooting when there are service communication failures. To get started troubleshooting, we want to provide a couple common failure scenarios and how to resolve them.
First, if you aren't seeing the services you expected in the diagram, we recommend checking that the service's upstreams are configured properly by reviewing the service definition.
Second, if there is a red line and red icon on the connection, you may have a deny
intention between the service and its upstream.
If you hover over that red icon, you’ll see an option to fix the connection by adding an allow
intention.
Finally, you can use this service mesh visualization to see a few basic metrics. There are two ways to add metrics to the mesh topology diagram. One way is to configure Prometheus to integrate with the Consul UI. With this integration, you can see stats like request rate, error rate, and latency for your service and its adjacent connections.
To set this up, add a ui_config
stanza to the agent config file, with metrics provider details:
The second way to add metrics to the mesh topology diagram is to add a custom metrics link to the agent config file. While this won’t show data directly on Consul’s topology diagram, it will provide your team with a metrics link on the diagram for each service in Consul. This way, operators and developers will be able to troubleshoot problems quickly from Consul by clicking the accessible link right from the service detail page.
To set this up, add a dashboard template URL into that ui_config
stanza in your agent config file (see docs).
Consul’s new mesh topology diagram helps users visualize and troubleshoot their service mesh configurations. To learn more and get started with Consul Enterprise or request a free trial, visit the Consul product page.
Attending KubeCon EU, either in person or online? Check out what HashiCorp is doing and talking about at the event, and learn about recent Kubernetes-related product features.
Configure Consul’s transparent proxy on virtual machines to find and connect to services in the service mesh with DNS.
HCS on Azure has been deprecated. HCP Consul on Azure is the preferred way to run your HashiCorp Consul clusters on Microsoft Azure.