Announcing HCP Consul management plane with cloud-based observability

This post was originally published June 13. It has been updated for the general availability of cluster linking.

In October of last year, we introduced a new management plane for HashiCorp Cloud Platform (HCP) Consul to provide global visibility and control for self-managed and HashiCorp-managed clusters. Today we're pleased to announce enhanced capabilities of the management plane for HCP Consul, with the introduction of cloud-based observability, now in beta release. Additionally, customers can now link new and existing self-managed Consul clusters in both virtual machine and Kubernetes environments directly through the HCP Consul management plane to manage Consul infrastructure with a centralized dashboard. Support for HCP Consul cluster peering, which was announced as a beta feature in March, is also now generally available.

Let's delve into the details:

»Cloud-based observability (beta)

Operators often face the complexity of monitoring the reliability of their systems and applications across distributed environments. With the introduction of cloud-based observability features, Consul operators now have the tools they need to gain deeper insights, troubleshoot more effectively, and seamlessly monitor the health of their Consul infrastructure. There are three key elements to this capability:

• Telemetry extraction
• Metrics storage
• Visualizations

»Telemetry extraction

Observability starts with the ability to extract valuable telemetry data from each Consul cluster. The introduction of a Consul telemetry collector empowers operators to effortlessly collect telemetry data from their Consul clusters and Envoy proxies. Based on the OpenTelemetry (OTEL) protocol, the Consul OTEL collector/exporter is a dedicated service that acts as a secure intermediary handling authorization, filtering, and telemetry data routing.

Our adoption of OTEL reinforces our dedication to supporting industry standards and enhancing third-party integration capabilities. It establishes a standardized framework that enables effortless data sharing and collaboration between our systems and the application performance management (APM) tools our customers depend on.

»Metrics storage

To ensure security, scalability, and availability, cloud-based observability is backed by Amazon Managed Service for Prometheus, which is the metric storage solution for Consul. Operators can now offload the burden of managing a metrics storage system and leverage the robustness and scalability of AWS.

Telemetry data is authorized, filtered, and routed to the storage backend following the Prometheus remote_write protocol, which provides a seamless experience for storing and accessing metrics. Security and fault partitioning are further enhanced with sharding across multiple AWS accounts, allowing operators to focus on analyzing data instead of managing underlying infrastructure.

»Visualizations

We understand the importance of an intuitive interface to explore and visualize metrics. Our newly introduced cluster-health dashboard offers a consolidated view of key indicators, simplifying monitoring of the Consul control plane's health and availability. This new view of key indicators can help operators react faster and proactively address potential issues.

»More cluster linking controls and capabilities (GA)

Earlier this year, we announced the general availability of the HCP Consul management plane, which allows Consul users to bootstrap new self-hosted Kubernetes clusters with a secure Consul configuration, view the health status of Consul clusters, and discover services in a unified view whether they are registered with self-managed or HCP-managed Consul. For platform and application development teams, the management plane also provides the ability to securely access the Consul UI directly through the HashiCorp Cloud Platform, eliminating the need to expose the UI in on-premises datacenters under tight network control.

Today we’ve announced that the HCP Consul management plane now lets operators link existing self-managed Consul clusters in both virtual machine and Kubernetes environments using an intuitive UI in HCP. Cluster linking is supported for both open source Consul and Consul Enterprise with the release of Consul 1.14.7 and 1.15.3.

We have also improved the experience of revoking HCP’s access to the operator's self-managed Consul cluster. When operators want to unlink their cluster from HCP, Consul can automatically clean up any HCP-related resources created during the linking process and leave the cluster exactly as it was before the link was established.

»Cluster peering controls (GA)

Cluster peering through the HCP Consul management plane is now generally available. This feature lets operators establish cross-cluster connectivity using a single interface, eliminating the need to log into individual Consul clusters for peering requests. Operators can use this feature to peer their Consul clusters running in public clouds, on-premises datacenters, and HCP.

»Getting started with the HCP Consul management plane

We are excited for users to try these new HCP Consul updates and further expand their service mesh implementations. You can try the new features on the HCP Consul management plane now at no cost and connect either your self-managed or HCP-managed Consul clusters. If you haven’t yet tried HCP Consul, HashiCorp offers a $50 credit so you can get up and running for free. For more information, check out the HCP Consul documentation.