Use Minikube to create multiple Kubernetes clusters with Consul and test cluster peering configurations in your local development environment.
As you scale Kubernetes, you will often find yourself with multiple clusters running different services. HashiCorp Consul’s service mesh on Kubernetes helps register and manage requests between services from a single point of control. But how do you manage traffic between services across multiple Consul datacenters on different Kubernetes clusters? Cluster peering connects Consul datacenters across multiple Kubernetes clusters using administrative partitions, and offers one point of control to manage traffic across Kubernetes clusters.
This post shows how to use Minikube to create multiple Kubernetes clusters in your local development environment to test Consul’s cluster peering and configure services across clusters. Testing locally allows you to verify configurations for services and the requests between services in Consul service mesh before applying changes to your production clusters. The post also dives into a set of Bash scripts to automate the creation and configuration of Kubernetes and Consul clusters that you can adapt for your own local testing environment.
Note that your local development environment must support Docker and offer sufficient resources to run multiple clusters. You can also use a driver that creates clusters in virtual machines. However, not all operating systems support virtual machine drivers. If you create a virtual machine with VirtualBox, you can attach a host-only network to each cluster and configure MetalLB to use that network.
By default, Minikube sets up each Kubernetes cluster on its own Docker bridge network. This allows clusters to communicate with each other but does not reflect the scenario of external access to a Kubernetes service through a load balancer. Attach each Minikube virtual machine to a second bridge network. The load balancers run in the second network to simulate mesh gateways communicating over a public network..
Create a network named metallb
using Docker, with an IP address range of 172.28.0.0/16. Restrict the IP range for Minikube clusters to 172.28.0.0/24 to ensure you do not have overlapping IP addresses for each load balancer:
Each load balancer in each cluster must have its own IP address range so they can route to each other.
Consul’s cluster peering requires a mesh gateway to facilitate communication between datacenters on different Kubernetes clusters. The mesh gateways require a load balancer to facilitate inbound and outbound requests to services in each Kubernetes cluster. For the load balancer, this example uses MetalLB, a Minikube addon, which creates a bare-metal load balancer for your cluster.
Start Minikube with the MetalLB addon and set a profile to indicate a specific Kubernetes cluster, in this example dc1
:
Connect the cluster to the metallb
network:
After the Kubernetes cluster starts, configure the ConfigMap
for MetalLB under addresses
with an IP address range in the subnet you allocated to the Docker network. For example, you can set the third octet of the network to the number of the cluster:
MetalLB will use IP addresses from this range when you request a Kubernetes service with a load balancer. You will need to configure each Kubernetes cluster with its own profile and update MetalLB’s ConfigMap
with an IP address range from the metallb
network.
Each Kubernetes cluster has its own Consul cluster and datacenter. Update the Helm values with a Consul datacenter to match the Minikube profile name. Set up TLS and enable mesh gateways for Consul cluster peering. By default, the Helm chart will create a Kubernetes service with a LoadBalancer
type for the mesh gateway. Install Consul to the consul
namespace:
After Consul runs, configure the mesh to peer through mesh gateways:
Update Consul’s proxies to access services through their local mesh gateway:
Repeat these configurations for each Kubernetes cluster you create through Minikube.
Peer only the clusters that have services communicating with each other in production to maintain least-privilege. The script included in this post peers all clusters to each other for testing purposes. This example peers dc1
to dc2
.
Configure dc1
as the peering acceptor, which creates a peering token and sets the target peer to dc2
:
Extract the peering token and copy it to dc2
:
Set up dc2
as the peering dialer, which initiates the peering request to dc1
:
Create a unique peering token for each pair of peered clusters.
For services to access other services in a peered cluster, export the service from one cluster to another:
Add an intention to allow a service in the peered cluster (dc1) to access the service in the target cluster (dc2):
Once you set up a local testing environment for cluster peering, you can run integration tests for services that need to connect across multiple Consul datacenters and Kubernetes clusters. For a fully automated local testing setup, check out my repository to configure Minikube and Consul across three clusters. Learn more about deploying in our cluster peering for Kubernetes r documentation and deploy an example to Kubernetes clusters on a cloud provider with our tutorial: Connect services between Consul datacenters with cluster peering.
Learn how to use GitOps to deploy and synchronize a Consul cluster on Kubernetes with Argo CD.
Consul 1.16 adds new reliability, scalability, security, service mesh UX, extensibility, and service discovery capabilities.
The HCP Consul management plane now offers deeper insights to your Consul deployments via cloud-based observability and seamlessly links new and existing self-managed Consul clusters.