HashiCorp Consul

HashiCorp Consul

Apr 17 2014 Armon Dadgar

Today we announce Consul, a solution for service discovery and configuration. Consul is completely distributed, highly available, and scales to thousands of nodes and services across multiple datacenters.

Some concrete problems Consul solves: finding the services applications need (database, queue, mail server, etc.), configuring services with key/value information such as enabling maintenance mode for a web application, and health checking services so that unhealthy services aren't used. These are just a handful of important problems Consul addresses.

Consul solves the problem of service discovery and configuration. Built on top of a foundation of rigorous academic research, Consul keeps your data safe and works with the largest of infrastructures. Consul embraces modern practices and is friendly to existing DevOps tooling.

Consul is already deployed in very large infrastructures across multiple datacenters and has been running in production for several months. We're excited to share it publicly.

Read on to learn more.

How it Works

;; QUESTION SECTION: ;web-frontend.service.consul. IN ANY

;; ANSWER SECTION: web-frontend.service.consul. 0 IN A 10.0.3.83 web-frontend.service.consul. 0 IN A 10.0.1.109

Consul has integrated health checks, so only healthy nodes and healthy services are returned via the query interfaces. The HTTP API has additional endpoints for finding all services as well as viewing the health state of every service.

Configuration data is set and retrieved using the HTTP API. The API is purposely very simple so that clients are very easy to write:

$ curl -X PUT -d 'bar' http://localhost:8500/v1/kv/foo true $ curl http://localhost:8500/v1/kv/foo [ { "CreateIndex": 100, "ModifyIndex": 200, "Key": "foo", "Flags": 0, "Value": "YmFy" } ]

Strong Foundation

Distributed messaging, leader election, security, data storage are not components anyone should build from scratch. Consul uses highly cited and peer-reviewed papers based on academic research as a foundation for these subsystems.

Messaging is done via gossip over both UDP and TCP connections. For gossip, we rely on Serf, built on top of SWIM. Serf is deployed around the world and powers gossip communication of many thousands of machines. Gossip is used because it is lightweight and performant. TCP is used where we need higher reliability for network traffic.

Leader election uses Raft, a consensus algorithm out of Stanford. Having a robust consensus algorithm makes the auto-healing nature of Consul a pleasure for ops.

Security is a serious concern for infrastructure tools. Consul has a multi-faceted approach. Gossip messaging is secured using a strong symmetric cipher provided by Serf. You can learn more about it here. RPC communication done over TCP can be secured using TLS with optional authentication for both clients and servers.

Data storage uses LMDB, an embedded database that is fully ACID compliant and provides MVCC. These properties keep the data Consul stores safe while also being extremely performant.

HashiCorp Built

At HashiCorp, we build solutions to DevOps problems that are technically sound and are a joy to use. We don't take shortcuts with the technologies we choose, and just as importantly we don't take shortcuts in the experience of using and operating our tools. As a result, HashiCorp-made tools are stable, scalable, and easy to use and operate.

Consul is the fourth such tool we've built. Previously, we've built Vagrant, Packer, and Serf. Consul works great with our other tools, but doesn't require the use of any of them.

Mitchell Hashimoto and Armon Dadgar will be present at GopherCon next week. We'll have stickers for all our products (including Consul). Find us and we'd be happy to talk about any of our tools with you.

Learn More

To learn more about Consul, please visit the Consul website. The following pages in particular are good next steps:

Your browser is out-of-date!

Update your browser to view this website correctly. Update my browser now

×