Get better observability and debuggability in your Nomad clusters with the new Event Stream feature in Nomad 1.0.
HashiCorp Nomad 1.0 includes a new feature called Event Stream: a function that allows operators to view and subscribe to a single unified timeline that streams all high-level events to better understand how the Nomad cluster is performing. By providing stronger tracing and debuggability, Event Steam helps operators unlock a new way to build applications on top of Nomad.
Before the Event Stream, subscribing to and monitoring for changes within a Nomad cluster required the use of blocking queries. Blocking queries can be used to wait for a potential change of a particular endpoint using long polling.
Blocking queries have a few drawbacks as they relate to subscribing to a set of changes within a cluster. One drawback is that blocking queries have no guarantee of a change. It’s possible that the query may time out or that there was an idempotent write that does not affect the result of the query that was being made. Another pain point is that blocking queries require coordination on the client if you want to monitor multiple types of requests such as Deployments and Jobs. A blocking query for each must be made, and then responses from each endpoint need to be strung together by the user.
With the event stream, we believe that Nomad now has a way to easily subscribe to a set of changes within a cluster through a single endpoint and a single request.
When enabled, Nomad will create a set of events when a state change occurs in Nomad. State changes occur in Nomad via its Finite State Machine (FSM). When a request is made to update a value in Nomad, like registering a new Job, the underlying state store is updated through the FSM with the updated values. In Nomad 1.0, these updated values are now converted into events and pushed onto the server’s event broker. The event broker must be enabled by setting enable_event_broker
in the server’s configuration block. Since events are sourced via the FSM this means that every server will have an identical set of events in its broker.
Each event contains a Topic
, Type
, Key
, Namespace
, FilterKeys
, Index
, and Payload
. The contents of Payload
depend on the event Topic. Responses from the /v1/event/stream
API are new line delimited JSON objects (ndjson). Since multiple values can be updated in a single raft transaction, each API response contains an Index
and set of Events.
{
"Index": 7,
"Events": [
{
"Topic": "Node",
"Type": "NodeRegistration",
"Key": "ccc4ce56-7f0a-4124-b8b1-a4015aa82c40",
"Namespace": "",
"FilterKeys": null,
"Index": 7,
"Payload": {
"Node": {... entire node object}
}
}
]
}
The event stream currently exists in the API so users can access the new event stream using an http client like curl or programmatically using the Nomad API package.
The following would subscribe to all events
$ curl -s -v -N http://127.0.0.1:4646/v1/event/stream
You can filter on certain topics by specifying a topic query param, where the key to the topic parameter is the event topic to subscribe to, and the value is the key to filter on.
$ curl -s -v -N \
--data-urlencode "topic=Node:ccc4ce56-7f0a-4124-b8b1-a4015aa82c40" \
--data-urlencode "topic=Deployment" \
--data-urlencode "topic=Job:web" \
http://127.0.0.1:4646/v1/event/stream
The Go package to use Nomad’s API makes it easy to subscribe to the event stream from your Go application. The following code segment shows how you can use the API to subscribe to Deployment events.
func main() {
client, _ := api.NewClient(&api.Config{})
eventsClient := client.EventStream()
// Subscribe to all Deployment events
topics := map[api.Topic][]string{
api.TopicDeployment: {"*"},
}
// Begin stream
ctx := context.Background()
eventCh, err := eventsClient.Stream(ctx, topics, 0, &api.QueryOptions{})
if err != nil {
fmt.Printf("received error %s", err)
os.Exit(1)
}
for {
select {
case <-ctx.Done():
return
case event := <-eventCh:
if event.Err != nil {
fmt.Printf("received error %s", err)
break
}
// Ignore heartbeats used to keep connection alive
if event.IsHeartbeat() {
continue
}
for _, e := range event.Events {
deployment, err := e.Deployment()
if err != nil {
fmt.Printf("received error %s", err)
continue
}
fmt.Printf("Deployment update %s", deployment.Status)
}
}
}
}
To see another example take a look at this slackbot integration to send deploy notifications.
We are excited to hear from the community on what you intend to build on top of the event stream! If you have questions, comments, or feedback around the event stream or are interested in a new type of event let us know on discuss!
Running Boundary workers as dynamic workloads can be challenging. Using the Nomad and Vault integration along with a custom Vault plugin, this process can be seamlessly automated.
HashiCorp Nomad 1.6 introduces node pools to help manage multi-tenant Nomad clusters, along with a redesigned job UI, Nomad Pack improvements, and more.
Learn about the internals of Nomad's evaluation broker and how we recently reduced scheduler loads by 90% during rapid cluster changes.