Get better observability and debuggability in your Nomad clusters with the new Event Stream feature in Nomad 1.0.
HashiCorp Nomad 1.0 includes a new feature called Event Stream: a function that allows operators to view and subscribe to a single unified timeline that streams all high-level events to better understand how the Nomad cluster is performing. By providing stronger tracing and debuggability, Event Steam helps operators unlock a new way to build applications on top of Nomad.
Before the Event Stream, subscribing to and monitoring for changes within a Nomad cluster required the use of blocking queries. Blocking queries can be used to wait for a potential change of a particular endpoint using long polling.
Blocking queries have a few drawbacks as they relate to subscribing to a set of changes within a cluster. One drawback is that blocking queries have no guarantee of a change. It’s possible that the query may time out or that there was an idempotent write that does not affect the result of the query that was being made. Another pain point is that blocking queries require coordination on the client if you want to monitor multiple types of requests such as Deployments and Jobs. A blocking query for each must be made, and then responses from each endpoint need to be strung together by the user.
With the event stream, we believe that Nomad now has a way to easily subscribe to a set of changes within a cluster through a single endpoint and a single request.
When enabled, Nomad will create a set of events when a state change occurs in Nomad. State changes occur in Nomad via its Finite State Machine (FSM). When a request is made to update a value in Nomad, like registering a new Job, the underlying state store is updated through the FSM with the updated values. In Nomad 1.0, these updated values are now converted into events and pushed onto the server’s event broker. The event broker must be enabled by setting enable_event_broker
in the server’s configuration block. Since events are sourced via the FSM this means that every server will have an identical set of events in its broker.
Each event contains a Topic
, Type
, Key
, Namespace
, FilterKeys
, Index
, and Payload
. The contents of Payload
depend on the event Topic. Responses from the /v1/event/stream
API are new line delimited JSON objects (ndjson). Since multiple values can be updated in a single raft transaction, each API response contains an Index
and set of Events.
{
"Index": 7,
"Events": [
{
"Topic": "Node",
"Type": "NodeRegistration",
"Key": "ccc4ce56-7f0a-4124-b8b1-a4015aa82c40",
"Namespace": "",
"FilterKeys": null,
"Index": 7,
"Payload": {
"Node": {... entire node object}
}
}
]
}
The event stream currently exists in the API so users can access the new event stream using an http client like curl or programmatically using the Nomad API package.
The following would subscribe to all events
$ curl -s -v -N http://127.0.0.1:4646/v1/event/stream
You can filter on certain topics by specifying a topic query param, where the key to the topic parameter is the event topic to subscribe to, and the value is the key to filter on.
$ curl -s -v -N \
--data-urlencode "topic=Node:ccc4ce56-7f0a-4124-b8b1-a4015aa82c40" \
--data-urlencode "topic=Deployment" \
--data-urlencode "topic=Job:web" \
http://127.0.0.1:4646/v1/event/stream
The Go package to use Nomad’s API makes it easy to subscribe to the event stream from your Go application. The following code segment shows how you can use the API to subscribe to Deployment events.
func main() {
client, _ := api.NewClient(&api.Config{})
eventsClient := client.EventStream()
// Subscribe to all Deployment events
topics := map[api.Topic][]string{
api.TopicDeployment: {"*"},
}
// Begin stream
ctx := context.Background()
eventCh, err := eventsClient.Stream(ctx, topics, 0, &api.QueryOptions{})
if err != nil {
fmt.Printf("received error %s", err)
os.Exit(1)
}
for {
select {
case <-ctx.Done():
return
case event := <-eventCh:
if event.Err != nil {
fmt.Printf("received error %s", err)
break
}
// Ignore heartbeats used to keep connection alive
if event.IsHeartbeat() {
continue
}
for _, e := range event.Events {
deployment, err := e.Deployment()
if err != nil {
fmt.Printf("received error %s", err)
continue
}
fmt.Printf("Deployment update %s", deployment.Status)
}
}
}
}
To see another example take a look at this slackbot integration to send deploy notifications.
We are excited to hear from the community on what you intend to build on top of the event stream! If you have questions, comments, or feedback around the event stream or are interested in a new type of event let us know on discuss!
HashiCorp adopts the community-created HCL Extension for Visual Studio Code and adds HCL 2.0 support.
Nomad 1.3 includes built-in service discovery, improved edge compute support, Container Storage Interface (CSI) general availability, and more.
The HashiCorp Releases API is now available. This API is your one-stop shop for finding and viewing extended metadata about HashiCorp product releases.