Building on Top of HashiCorp Nomad’s Event Stream

HashiCorp Nomad 1.0 includes a new feature called Event Stream: a function that allows operators to view and subscribe to a single unified timeline that streams all high-level events to better understand how the Nomad cluster is performing. By providing stronger tracing and debuggability, Event Steam helps operators unlock a new way to build applications on top of Nomad.

» Before the Event Stream

Before the Event Stream, subscribing to and monitoring for changes within a Nomad cluster required the use of blocking queries. Blocking queries can be used to wait for a potential change of a particular endpoint using long polling.

Blocking queries have a few drawbacks as they relate to subscribing to a set of changes within a cluster. One drawback is that blocking queries have no guarantee of a change. It’s possible that the query may time out or that there was an idempotent write that does not affect the result of the query that was being made. Another pain point is that blocking queries require coordination on the client if you want to monitor multiple types of requests such as Deployments and Jobs. A blocking query for each must be made, and then responses from each endpoint need to be strung together by the user.

With the event stream, we believe that Nomad now has a way to easily subscribe to a set of changes within a cluster through a single endpoint and a single request.

» The Nomad Event Stream

When enabled, Nomad will create a set of events when a state change occurs in Nomad. State changes occur in Nomad via its Finite State Machine (FSM). When a request is made to update a value in Nomad, like registering a new Job, the underlying state store is updated through the FSM with the updated values. In Nomad 1.0, these updated values are now converted into events and pushed onto the server’s event broker. The event broker must be enabled by setting enable_event_broker in the server’s configuration block. Since events are sourced via the FSM this means that every server will have an identical set of events in its broker.

» What Does an Event Look Like?

Each event contains a Topic, Type, Key, Namespace, FilterKeys, Index, and Payload. The contents of Payload depend on the event Topic. Responses from the /v1/event/stream API are new line delimited JSON objects (ndjson). Since multiple values can be updated in a single raft transaction, each API response contains an Index and set of Events.

{
  "Index": 7,
  "Events": [
    {
      "Topic": "Node",
      "Type": "NodeRegistration",
      "Key": "ccc4ce56-7f0a-4124-b8b1-a4015aa82c40",
      "Namespace": "",
      "FilterKeys": null,
      "Index": 7,
      "Payload": {
        "Node": {... entire node object}
      }
    }
  ]
}

» Accessing the Event Stream

The event stream currently exists in the API so users can access the new event stream using an http client like curl or programmatically using the Nomad API package.

The following would subscribe to all events

$ curl -s -v -N http://127.0.0.1:4646/v1/event/stream

You can filter on certain topics by specifying a topic query param, where the key to the topic parameter is the event topic to subscribe to, and the value is the key to filter on.

$ curl -s -v -N \
  --data-urlencode "topic=Node:ccc4ce56-7f0a-4124-b8b1-a4015aa82c40" \
  --data-urlencode "topic=Deployment" \
  --data-urlencode "topic=Job:web" \
  http://127.0.0.1:4646/v1/event/stream

The Go package to use Nomad’s API makes it easy to subscribe to the event stream from your Go application. The following code segment shows how you can use the API to subscribe to Deployment events.

func main() {
    client, _ := api.NewClient(&api.Config{})

    eventsClient := client.EventStream()

    // Subscribe to all Deployment events
    topics := map[api.Topic][]string{
        api.TopicDeployment: {"*"},
    }

    // Begin stream
    ctx := context.Background()
    eventCh, err := eventsClient.Stream(ctx, topics, 0, &api.QueryOptions{})
    if err != nil {
        fmt.Printf("received error %s", err)
        os.Exit(1)
    }

    for {
        select {
        case <-ctx.Done():
            return
        case event := <-eventCh:
            if event.Err != nil {
                fmt.Printf("received error %s", err)
                break
            }

            // Ignore heartbeats used to keep connection alive
            if event.IsHeartbeat() {
                continue
            }

            for _, e := range event.Events {
                deployment, err := e.Deployment()
                if err != nil {
                    fmt.Printf("received error %s", err)
                    continue
                }

                fmt.Printf("Deployment update %s", deployment.Status)
            }
        }
    }
}

To see another example take a look at this slackbot integration to send deploy notifications.

» Share Your Feedback

We are excited to hear from the community on what you intend to build on top of the event stream! If you have questions, comments, or feedback around the event stream or are interested in a new type of event let us know on discuss!