Learn how to build observability as code infrastructure automation using New Relic, Terraform Cloud, and Google’s 4 Golden Signals of SRE.
At New Relic, we are heavy users of HashiCorp Terraform, which we use to provision all kinds of infrastructure and services, including New Relic dashboards and alerts. Terraform providers are a big efficiency driver for us and for our customers that use Terraform to automate New Relic setup in their infrastructures. One of our customers, 27Global, described their efficiency gains through Terraform well:
"Historically, our cloud engineers would need to manually add monitoring to the infrastructure, and our developers would have to build monitoring into the applications. Now, using the Terraform Provider from New Relic, we take advantage of APIs to automatically add monitoring to our applications and infrastructure." — Thomas Martin, Director of Site Reliability at 27Global
In this guide, you’ll learn how to use the New Relic Terraform Provider on Terraform Cloud. We’ll walk through how to provision an alert policy, create four alert conditions, and set up a notification channel. The alert conditions are based on the Four Golden Signals of monitoring introduced in Google’s Site Reliability Engineering book.
Once our New Relic alerts are set up with Terraform, you will be able to integrate Terraform Cloud into your workflow.
With Terraform Cloud, you gain several advantages in maintaining your Terraform configuration locally. Terraform Cloud offers remote execution of your Terraform plans, remote state management, and provides a solid foundation to scale up as your team and infrastructure grow.
If you want to follow along via video to set up Terraform on your local machine first, feel free to check out the getting started video my colleague Jon Thurman put together.
To use this guide, you should have some basic knowledge of both New Relic and Terraform. You will need to create a Terraform Cloud account if you don’t already have one. You will want to create a new GitHub repository so you can sync your code changes to Terraform Cloud. If you haven't deployed a New Relic open source agent yet, install New Relic for your application. Also, install the Terraform CLI.
Start by cloning your new GitHub repository into a directory named terraform-project
, initialize the working directory, and create a Terraform configuration file:
Next, instruct Terraform to install and use the New Relic provider, by setting the terraform
and required_providers
blocks in main.tf:
In this code block, you're setting the required version of Terraform to 0.13.x and setting the New Relic provider to the latest 2.x version. Using the right version constraints for your setup will provide better stability with your Terraform runs.
Now that you've set your Terraform and New Relic provider versions, you need to configure the New Relic provider.
With terraform
all set, configure the New Relic provider
with the following items:
NRAK-
.US
if your account settings page is located at one.newrelic.com
, and EU
if your account is located at one.eu.newrelic.com
.We recommend configuring the New Relic provider using environment variables to help keep sensitive credential information out of your codebase. This also allows for setting your environment variables in your Terraform Cloud workspace.
To get started using environment variables, you will want to set the following in your local environment:
It’s important to note that you will also be adding these environment variables to your Terraform Cloud workspace that you will be set up later in this guide.
Since you are using environment variables to configure the New Relic provider, your provider
block in your main.tf file can be left empty.
By setting these values on the New Relic provider, you're configuring the provider to make changes on behalf of your account through New Relic APIs.
For more information about configuring the New Relic provider, please feel free to check out our official provider documentation.
At this point, it’s a good idea to commit and push your code to your repository before you move on to wiring up your repository in Terraform Cloud.
With your GitHub repository updated with your Terraform configuration code, it’s time to integrate Terraform Cloud into the workflow. Terraform Cloud Workspaces offer a way to organize your infrastructure. You can think of a Terraform Cloud Workspace as a working directory for a collection of infrastructure configurations.
We recommend following Terraform Cloud’s getting started guide to get up and running with a new workspace. The guide provides an example repository to link to the workspace, however, you will want to use the GitHub repository you created earlier.
As part of the process, make sure to configure the environment variables mentioned earlier to the workspace.
Add each of your associated New Relic environment variables using the screenshot below as an example. For sensitive credentials, we recommend checking the “sensitive” checkbox.
After you’ve created your workspace and linked it to your repository, you can start to build out the rest of your New Relic Terraform configuration. As you build out each new resource, you can push up your changes to your repository and then use Terraform Cloud to execute your plan to provision your resources. You can even set up your workspace to automatically pick up your changes and apply your Terraform plan as you push updates.
You may also want to adjust some additional workspace options, such as “auto apply,” which will automatically apply your Terraform plan when you push up changes to your main branch. These settings can be found under Settings > General.
Lastly, you’ll want to make sure your workspace is configured to use the same version of Terraform as you configured in your main.tf file. The Terraform version specified in your main.tf file was Terraform version 0.13.x, so you will want to set your workspace’s Terraform version to the latest 0.13.x, which at the time of writing is 0.13.6.
Now that you have a Terraform Cloud workspace and your New Relic provider is configured, initialize Terraform:
terraform init
When Terraform finishes installing and registering the New Relic provider, you'll receive a success message and some actionable next steps, such as running terraform plan
. But thanks to Terraform Cloud, manually running Terraform commands is unnecessary other than to potentially test your configuration with a dry run.
With the New Relic provider configured and successfully initialized, you can define an alerting strategy for your application.
Since you're targeting a specific application, use a newrelic_entity
to fetch the application information from New Relic and allow us to reference that data elsewhere in the configuration:
Next, create a newrelic_alert_policy
. Give the policy a dynamic name based on your application's name. This helps specify the scope of the policy:
At this point, you should be able to test your configuration with a dry run:
terraform plan
You will see some output that displays Terraform's execution plan if you were to run terraform apply
yourself, but we’re only running this manually to ensure things are set up properly.
In this case, the plan shows you that Terraform will create a new alert policy when you run terraform apply
, but we’re going to let Terraform Cloud handle that for us.
Instead of running terraform apply
, commit your code changes and push to your repository.
With your updates pushed up, queue a Terraform run in your workspace:
You will see similar terraform plan
output, but this time it will be within the Terraform Cloud UI. If you have the “auto apply” settings turned on, a successful Terraform plan will result in your plan getting automatically applied, otherwise you will need to use the UI to apply the plan.
Once you’ve applied your Terraform plan, log in to New Relic and navigate to Alert Policies to confirm that Terraform created your new policy.
As you move through the next steps of creating alert conditions, you can commit and push your changes and then use Terraform Cloud to execute your Terraform plan and apply the updates. After each update, refresh your alert policy webpage to see the new alert conditions added to your policy.
Next, add alert conditions for your application based on the four golden signals: latency, traffic, errors, and saturation. Apply these alert conditions to the alert policy you created in the previous step.
Most folks want to avoid slow response times. You can create a newrelic_alert_condition
that triggers if the overall response time of your application rises above five seconds for five minutes:
Note that you're linking this alert condition to the previously configured alert policy with policy_id
.
Traffic represents how much demand is placed on your system at any given moment. Throughput is a metric that measures how much traffic goes to your application. Create a newrelic_alert_condition
that triggers if the overall response rate of your application falls below five requests per minute for five minutes:
This type of alert is useful when you expect a constant baseline of traffic throughout the day — a drop off in traffic can indicate a problem.
If your application's error rate spikes, you need to know about it. Create a newrelic_alert_condition
that triggers if your application's error rate rises above 5% for five minutes:
Saturation represents how "full" your service is and can take many forms, such as CPU time, memory allocation, or queue depth. In this example, assume you already have a New Relic Infrastructure agent installed on the hosts serving your application, and you want to configure an alert for when CPU utilization spikes above a certain threshold:
For the Infrastructure alert, you created a newrelic_infra_alert_condition
that triggers if the aggregate CPU usage on these hosts rises above 90% for five minutes.
Now that you've configured some important alert conditions, add a notification channel to your alert policy to ensure the proper folks get notified when an alert triggers. To do so, use a newrelic_alert_channel
.
To begin, create an email notification channel to send alert notifications to your email. Use this when you want to notify a specific person or team when alerts are triggered:
If you want to specify multiple recipients
, use a comma-delimited list of emails.
Last, but not least, in order to apply the notification channel to your alert policy, create a newrelic_alert_policy_channel
.
A newrelic_alert_policy_channel
links the notification channel you just created to your alert policy.
To finalize your golden signal alerts configuration, commit and push up your changes, then queue your plan again one last time from Terraform Cloud to make sure everything is up to date.
new_relic_alert_channel
supports several types of notification channels, including: email, slack, and pagerduty. So, if you want to explore this more, try creating an alert channel for a second channel type, such as Slack:
Before you apply
this change, you need to add the New Relic Slack App to your Slack account and select a Slack channel to send the notification. With this new alert channel, triggered alerts send notifications to the Slack channel of your choice.
As your team evaluates the alerting system you’ve put in place, you’ll find that you may need to tweak configuration values, such as the alert threshold and duration. If you manage your Terraform project in a remote repository, you can submit a pull request so your team can review these changes alongside the rest of your code contributions.
In addition to maintaining your infrastructure code with a version control system, Terraform Cloud provides a safe and secure way to deploy your infrastructure changes. It also provides a great mechanism for managing your organization’s teams.
For more information regarding various approaches to building a CI/CD pipeline with GitHub and Terraform Cloud, HashiCorp offers a practitioner's guide to help you figure out which approach would work best for your organization and a recommended practices guide to learn more about their recommended workflow and how to evolve your provisioning practices.
Congratulations! You're officially practicing observability as code. Ready for more advanced New Relic guides and tutorials? Visit the New Relic developer site.
Version 5.0 of the HashiCorp Terraform AWS provider brings improvements to default tags, allowing practitioners to set tags at the provider level.
Learn how HashiCorp Terraform supports the deployment of Azure Linux container host for Azure Kubernetes Service (AKS).
New CI/CD pipeline templates for GitHub Actions and GitLab CI provide prescriptive guides for integrating with Terraform Cloud, and a new integration tool can help build workflows with other CI/CD platforms.