Creating a multi-cloud golden image pipeline with Terraform Cloud and HCP Packer

In today’s multi-cloud world, images (such as AMIs for Amazon EC2, virtual machines, Docker containers, and more) lay the foundation for modern infrastructure, security, networking, and applications. Enterprises adopting multi-cloud typically start by using Terraform for centralized provisioning, but Terraform does not handle the details of image creation and management.

In many organizations, the workflows in place to create and manage images are siloed, time-consuming, and complex, leading to slow spin-up times and human errors that pose security risks. Organizations need standard processes to ensure all images throughout their infrastructure estate are secure, compliant, and easily accessible.

»Image management challenges

As organizations deploy fleets of images to support services across cloud and private environments, the complexity and scope of these services often involve multiple different teams. Without consistent, central processes and tooling in place, organizations can experience variability in their image workflows creating several challenges:

• Inconsistencies: With different image practices, teams are prone to achieve different outcomes with varying levels of infrastructure performance.
• Risks: Manual, checklist-driven procedures to apply security standards lead to human error in the form of misconfigured and insecure images that can introduce security threats to the organization and result in outages.
• Delays: Teams may duplicate efforts and spend excessive time manually building images for different environments, increasing time to deployment.

To combat these issues, organizations and their platform teams need to establish a central shared service for their image creation and management workflows.

»Solution: Creating a golden image pipeline

Implementing a golden image pipeline with Terraform Cloud with HCP Packer unifies provisioning and image workflows to shift security left and automate image management across downstream builds and provisioning pipelines. HCP Packer helps platform teams establish a unified workflow management system across groups within an organization. This provides policy and governance, organization-wide visibility, ease of integration with peripheral technologies, and overall reliability at scale.

By integrating HCP Packer into their Terraform Cloud workflows, organizations can:

1. Standardize image creation and ensure all builds deployed are secure and compliant
2. Track all image builds and associated metadata in a central artifact registry
3. Automate provisioning pipelines and continuously monitor infrastructure health
4. Simplify image lifecycle management

»Step 1. Standardize image creation with Packer

The first step in creating a golden image pipeline is to create a set of golden images with HashiCorp Packer. A “golden image” is an approved image that acts as a template on top of which developers can build applications. These images contain the most up-to-date common system packages, logging and monitoring tools, security patches, and configuration hardening. Packer simplifies golden image creation by enabling organizations to leverage the HashiCorp Configuration Language (HCL). HCL simplifies the process of embedding all organizational requirements —such as security and operational details — into golden images. Codification also enables collaboration; changes can be reviewed by the appropriate stakeholders using standard version-control workflows before being implemented.

Packer configurations are defined using templates that enable users to leverage common configurations across multiple image builds. Templates consist of a series of declarations and commands for Packer to follow when generating a new image build. The template specifies what plugins (builders, data sources, provisioners, post-processors) to use, how to configure each of those plugins, and in what order to run them.

»Step 2. Track images at scale in the HCP Packer artifact registry

When a new golden image is created, this new version is automatically published to HCP Packer by including a simple hcp_packer_registry block in the template. HCP Packer serves as a managed registry that stores image metadata, including when they were created, the associated cloud provider, and any custom labels specified in your image build. The HCP Packer artifact registry helps you track information about images, clearly designate which versions are approved for consumption, and query the right images to use in both Packer and Terraform configurations. Access to this centralized library helps align the workflows of image creation and deployment, allowing operations and development teams to work together to manage, track, and govern all artifacts across your infrastructure estate.

»Manage images with channels

A core feature of HCP Packer that enables collaboration across teams is image channels. With channels, you can label image versions to describe the quality and stability of a build. By assigning human-readable names, downstream consumers can easily reference the images in Packer templates and Terraform configurations. For example, you can designate a specific channel for testing, allowing users to promote new versions and quickly spin up an instance to validate the image. Once the new version passes the required tests, it can be promoted to the stable channel, alerting downstream consumers that it is approved and ready for deployment. This workflow provides teams with vetted, ready-built artifacts that supply standard services in a plug-and-play fashion. Consumers can tailor versions of artifacts to streamline their efforts in the updating and release stages and ensure they are referencing the latest version without having to update their code directly. This image promotion workflow can also be automated in a declarative fashion using the HCP provider for Terraform.

»Step 3. Automate provisioning and monitor health with Terraform Cloud

With a golden image built, published, validated, and promoted to your organization's stable channel, Terraform runs referencing the updated version can now be queued automatically for any workspace using the channel. The image updates across downstream provisioning pipelines can take place autonomously with auto-apply settings or be gated by manual approval processes. The Terraform Cloud run task for HCP Packer helps prevent the deployment of non-approved images with:

• Data source image validation to scan your Terraform plan for references to the HCP Packer iteration and image data sources, warning you or blocking the run if any referenced data is associated with a revoked image version
• Resource image validation to scan your Terraform configuration for resources that use hard-coded machine image IDs and check if the image is tracked by HCP Packer. It will also warn users if the image is associated with a revoked iteration and prompt them to reference the HCP Packer data source instead for better tracking and management capabilities

With this automation, teams can integrate images easily onto a larger workflow framework to complement automated delivery pipelines.

»Drift detection and continuous validation

With the new image version successfully approved and provisioned, the next step is to perform health assessments to make sure this infrastructure doesn't change over time. Even with a standardized initial provisioning process, settings on infrastructure can still be modified or circumvented, opening up your infrastructure to the possibility of configuration drift. Drift is the term for when the real-world state of your infrastructure differs from the state defined in your configuration. Drift occurs when a user modifies resources outside of the Terraform workflow.

Terraform Cloud’s drift detection allows users to actively monitor their infrastructure for these changes and receive alerts when they take place. From the health assessments dashboard they can quickly uncover the root cause for the change, determine if it is necessary, and accept it or automatically remediate if not.

Terraform Cloud can also perform health checks for custom conditions and assertions with continuous validation. Users can monitor whether the functional validations defined in Terraform code continue to pass over time and receive an alert when an assertion fails. For example, you can monitor whether your website returns an expected status code, whether an API gateway certificate is valid, or whether the image artifact referenced from an HCP Packer channel is too old or has a scheduled revocation.

These two features provide users with flexible options to validate their infrastructure uptime, health, and security — all in one place without requiring additional tools.

»Step 4. Manage image lifecycles

If one of your golden images is outdated or possesses a vulnerability, you may need to revoke it to prevent infrastructure deployments from using it. HCP Packer and Terraform Cloud help provide a unified and simple revocation workflow across downstream builds and provisioning pipelines. When a golden image version is updated in an HCP Packer channel, any deployments using that image are simply re-run to pick up the new association. HCP Packer offers this simplified revocation workflow through the following:

1. Scheduled revocation: Plan revocation for a future end-of-life date or revoke the image version immediately
2. Inherited revocation: Builds on HCP Packer’s image ancestry tracking and established parent/child relationship to revoke just the base golden image or all associated downstream images
3. Channel rollback: Uses channel assignment history to provide quicker remediation of released artifacts by providing the option to roll back channels to their previously assigned iteration

»Business impact

Integrating HCP Packer’s image management capabilities into existing Terraform Cloud workflows brings a number of key benefits:

»Lower risk

Never deploy insecure images: Embed security and compliance requirements into all images across your cloud environments, set EOL dates, and automate revocation.

»Faster speed

Decrease time to deployment: Speed deployment by creating and reusing images from a single source configuration file, connecting to VCS, and collaborating across teams.

»Increased efficiency

Automate image management: Standardize image versions, change a golden image once, and automatically update across downstream builds.

»Better together

Using Terraform Cloud and HCP Packer together can help users achieve a standardized and efficient approach to their provisioning workflows. By following the steps outlined in this post, organizations can simplify their multi-cloud imaging processes leading to faster spin-up times, reduced human errors, and secure management of their entire infrastructure estate.

To learn more about building a golden image pipeline with Terraform Cloud and HCP Packer, check out the image management whitepaper, tutorial, and learn lab.

Get started with Terraform Cloud and​​ HCP Packer for free to begin provisioning and managing your infrastructure in any environment.