Learn how creating a golden image pipeline can help unify and streamline your imaging and provisioning workflows throughout your infrastructure estate.
In today’s multi-cloud world, images (such as AMIs for Amazon EC2, virtual machines, Docker containers, and more) lay the foundation for modern infrastructure, security, networking, and applications. Enterprises adopting multi-cloud typically start by using Terraform for centralized provisioning, but Terraform does not handle the details of image creation and management.
In many organizations, the workflows in place to create and manage images are siloed, time-consuming, and complex, leading to slow spin-up times and human errors that pose security risks. Organizations need standard processes to ensure all images throughout their infrastructure estate are secure, compliant, and easily accessible.
As organizations deploy fleets of images to support services across cloud and private environments, the complexity and scope of these services often involve multiple different teams. Without consistent, central processes and tooling in place, organizations can experience variability in their image workflows creating several challenges:
To combat these issues, organizations and their platform teams need to establish a central shared service for their image creation and management workflows.
Implementing a golden image pipeline with Terraform Cloud with HCP Packer unifies provisioning and image workflows to shift security left and automate image management across downstream builds and provisioning pipelines. HCP Packer helps platform teams establish a unified workflow management system across groups within an organization. This provides policy and governance, organization-wide visibility, ease of integration with peripheral technologies, and overall reliability at scale.
By integrating HCP Packer into their Terraform Cloud workflows, organizations can:
Golden image pipeline with Terraform Cloud and HCP Packer
The first step in creating a golden image pipeline is to create a set of golden images with HashiCorp Packer. A “golden image” is an approved image that acts as a template on top of which developers can build applications. These images contain the most up-to-date common system packages, logging and monitoring tools, security patches, and configuration hardening. Packer simplifies golden image creation by enabling organizations to leverage the HashiCorp Configuration Language (HCL). HCL simplifies the process of embedding all organizational requirements —such as security and operational details — into golden images. Codification also enables collaboration; changes can be reviewed by the appropriate stakeholders using standard version-control workflows before being implemented.
Packer configurations are defined using templates that enable users to leverage common configurations across multiple image builds. Templates consist of a series of declarations and commands for Packer to follow when generating a new image build. The template specifies what plugins (builders, data sources, provisioners, post-processors) to use, how to configure each of those plugins, and in what order to run them.
The "packer" block contains Packer settings, such as specifying a required Packer version, the "source" block configures a specific builder plugin, and the "build" block defines what Packer should do with the image after it launches
When a new golden image is created, this new version is automatically published to HCP Packer by including a simple hcp_packer_registry
block in the template. HCP Packer serves as a managed registry that stores image metadata, including when they were created, the associated cloud provider, and any custom labels specified in your image build. The HCP Packer artifact registry helps you track information about images, clearly designate which versions are approved for consumption, and query the right images to use in both Packer and Terraform configurations. Access to this centralized library helps align the workflows of image creation and deployment, allowing operations and development teams to work together to manage, track, and govern all artifacts across your infrastructure estate.
A core feature of HCP Packer that enables collaboration across teams is image channels. With channels, you can label image versions to describe the quality and stability of a build. By assigning human-readable names, downstream consumers can easily reference the images in Packer templates and Terraform configurations. For example, you can designate a specific channel for testing, allowing users to promote new versions and quickly spin up an instance to validate the image. Once the new version passes the required tests, it can be promoted to the stable channel, alerting downstream consumers that it is approved and ready for deployment. This workflow provides teams with vetted, ready-built artifacts that supply standard services in a plug-and-play fashion. Consumers can tailor versions of artifacts to streamline their efforts in the updating and release stages and ensure they are referencing the latest version without having to update their code directly. This image promotion workflow can also be automated in a declarative fashion using the HCP provider for Terraform.
Reference the HCP Packer registry directly from Terraform Cloud
With a golden image built, published, validated, and promoted to your organization's stable channel, Terraform runs referencing the updated version can now be queued automatically for any workspace using the channel. The image updates across downstream provisioning pipelines can take place autonomously with auto-apply settings or be gated by manual approval processes. The Terraform Cloud run task for HCP Packer helps prevent the deployment of non-approved images with:
With this automation, teams can integrate images easily onto a larger workflow framework to complement automated delivery pipelines.
Reference HCP Packer in your Terraform Cloud workflows.
With the new image version successfully approved and provisioned, the next step is to perform health assessments to make sure this infrastructure doesn't change over time. Even with a standardized initial provisioning process, settings on infrastructure can still be modified or circumvented, opening up your infrastructure to the possibility of configuration drift. Drift is the term for when the real-world state of your infrastructure differs from the state defined in your configuration. Drift occurs when a user modifies resources outside of the Terraform workflow.
Terraform Cloud’s drift detection allows users to actively monitor their infrastructure for these changes and receive alerts when they take place. From the health assessments dashboard they can quickly uncover the root cause for the change, determine if it is necessary, and accept it or automatically remediate if not.
View the last time drift was checked, the resources detected as being in a state of drift, and a visualization showing which attributes have changed
Terraform Cloud can also perform health checks for custom conditions and assertions with continuous validation. Users can monitor whether the functional validations defined in Terraform code continue to pass over time and receive an alert when an assertion fails. For example, you can monitor whether your website returns an expected status code, whether an API gateway certificate is valid, or whether the image artifact referenced from an HCP Packer channel is too old or has a scheduled revocation.
Validate that provisioned resources are still in a healthy state based on your custom conditions
These two features provide users with flexible options to validate their infrastructure uptime, health, and security — all in one place without requiring additional tools.
If one of your golden images is outdated or possesses a vulnerability, you may need to revoke it to prevent infrastructure deployments from using it. HCP Packer and Terraform Cloud help provide a unified and simple revocation workflow across downstream builds and provisioning pipelines. When a golden image version is updated in an HCP Packer channel, any deployments using that image are simply re-run to pick up the new association. HCP Packer offers this simplified revocation workflow through the following:
Simplify revocation processes across provisioning pipelines
Integrating HCP Packer’s image management capabilities into existing Terraform Cloud workflows brings a number of key benefits:
Never deploy insecure images: Embed security and compliance requirements into all images across your cloud environments, set EOL dates, and automate revocation.
Decrease time to deployment: Speed deployment by creating and reusing images from a single source configuration file, connecting to VCS, and collaborating across teams.
Automate image management: Standardize image versions, change a golden image once, and automatically update across downstream builds.
Using Terraform Cloud and HCP Packer together can help users achieve a standardized and efficient approach to their provisioning workflows. By following the steps outlined in this post, organizations can simplify their multi-cloud imaging processes leading to faster spin-up times, reduced human errors, and secure management of their entire infrastructure estate.
To learn more about building a golden image pipeline with Terraform Cloud and HCP Packer, check out the image management whitepaper, tutorial, and learn lab.
Get started with Terraform Cloud and HCP Packer for free to begin provisioning and managing your infrastructure in any environment.
Terraform Enterprise now supports more flexible deployment options for self-hosted environments, including cloud-managed Kubernetes services.
As the final step in moving towards external plugins, HashiCorp Packer will stop bundling official plugins within its binary releases. Here’s what that means.
Assigning agents at the organization level provides a faster, more consistent, and scalable approach to agent pool configuration.