In this document, we describe the application delivery lifecycle and the HashiCorp approach to modern application delivery.
At HashiCorp we are focused on providing the tools that allow organizations to adopt the cloud and automate their infrastructure, with the end goal of accelerating application delivery. This means we have a wide span on concerns, including provisioning, securing, running, and connecting applications. Our approach applies a set of infrastructure management principles to the application delivery lifecycle delivered through a suite of products.
Software is never finished. Instead, it constantly evolves with the addition of new features, bug fixes, and new architectures. This means software delivery is a continuous lifecycle and not a one time event. We believe this lifecycle has seven steps which are necessary and sufficient for most organizations:
Together these steps are both necessary and sufficient for delivering most applications. Depending on the organization, additional steps may be required between these anchor points, but often those additions can be integrated into these steps above. These steps are technology agnostic and reflect the workflow challenges in delivering an application, regardless of the underlying technology choices.
The continuous evolution of software means application delivery is naturally modeled as a continuous process as well; the infrastructure that supports an application must continuously evolve to meet the changing requirements of the application. For teams building, managing, or supporting applications and infrastructure, we are tasked with two critical responsibilities:
While managing complexity and risk are the guiding principles of infrastructure management, there are a number of techniques we can apply to achieve them:
The application delivery lifecycle and principles of infrastructure management are fundamental and apply to any technology. However, there are several properties of the cloud that affect the tools and techniques we use:
There is a deep amount of domain knowledge for each step in the application delivery process, making it impractical for an individual to be an expert in every area. Instead, there is a natural specialization of knowledge and a process which allows all the domain experts to collaborate. Typically the following groups are involved:
Each of these roles have a different set of skills and expertise, and there are many ways in which these teams can work together. The Waterfall methodology is the most common, while the DevOps approach is being broadly adopted.
In many traditional software organizations, Waterfall is the dominant model used to deliver applications. This approach prioritizes managing risk and sequentially flowing work between various groups. This tends to be very slow and reflects the challenges of delivering desktop applications that could not be easily updated.
On the other hand, organizations adopting a DevOps approach create "APIs" between teams so that the details of each role can be encapsulated, and each group can work independently. This prioritizes agility, allowing tasks to be done in parallel. It results in a lower cost for updating software, especially online applications, which don't need stringent controls over risk because the platform allows instant fixes and updates.
If we integrate all the people and application delivery steps into a pipeline, we get the following:
From this view, we can see that each role has a few concerns that overlap with each other at a few touch points. These touch points provide the coordination that allows each group to work in parallel. Specifically, we decompose this into four layers: provision, secure, run, and connect. Developers and operators are decoupled by using Nomad to run applications. This splits the provisioning responsibilities and reduces the coordination that is required. Vault is used to decouple security teams from developers and operators, allowing them to secure the infrastructure by brokering access to credentials and secrets programmatically. Security teams still control access and policies, and with Vault they can rotate secrets automatically without tight coordination. Lastly, Consul allows all the applications to connect together at runtime without any tight coupling. It also provides operations and security with fine-grained, centralized network segmentation controls.
In the pipeline view there are a number of HashiCorp products, each of them serving a specific function:
Vagrant provides a development environment that closely mimics production. This dev/prod parity avoids the "it worked on my machine" class of bugs and allows developers to have a fast feedback loop between development and testing. By codifying the setup of a Vagrant environment, new developers can quickly and reliably be on-boarded.
Packer is used to build artifacts, ranging from Docker containers, to AWS AMI's, to VMDK's. It is used to take source code, configuration management, and other provisioning information to build artifacts. These artifacts are usually immutable and versioned.
Terraform provides a consistent way to provision and manage resources across hundreds of providers and thousands of resource types. This includes low-level storage, compute, and network from cloud service providers, and higher level services like DNS, SMS, and CDNs. It uses an infrastructure as code approach, enabling operators to manage complex fleets in a modular and composable way. Terraform Enterprise provides centralized collaboration, coordination, and governance, similar to GitHub for developers.
Nomad is a cluster manager and application scheduler. It pools together the resources of many machines and dynamically schedules applications based on declarative job files. Job files give developers an infrastructure as code way to deploy applications abstracted from hardware, while decoupling operators who are managing the underlying fleet.
Consul provides a toolkit to support service-oriented architectures. Applications broadcast their availability and register health checks for monitoring. Applications can discover their upstream services via DNS or by querying Consul with a RESTful HTTP API. Consul provides load balancing and uses health checks to route around failures. It also provides a Key/Value store which can be used for application configuration and high availability via leader election.
Vault provides a centralized service for brokering access to credentials and secrets. Security teams can manage policies, delegate access, publish secrets, and audit access. Developers, operators, and applications can access the secret material they need in a secure fashion. Vault also provides key management and cryptographic offload to encrypt PII or other sensitive data. Higher level features include brokering SSH access, dynamic credential generation, and PKI.
Aside from the HashiCorp tools, there are other pieces in the pipeline view where existing tools and technologies are leveraged:
While HashiCorp has several products, getting started doesn't have to be complicated. Our modular approach to tooling allows for incremental adoption of one or more tools. We recommend starting with a well-scoped project, both to minimize risks and define a clear success criteria. Below are some recommended projects:
Manage Secrets with Vault
Many organizations suffer from secret sprawl, where privileged material like credentials, API tokens, and TLS certificates are stored in many different systems in plaintext including source code, shell scripts, or configuration management. This is a recipe for disastrous data breaches. Instead, secrets should be managed in Vault, where they are encrypted in transit and at rest, with central authentication, authorization, and auditing. Organizations can start by standing up a Vault cluster, moving existing secrets, and integrating applications.
Enable Service Discovery with Consul
As microservices or SOA are adopted, services need the ability to discover and route to their upstreams. For example, web servers needs to communicate with backend API servers. Consul provides a toolkit of features to enable SOA. Organizations can start by standing up a Consul cluster, registering a few services, and using DNS or
consul-template to begin integration with downstream services.
Provision Cloud Infrastructure with Terraform
Cloud adoption provides an opportunity to experiment with new tools without changing existing processes. Terraform supports all the major cloud service providers, and there is a repository of rich documentation and examples on using it to provision cloud resources with each of those providers. Organizations can start by provisioning cloud resources for greenfield projects with Terraform. As more comfort is gained, existing applications can be brought under management and the scope of Terraform.
Build Images with Packer
HashiCorp tools generally push for an immutable model of management, although it's not a prerequisite. Packer allows for machine images to be easily created across dozens of targets including container and cloud VM images. Building immutable images with Packer is a good starting point to leverage existing configuration management and provisioning tools while adopting a more immutable approach, reducing operational complexity and risk of provisioning-time failures.
Run a Container with Nomad
Containers are a convenient way to package applications regardless of language or framework. They're useful as a standard unit to ship around teams in your organization. Nomad provides a simple way to schedule containers, whether it's one container or one million. Organizations can start playing with container schedulers by setting up a Nomad cluster and running Docker containers. This can be extended using Consul for service discovery and load balancing, and Vault for managing and distributing secrets inside the containers.
We have discussed the various challenges of application delivery, along with the principles HashiCorp applies in solving them, but often the value in our approach is implicit. There are several tangible and intangible values, including:
Cloud Adoption HashiCorp provides a product suite that enables organizations to adopt a single cloud or multiple clouds, both public and private. The workflow-centric view allows the platform specific differences to be accommodated without many cloud-specific workflows.
Applying an infrastructure as code approach across our product suite enables automation of the entire application delivery process. This increases the agility of all teams involved, reduces human errors, and improves security.
Empowering Developer, Operator, and Security Teams
Delineating the application delivery challenge into the sub-problems and using tools instead of tickets to coordinate between teams provides individuals more autonomy via self-service for developers and a decoupling of concerns for operators and security teams.
A workflow-centric approach to application delivery allows heterogeneous technologies to be used easily. This simplifies hybrid cloud adoption but also enables easier experimentation and adoption of new tools and services without changing workflows.
Integrating security into each step of the application delivery process and placing zero trust in the network is required for the security challenges of today. HashiCorp Vault provides a security foundation and integrates with other HashiCorp and third-party products to provide a holistic security solution.
Reduced Complexity and Risk
From previous sections you've seen examples of how HashiCorp's approaches to IT challenges reduce complexity and risk. The simple management of security, system resilience, and infrastructure construction help engineers maintain distributed systems, keeping them clean and agile.
Adopting a policy as code approach can remove compliance as a bottleneck in the application delivery process. Instead of filing tickets and manually enforcing business policies, compliance checks can be codified and automatically enforced. This provides a "sandbox" that developers and operators can work in without needing to wait for a slow approval process.
Software and infrastructure best practices will continue evolving to solve new challenges. HashiCorp addresses these challenges with a principled but pragmatic approach. For organizations adopting cloud, HashiCorp provides the products necessary to make the transition from traditional data centers to dynamic infrastructure services without having to rewrite your applications. The HashiCorp stack makes this transition safe and incremental, allowing your organization to gradually adopt more beneficial DevOps practices, with the eventual goal of empowering developers, operators, and security teams via automation.
Our open source tools are used by millions of users in every geography and industry sector. Our enterprise products are focused on solving the organizational challenges of the Global 10K, including collaboration, governance, and compliance. Our customers include many of the Fortune 500 such as Capital One, Salesforce, Verizon, and SAP.
Publishing Packer Plugins to the Masses
On-demand access to earnings via self-serve infrastructure at Earnin
HashiCorp Deep Dive Demos from Ignite and KubeCon Europe
How Remote Work is Driving the Need for Multi-Cloud DevSecOps: How to Build a Pipeline