Learn how to build scalable, role-based SSH access with SSH certificates and HashiCorp Vault.
As enterprises accelerate their digital strategies, adoption of hybrid or multi-cloud architectures are becoming the new norm, requiring a fundamental shift in how infrastructure is provisioned and managed. When managing secure administration access to Unix-like servers, SSH is still the standard connectivity method. However, it does come with challenges and risks, especially in relation to key management.
The common reaction to the growing risks around key management is to simply seek out “SSH key management tools” in Google and review industry reports. But implementing products in this one category is just treating a symptom of a bigger challenge: secrets management. In this blog post, I will show you an SSH access architecture that will simplify SSH access using a scalable, secure, and consistent experience both on-premises and in public clouds. You can also view a demonstration of this architecture in this presentation: Leveraging Signed SSH for Remote Access with Vault.
Let’s begin by reviewing the limitations of SSH key-based authentication and what problems we are trying to solve:
I’ve already mentioned that organizations need a solution that addresses all of the challenges listed above — improving the SSH user experience and the security of SSH access to your hosts using a standard architecture that can be deployed in any environment. Many of the Global 2000 customers I’ve worked with are getting this result using a combination of HashiCorp Vault and SSH certificate authentication. I’m going to show you how this looks.
This architecture is designed to achieve the following outcomes:
SSH certificates work in a way similar to SSL certificates. SSH certs are simply a public key signed by a trusted entity called the certificate authority (CA). They introduce important features that will be used in this architecture:
There are many SSH host configurations. Some are not used commonly since they do not typically apply to SSH key authentication methods. There is a specific configuration that is used to complete this architecture, the sshd AuthorizedPrincipalsFile configuration.
HashiCorp Vault is a secrets management solution that programmatically brokers access to systems for both humans and machines. It can provide just-in-time secrets such as database credentials, PKI certificates, cloud IAM credentials, and many others.
In this use case, Vault will use its SSH Secrets Engine, allowing it to function as our SSH CA. It also provides granular access controls to SSH certificate parameters and signing, which is enforced by Vault policies.
My architecture to implement SSH certificate authentication with HashiCorp Vault looks like this:
The numbers in the diagram represent the following steps:
To expand on this, the Vault SSH Secret Engine can contain multiple Vault roles, where each role will contain the parameters that will be used during the SSH key signing. This allows different SSH certificates to be signed with different parameters and principals depending on the Vault role configurations. By using Vault policies, we achieve further control over who has access to these SSH CA roles.
The Vault SSH Secret Engine and roles diagram below illustrates an example:
Once a user successfully authenticates to Vault, a Vault token will be dynamically generated with an attached policy that dictates which services and secrets can be accessed by the user.
For the configurations below, you will need to have Vault running and unsealed. It is possible to test these configurations out locally by running Vault in dev mode. For Vault installation instructions, read the getting started with Vault guide.
The configurations will be used to set up the following user access requirements.
I will expand on these requirements in the subsequent sections.
The following steps will be used to configure Vault. For all the configurations I’ll use the Vault CLI.
For the sake of simplicity, we’ll use the UserPass authentication method. Vault will act as your identity broker, giving you the ability to leverage many other authentication methods that Vault supports such as LDAP or OIDC authentication. Here is an example of how to set up OIDC authentication with Azure AD.
Let’s set up three Vault accounts to represent the users that require SSH client access to hosts.
The Vault policies will be set up at a later stage.
The Vault SSH secret engine will need to be mounted and a signing key generated.
You should get the following output, showing the SSH CA public key, which will be used later on in the host configurations.
To sign the client keys, we will configure Vault roles to sign and issue SSH certificates with specific configurations based on users’ functional roles.
Three Vault SSH roles will be configured for signing SSH client keys, where each role will sign for a specific SSH principal.
Before we get to the code, A few important notes regarding Vault SSH role configurations:
allowed_users
: This is the list of allowed users this CA role will sign for. If the requester attempts to get a key signed by specifying a different user not in the allowed_users
list for that role, it will fail.
ttl
: This is where certificate expiry is set when signing an SSH key. In this example it is set for 30 minutes. Once the certificate expires, a user must authenticate to Vault and request another signed SSH certificate.
administrator-role
team-a-role
team-b-role
For each user created earlier, a corresponding policy will need to be configured. Here are the policy names:
These policies will restrict each user’s access to their authorized Vault SSH role for key signing. For example, Alice is an administrator and will require the SSH certificate signed with the administrator
principal. She will have permissions to get her SSH public key signed by the administrator-role. She will not be able to use any other Vault SSH role as per the policy. There is an implicit deny on everything else.
administrator-policy
team-a-policy
team-b-policy
A few steps are required to complete the SSH configurations on the host, however you can automate this setup by baking in these configurations into golden image servers or using configuration management tools. These steps are:
AuthorizedPrincipalsFile
and SSH principal names.sshd_config
and restart the SSH service.
Let's proceed with the configuration:These Linux accounts will be used for administrator and application user logins.
Navigate to the SSH directory and create a file that contains the SSH CA public key, this was previously configured in the Mount Vault SSH Certificate Secret Engine and Generate SSH CA Key Pair section.
The AuthorizedPrincipalsFile
configurations are important to further control which SSH principals are accepted for certificate authentication. For client authentication to be successful, the principal in the signed SSH certificate must appear in the AuthorizedPrincipalsFile
file. For now, let’s set up AuthorizedPrincipalsFile
for the administrator and team-a principals only. We will revisit team-b principal later on.
Update the SSH configuration file to reflect the required changes and enable SSH Certificate authentication. Add the following configurations to the sshd_config
file.
The AuthorizedPrincipalsFile
is the path containing the files listing the accepted principal names. The %u
signifies that the file name will be the username of the Linux user. So in our case, the admin
user will contain the administrator
principal, the appadmin
user will contain the team-a
principal.
Don’t forget to restart the sshd service.
The user will only need to create an SSH key pair. The user’s SSH public key will be signed by the Vault SSH CA and returned to the user. This signed SSH certificate will then be used to connect to the target host.
Let's go through what that may look like for Alice, who is a systems administrator.
Let’s login to Vault as Alice (administrator role), she should have permissions to access both servers. We are going to use the Vault CLI to authenticate with the UserPass authentication method. Notice the assigned policy:
Once authenticated, she should have permissions to use the Vault SSH role administrator-role
to sign her public SSH key.
The signed certificate that is returned can be output to alice-signed-key.pub
.
Take note of the valid_principals
requested: administrator
. If Alice attempts to request any other principal not in the allowed_users list
of the Vault SSH CA role, it will fail. This ensures that only authorized lists of SSH principals can be signed for, preventing a user from requesting other principals used by other teams.
In the next code snippet, note the contents of the certificate, specifically the Key ID
, ttl
, and the configured principal. It is possible to adjust these configurations in the Vault SSH role. For example, you could extend the TTL of the certificate to one hour or another length of time.
The client should be able to SSH into the host using the signed SSH certificate:
If Alice attempts to login with any other username, for example appadmin
user, it will fail:
If you recall, the AuthorizedPrincipalsFile
configuration for appadmin
only has team-a
as a listed principal. The administrator
principal is not listed in the AuthorizedPrincipalsFile
admin file.
Bob is in team-a
and requirements specify that Bob should also have SSH access to the server. Let’s test Bob’s login:
Generate an SSH key pair for Bob:
Login to Vault:
Let's get the public key signed by Vault:
Let's confirm the principal in the certificate team-a
:
Now Bob can use the signed certificate to sign in with the appadmin
user and team-a
principal:
However, if Bob attempts to use the admin user, it will fail since the team-a
principal is not allowed in the admin user on the host:
What if Bob attempts to get his SSH public key signed with the administrator
principal?
As for Tim, he will have no access at all to the server since the team-b
principal has not been added to any of the AuthorizedPrincipalsFile
configurations. Feel free to follow the same steps shown earlier for Bob to test access.
To allow Tim access to this specific server, the AuthorizedPrincipalsFile
configuration will need to include the team-b
principal under the appadmin
file:
There are many advantages to this architecture for SSH access. Here are a few:
AuthorizedPrincipalsFile
in conjunction with Vault SSH roles you can provide granular SSH access to selective hosts based on the user’s role, function, or team.Above is an architecture diagram of multiple Vault SSH CAs managing multiple SSH roles for different permissions.
We have covered a lot in this post, with detailed insights on how to effectively manage SSH access at scale using Vault and SSH certificates. Managing SSH access across multiple environments with SSH key authentication is challenging and many enterprises struggle as they scale while trying to manage SSH access to hundreds or thousands of hosts. For more information, here are a few resources:
HashiCorp Vault 1.15 contains a range of updates from UI updates and PKI enhancements to betas for Enterprise secrets sync, Enterprise seal high availability, and event monitoring.
Learn about the ACME protocol for PKI, the common problems it solves, and why it should be part of your certificate management roadmap.
New HashiCorp Vault ecosystem integrations extend security use cases for customers.