Learn how to build scalable, role-based SSH access with SSH certificates and HashiCorp Vault.
As enterprises accelerate their digital strategies, adoption of hybrid or multi-cloud architectures are becoming the new norm, requiring a fundamental shift in how infrastructure is provisioned and managed. When managing secure administration access to Unix-like servers, SSH is still the standard connectivity method. However, it does come with challenges and risks, especially in relation to key management.
The common reaction to the growing risks around key management is to simply seek out “SSH key management tools” in Google and review industry reports. But implementing products in this one category is just treating a symptom of a bigger challenge: secrets management. In this blog post, I will show you an SSH access architecture that will simplify SSH access using a scalable, secure, and consistent experience both on-premises and in public clouds. You can also view a demonstration of this architecture in this presentation: Leveraging Signed SSH for Remote Access with Vault.
Let’s begin by reviewing the limitations of SSH key-based authentication and what problems we are trying to solve:
I’ve already mentioned that organizations need a solution that addresses all of the challenges listed above — improving the SSH user experience and the security of SSH access to your hosts using a standard architecture that can be deployed in any environment. Many of the Global 2000 customers I’ve worked with are getting this result using a combination of HashiCorp Vault and SSH certificate authentication. I’m going to show you how this looks.
This architecture is designed to achieve the following outcomes:
SSH certificates work in a way similar to SSL certificates. SSH certs are simply a public key signed by a trusted entity called the certificate authority (CA). They introduce important features that will be used in this architecture:
There are many SSH host configurations. Some are not used commonly since they do not typically apply to SSH key authentication methods. There is a specific configuration that is used to complete this architecture, the sshd AuthorizedPrincipalsFile configuration.
HashiCorp Vault is a secrets management solution that programmatically brokers access to systems for both humans and machines. It can provide just-in-time secrets such as database credentials, PKI certificates, cloud IAM credentials, and many others.
In this use case, Vault will use its SSH Secrets Engine, allowing it to function as our SSH CA. It also provides granular access controls to SSH certificate parameters and signing, which is enforced by Vault policies.
My architecture to implement SSH certificate authentication with HashiCorp Vault looks like this:
The numbers in the diagram represent the following steps:
To expand on this, the Vault SSH Secret Engine can contain multiple Vault roles, where each role will contain the parameters that will be used during the SSH key signing. This allows different SSH certificates to be signed with different parameters and principals depending on the Vault role configurations. By using Vault policies, we achieve further control over who has access to these SSH CA roles.
The Vault SSH Secret Engine and roles diagram below illustrates an example:
Once a user successfully authenticates to Vault, a Vault token will be dynamically generated with an attached policy that dictates which services and secrets can be accessed by the user.
For the configurations below, you will need to have Vault running and unsealed. It is possible to test these configurations out locally by running Vault in dev mode. For Vault installation instructions, read the getting started with Vault guide.
The configurations will be used to set up the following user access requirements.
I will expand on these requirements in the subsequent sections.
The following steps will be used to configure Vault. For all the configurations I’ll use the Vault CLI.
For the sake of simplicity, we’ll use the UserPass authentication method. Vault will act as your identity broker, giving you the ability to leverage many other authentication methods that Vault supports such as LDAP or OIDC authentication. Here is an example of how to set up OIDC authentication with Azure AD.
Let’s set up three Vault accounts to represent the users that require SSH client access to hosts.
vault auth enable userpass vault write auth/userpass/users/alice password="passw0rd" policies="administrator-policy" vault write auth/userpass/users/bob password="passw0rd" policies="team-a-policy" vault write auth/userpass/users/tim password="passw0rd" policies="team-b-policy"
The Vault policies will be set up at a later stage.
The Vault SSH secret engine will need to be mounted and a signing key generated.
vault secrets enable -path=ssh-client-signer ssh vault write ssh-client-signer/config/ca generate_signing_key=true
You should get the following output, showing the SSH CA public key, which will be used later on in the host configurations.
Success! Enabled the ssh secrets engine at: ssh-client-signer/ Key Value— — — — -public_key ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDthG9+wjEvCgqBVlBpifXK2PhjXzmjt2+3kN6J6CGsZpeevdlMtxutaAmALfREGYKGol4pqwzJrT5iGp3LAenuiC97x1skItXWrM+BYwyiaU0f8XjBIOPrviBN0v+H6XgoFlujxvdKgJ4+diORkaqtW4gz3Fhe/Gq/3s5WZ5MnB+UZsvYPhd0zuQorEoaYAqarwxq30XPSYEFaH6zqefddJaejMd8PiKGuFdhLHjnZ5jU0r+vpZfrcuQ+81RhVxD4wtiKmouO3zW8bBVrYoURKLap5UlPu8RzPwbJ3PtqtNDkV7SxeNvbXP9mmfkwyBqsU4zaenDRlFqc8UG0SyU6JMM4nL6pLpicC8EC/tHb434U/WwmuHrYhKmpXt25G6cCAHyWxu/9WXqrj4C4Cn2w495WgxGs3EqM+DotSRPw7GkfyCujnpCddcjPc1L5RdHk/tGI26UrnxMHOEBa+zXVmTlEeRyqg3HCSwBejAk3AT1lk1I+D89ANWDgipnNY7UZG+lSsRZHyiKKbfyhaoUFo8JW0KR6vujqflPrzmOHHHyW6zYmBKO+EYTlaiwWAV8nUMl5XfnARmq4Lv6WL3dbdhur6LWY1WiDNviByntYaEDViRXl6lyqrYhrXONJbi0PLB9HuKL7z4m3K9OYgYS5ofFjoCVb9HGytGgEko1wQ==
To sign the client keys, we will configure Vault roles to sign and issue SSH certificates with specific configurations based on users’ functional roles.
Three Vault SSH roles will be configured for signing SSH client keys, where each role will sign for a specific SSH principal.
Before we get to the code, A few important notes regarding Vault SSH role configurations:
allowed_users
: This is the list of allowed users this CA role will sign for. If the requester attempts to get a key signed by specifying a different user not in the allowed_users
list for that role, it will fail.
ttl
: This is where certificate expiry is set when signing an SSH key. In this example it is set for 30 minutes. Once the certificate expires, a user must authenticate to Vault and request another signed SSH certificate.
administrator-role
vault write ssh-client-signer/roles/administrator-role -<<EOH{ “allow_user_certificates”: true, “allowed_users”: “administrator”, “allowed_extensions”: “”, “default_extensions”: [ { “permit-pty”: “” } ], “key_type”: “ca”, “default_user”: “administrator”, “ttl”: “30m0s”}EOH
team-a-role
vault write ssh-client-signer/roles/team-a-role -<<EOH{ “allow_user_certificates”: true, “allowed_users”: “team-a”, “allowed_extensions”: “”, “default_extensions”: [ { “permit-pty”: “” } ], “key_type”: “ca”, “default_user”: “team-a”, “ttl”: “30m0s”}EOH
team-b-role
vault write ssh-client-signer/roles/team-b-role -<<EOH{ “allow_user_certificates”: true, “allowed_users”: “team-b”, “allowed_extensions”: “”, “default_extensions”: [ { “permit-pty”: “” } ], “key_type”: “ca”, “default_user”: “team-b”, “ttl”: “30m0s”}EOH
For each user created earlier, a corresponding policy will need to be configured. Here are the policy names:
These policies will restrict each user’s access to their authorized Vault SSH role for key signing. For example, Alice is an administrator and will require the SSH certificate signed with the administrator
principal. She will have permissions to get her SSH public key signed by the administrator-role. She will not be able to use any other Vault SSH role as per the policy. There is an implicit deny on everything else.
administrator-policy
vault policy write administrator-policy — << EOF# List available SSH rolespath “ssh-client-signer/roles/*” { capabilities = [“list”]}# Allow access to SSH rolepath “ssh-client-signer/sign/administrator-role” { capabilities = [“create”,”update”]}EOF
team-a-policy
vault policy write team-a-policy — << EOF# List available SSH rolespath “ssh-client-signer/roles/*” { capabilities = [“list”]}# Allow access to SSH rolepath “ssh-client-signer/sign/team-a-role” { capabilities = [“create”,”update”]}EOF
team-b-policy
vault policy write team-b-policy — << EOF# List available SSH rolespath “ssh-client-signer/roles/*” { capabilities = [“list”]}# Allow access to SSH rolepath “ssh-client-signer/sign/team-b-role” { capabilities = [“create”,”update”]}EOF
A few steps are required to complete the SSH configurations on the host, however you can automate this setup by baking in these configurations into golden image servers or using configuration management tools. These steps are:
AuthorizedPrincipalsFile
and SSH principal names.sshd_config
and restart the SSH service.
Let's proceed with the configuration:These Linux accounts will be used for administrator and application user logins.
sudo useradd -m adminsudo useradd -m appadmin
Navigate to the SSH directory and create a file that contains the SSH CA public key, this was previously configured in the Mount Vault SSH Certificate Secret Engine and Generate SSH CA Key Pair section.
cd /etc/ssh sudo echo ‘ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDthG9+wjEvCgqBVlBpifXK2PhjXzmjt2+3kN6J6CGsZpeevdlMtxutaAmALfREGYKGol4pqwzJrT5iGp3LAenuiC97x1skItXWrM+BYwyiaU0f8XjBIOPrviBN0v+H6XgoFlujxvdKgJ4+diORkaqtW4gz3Fhe/Gq/3s5WZ5MnB+UZsvYPhd0zuQorEoaYAqarwxq30XPSYEFaH6zqefddJaejMd8PiKGuFdhLHjnZ5jU0r+vpZfrcuQ+81RhVxD4wtiKmouO3zW8bBVrYoURKLap5UlPu8RzPwbJ3PtqtNDkV7SxeNvbXP9mmfkwyBqsU4zaenDRlFqc8UG0SyU6JMM4nL6pLpicC8EC/tHb434U/WwmuHrYhKmpXt25G6cCAHyWxu/9WXqrj4C4Cn2w495WgxGs3EqM+DotSRPw7GkfyCujnpCddcjPc1L5RdHk/tGI26UrnxMHOEBa+zXVmTlEeRyqg3HCSwBejAk3AT1lk1I+D89ANWDgipnNY7UZG+lSsRZHyiKKbfyhaoUFo8JW0KR6vujqflPrzmOHHHyW6zYmBKO+EYTlaiwWAV8nUMl5XfnARmq4Lv6WL3dbdhur6LWY1WiDNviByntYaEDViRXl6lyqrYhrXONJbi0PLB9HuKL7z4m3K9OYgYS5ofFjoCVb9HGytGgEko1wQ==’ > trusted-CA.pem
The AuthorizedPrincipalsFile
configurations are important to further control which SSH principals are accepted for certificate authentication. For client authentication to be successful, the principal in the signed SSH certificate must appear in the AuthorizedPrincipalsFile
file. For now, let’s set up AuthorizedPrincipalsFile
for the administrator and team-a principals only. We will revisit team-b principal later on.
cd /etc/ssh mkdir auth_principals/ sudo echo ‘administrator’ > adminsudo echo ‘team-a’ > appadmin
Update the SSH configuration file to reflect the required changes and enable SSH Certificate authentication. Add the following configurations to the sshd_config
file.
AuthorizedPrincipalsFile /etc/ssh/auth_principals/%uChallengeResponseAuthentication noPasswordAuthentication noTrustedUserCAKeys /etc/ssh/trusted-CA.pem
The AuthorizedPrincipalsFile
is the path containing the files listing the accepted principal names. The %u
signifies that the file name will be the username of the Linux user. So in our case, the admin
user will contain the administrator
principal, the appadmin
user will contain the team-a
principal.
Don’t forget to restart the sshd service.
sudo service sshd restart
The user will only need to create an SSH key pair. The user’s SSH public key will be signed by the Vault SSH CA and returned to the user. This signed SSH certificate will then be used to connect to the target host.
Let's go through what that may look like for Alice, who is a systems administrator.
ssh-keygen -b 2048 -t rsa -f ~/.ssh/alice-keyssh-add ~/.ssh/alice-key
Let’s login to Vault as Alice (administrator role), she should have permissions to access both servers. We are going to use the Vault CLI to authenticate with the UserPass authentication method. Notice the assigned policy:
vault login -method=userpass username=alice password=passw0rd Success! You are now authenticated. The token information displayed below is already stored in the token helper. You do NOT need to run “vault login” again. Future Vault requests will automatically use this token.Key Value — — — — -token s.QuJwJTa14g9EvPUPsVx0Lta2token_accessor wzxbGGElsJsyLibwVHOyBLjMtoken_duration 768htoken_renewable truetoken_policies [“administrator-policy” “default”]identity_policies []policies [“administrator-policy” “default”]token_meta_username alice
Once authenticated, she should have permissions to use the Vault SSH role administrator-role
to sign her public SSH key.
The signed certificate that is returned can be output to alice-signed-key.pub
.
vault write -field=signed_key ssh-client-signer/sign/administrator-role \ public_key=@$HOME/.ssh/alice-key.pub valid_principals=administrator > ~/.ssh/alice-signed-key.pub
Take note of the valid_principals
requested: administrator
. If Alice attempts to request any other principal not in the allowed_users list
of the Vault SSH CA role, it will fail. This ensures that only authorized lists of SSH principals can be signed for, preventing a user from requesting other principals used by other teams.
In the next code snippet, note the contents of the certificate, specifically the Key ID
, ttl
, and the configured principal. It is possible to adjust these configurations in the Vault SSH role. For example, you could extend the TTL of the certificate to one hour or another length of time.
ssh-keygen -Lf ~/.ssh/alice-signed-key.pubalice-signed-key.pub: Type: ssh-rsa-cert-v01@openssh.com user certificate Public key: RSA-CERT SHA256:xSGrnRx5QLitljNNWonCJtAzNhGqqVkt06hvlHSCy0w Signing CA: RSA SHA256:ZMDd6dr1awUMgkrYEwx6KO76BlIjTkvBTbxoHXryMHc (using ssh-rsa) Key ID: “vault-userpass-alice-c521ab9d1c7940b8ad96334d5a89c226d0333611aaa9592dd3a86f947482cb4c” Serial: 1539122095861524177 Valid: from 2021–06–10T14:52:30 to 2021–06–10T15:23:00 Principals: administrator Critical Options: (none) Extensions: permit-pty
The client should be able to SSH into the host using the signed SSH certificate:
ssh -i ~/.ssh/alice-signed-key.pub admin@server "whoami"admin
If Alice attempts to login with any other username, for example appadmin
user, it will fail:
ssh -i ~/.ssh/alice-signed-key.pub appadmin@server “whoami”appadmin@server: Permission denied (publickey).
If you recall, the AuthorizedPrincipalsFile
configuration for appadmin
only has team-a
as a listed principal. The administrator
principal is not listed in the AuthorizedPrincipalsFile
admin file.
Bob is in team-a
and requirements specify that Bob should also have SSH access to the server. Let’s test Bob’s login:
Generate an SSH key pair for Bob:
ssh-keygen -b 2048 -t rsa -f ~/.ssh/bob-key -q -N “” 0>&-ssh-add ~/.ssh/bob-key
Login to Vault:
vault login -method=userpass username=bob password=passw0rd Success! You are now authenticated. The token information displayed below is already stored in the token helper. You do NOT need to run “vault login” again. Future Vault requests will automatically use this token.Key Value — — — — -token s.9INPgEYJQqRKeZ0FxRekQpHDtoken_accessor dUYyvvFsCbGTRKQNI4O8K9WOtoken_duration 768htoken_renewable truetoken_policies [“default” “team-a-policy”]identity_policies []policies [“default” “team-a-policy”]token_meta_username bob
Let's get the public key signed by Vault:
vault write -field=signed_key ssh-client-signer/sign/team-a-role \ public_key=@$HOME/.ssh/bob-key.pub valid_principals=team-a > ~/.ssh/bob-signed-key.pub
Let's confirm the principal in the certificate team-a
:
ssh-keygen -Lf ~/.ssh/bob-signed-key.pub bob-signed-key.pub: Type: ssh-rsa-cert-v01@openssh.com user certificate Public key: RSA-CERT SHA256:moR3M+yGM2sQvRSmgobZx5OH/14rpGYIbmo9dw+VePg Signing CA: RSA SHA256:APvCMzgQirBY6PX8ZSaXgXVO6Bpops17pjHVo1RhUHo (using ssh-rsa) Key ID: “vault-userpass-bob-9a847733ec86336b10bd14a68286d9c79387ff5e2ba466086e6a3d770f9578f8” Serial: 3154484434744577453 Valid: from 2021–06–11T17:26:49 to 2021–06–11T17:57:19 Principals: team-a Critical Options: (none) Extensions: permit-pty
Now Bob can use the signed certificate to sign in with the appadmin
user and team-a
principal:
ssh -i ~/.ssh/bob-signed-key.pub -i ~/.ssh/bob-key appadmin@server “whoami”appadmin
However, if Bob attempts to use the admin user, it will fail since the team-a
principal is not allowed in the admin user on the host:
ssh -i ~/.ssh/bob-signed-key.pub -i ~/.ssh/bob-key admin@server “whoami”admin@server: Permission denied (publickey).
What if Bob attempts to get his SSH public key signed with the administrator
principal?
vault write -field=signed_key ssh-client-signer/sign/team-a-role \ public_key=@$HOME/.ssh/bob-key.pub valid_principals=administrator > ~/.ssh/bob-signed-key.pubError writing data to ssh-client-signer/sign/administrator-role: Error making API request.URL: PUT http://localhost:8200/v1/ssh-client-signer/sign/team-a-roleCode: 400. Errors:* administrator is not a valid value for valid_principals
As for Tim, he will have no access at all to the server since the team-b
principal has not been added to any of the AuthorizedPrincipalsFile
configurations. Feel free to follow the same steps shown earlier for Bob to test access.
To allow Tim access to this specific server, the AuthorizedPrincipalsFile
configuration will need to include the team-b
principal under the appadmin
file:
cd /etc/ssh/auth_principals sudo echo ‘team-b’ >> appadmin
There are many advantages to this architecture for SSH access. Here are a few:
AuthorizedPrincipalsFile
in conjunction with Vault SSH roles you can provide granular SSH access to selective hosts based on the user’s role, function, or team.Above is an architecture diagram of multiple Vault SSH CAs managing multiple SSH roles for different permissions.
We have covered a lot in this post, with detailed insights on how to effectively manage SSH access at scale using Vault and SSH certificates. Managing SSH access across multiple environments with SSH key authentication is challenging and many enterprises struggle as they scale while trying to manage SSH access to hundreds or thousands of hosts. For more information, here are a few resources:
Learn how to automate the unsealing and snapshotting of HashiCorp Vault using HashiCorp Nomad and Vault Unsealer.
Leverage HashiCorp Vault as a trusted certificate authority (CA) to issue short-lived code signing certificates to a GitHub Actions workflow.
Golden patterns for infrastructure and security automation workflows lie at the core of The Infrastructure Cloud. Here’s how to implement them using HashiCorp Cloud Platform services.