From key sprawl to scalable control: Rethinking SSH access

Static SSH keys, as part of the SSH protocol, have been around for twenty years and cemented themselves within organizations worldwide as the de facto method for securely connecting to resources. But what works for a handful of servers quickly becomes a security and operational nightmare at scale. Add in contractors and third-party vendors, and the situation gets even more risky.

Static keys live for a long time, get shared across teams, and leave no audit trail — making it nearly impossible to know who accessed what, when, or why. The good news? By transitioning from static keys to SSH certificates and using modern secrets management and secure remote access technologies, organizations can transform SSH access from an unmanageable liability into a scalable, secure, and auditable process that actually works for modern, dynamic environments.

»Static SSH keys: Security and scalability challenges

Public/private SSH key pairs are typically generated with the public key residing on the SSH target resource and the private key in the control of the human end-user — this is where key handling and distribution get bogged down in manual workflows or suffer from human error. From personal experience, these keys are often:

• Static and rarely rotated
• Shared between multiple individuals/teams within an organization
• A major cost in operational time when manually placing all the keys on all devices within the estate
• Used in risky contractor and third-party workflows

Perhaps this “method” would have some merit if your IT estate was small enough for one or two people to manage their own respective, individual server, and that was never ever expected to grow. But realistically, this situation doesn’t exist for enterprises or scaling companies, so when teams grow, when resources scale, when access requirements expand, you will start to see this unmanageable sprawl within your environment.

Aside from the clear security concerns, there are other concerning challenges that come with using static keys, including auditability and traceability.

How would you know who logged onto a device and what commands were issued during the lifetime of that session? Were any files removed or copied to a different location? These are all unknown if you are sharing the private key among multiple users, which makes it impossible to trace and manage.

Some teams try to solve this by creating a different set of keys for different groups, but this just increases the management overhead, which makes it even more unlikely that rotation best practices are followed.

We know there are organizations right now that are giving these SSH keys to third parties and contractors without a strict rotation policy because we’ve read about the breaches in the news. The story typically goes like this:

• External parties are given access to sensitive and mission-critical services through SSH keys.
• With those keys, a user can operate practically incognito and without a trace.
• With a poor rotation strategy in place, those keys end up allowing access long after the third party’s contract ends.
• There are no limits on the third party sharing the keys with multiple people, and eventually a bad actor ends up finding them and using them to execute a long-term breach.

How do we rethink SSH access and key rotation to make sure this story doesn’t keep happening?

»Evolving to SSH certificates

SSH certificates are the natural evolution of the standard public/private key SSH access. Instead of using keys that are not scalable, have no automatic revocation, and no expiration, signed SSH certificates help to alleviate some of these issues and concerns.

»How they work

A public and private key are generated, but a certificate authority (CA) is now introduced. The private key remains private, but the public key is signed by the CA’s private key, which in turn creates a signed SSH certificate. When the CA signs the SSH certificate, criteria such as the validity of the certificate, or who the certificate belongs to, can be set.

»Why SSH certificates?

With SSH certificates, you no longer have to be concerned with managing a bunch of long-lived and static SSH keys to provide secure access to resources. Organizations can move to a centralized method of distributing certificates for SSH access, which can adequately scale as organizations grow.

Here’s an example workflow:

1. Users generate an SSH key pair (if it’s not already created) and send the public key to the CA admin. There will likely be some metadata included specifying the validity period and/or usernames allowed to access the SSH target.
2. The CA administrator signs the public key to generate the certificate and returns it to the requester.
3. From here, the user can then SSH to the target machine using their private key and signed certificate. In order for the target machine to trust the CA that signed the certificate, the target’s SSH configuration file includes the CA’s public key.

This provides a more secure method and workflow in comparison to static keys, as previously discussed.

»Mechanics of SSH certificates

SSH certificates do not introduce new encryption algorithms  —  they wrap existing public keys in a signed certificate. So the type of key used in the certificate is based on the underlying public key algorithm.

»Encryption algorithms

Two of the most common algorithms used are RSA (Rivest Shamir Adleman) and ED25519 (Edwards-curve Digital Signature Algorithm). A key difference between these two encryption methods is that ED25519 uses a smaller key size (public key being only 68 characters long) compared with the RSA size of 1024 to 4096 bits, yet ED25519 is comparable to an RSA key size of 3000 bits. To add some more context, the default recommended size for RSA keys to be “secure” is at least 2048 bits, so ED25519 can be considered more secure in this example. Additionally, from a performance perspective, ED25519 has the advantage that it is faster than RSA in relation to key generation and validation of signatures, which has a positive impact on the holistic workflow and user experience.

»Limiting access

As mentioned previously, the SSH target that a user is making a connection to needs to have a trusted CA configured to sign the SSH certificates. This configuration change needs to be saved within the sshd.service or sshd_config file.

As part of the configuration, it is necessary to add the usernames that will be allowed access to the SSH target(s) in question. This can be achieved using principals, which are effectively the Linux usernames that accompany the SSH certificate when a user logs onto an SSH host.

Using Amazon Linux as an example, the default username is ec2-user. However, it may be prudent to ensure that you don’t widely distribute the privileges associated with what is effectively a root user.

In the example above, this command signs an SSH public key using a CA private key, creating a signed SSH certificate (id_ed25519.pub), which can then be used to authenticate on the SSH target. This command passes in specific flags, which are explained below:

• -s/path/to/ca_key — This command signs the public key using the CA private key found at /path/to/ca_key. This key is trusted by the SSH target.
• -I example-session — This is a human-readable name for the certificate, akin to a label.
• -n ec2-user,readonly,admin — These are the principals, i.e. the list of allowed usernames this certificate can be used as. If the command ssh readonly@host_address was entered, and the certificate lists readonly as a principal, it will be accepted.
• id_ed25519.pub — This is the public key being signed. The result will be id_ed25519-cert.pub, the SSH certificate.

Although this method is a marked improvement in comparison to the limitations previously discussed with static and long-lived SSH keys, there is the additional management and configuration of the CAs to contend with and the need for a suitable workflow to allow users to request the SSH certificates. Organizations need to ensure they have the resources and skills to manage and maintain this.

»Evolving further, with HashiCorp an IBM company

HashiCorp provides solutions that address organizational challenges around secrets management by centralizing and automatically rotating long-lived, static credentials. This is the primary use case for HashiCorp Vault. To secure user access across dynamic environments using identity-driven controls, HashiCorp also offers Boundary. The integration between Vault and Boundary further simplifies and automates SSH certificates at scale for large enterprises.

With Vault, you can use the SSH secrets engine, which provides secure authentication and authorization for access to machines via the SSH protocol. Boundary can then use Vault’s SSH secrets engine to generate ephemeral signed SSH certificates and provide users with passwordless SSH connectivity using a method called credential injection. With credential injection, the Boundary controllers return credentials to the Boundary workers, or self-managed workers, and start a session with an SSH target where the user/client never has access to the credentials.

The diagram below illustrates the holistic workflow and shows the integration between Boundary and Vault.

When a user initiates an SSH connection to an SSH target host, a public and private key are generated. There are two options here, depending on the type of Vault endpoint that is used within the Boundary credential library for SSH certification:

• /issue: Vault generates and signs the key pair for you.
• /sign: The Boundary controllers generate the key pair and then send it to Vault to sign.

The workflow depicted in the diagram above is explained below:

1. Boundary generates the public and private key.
2. The public key is then sent by the Boundary controllers to Vault.
3. Vault signs the key and generates an SSH certificate.
4. The keys and certificate are sent by the Boundary controller to the self-managed worker.
5. The self-managed worker injects the credentials into the session, so when the user connects to the target via SSH they are automatically authenticated and connected to the command-line, without having to know what or where the credentials are.

»Better security outcomes

Traceability and access controls: By handling credentials on behalf of the user, Boundary creates a more secure and streamlined experience for the user. It doesn’t matter how many people are trying to connect to the same target (excluding any consideration around limitations on concurrent connections etc.), a separate public/private key will be generated for each user.

Minimized exposure window: Once the SSH session to the target is terminated, so too are the credentials, which greatly enhances the security posture, as there is no longer the requirement to rely on long-lived credentials or the concern about credentials being leaked.

Less toil: From an end-user’s perspective, the end-to-end workflow is also dramatically improved. There isn’t the necessity for staff to go through any ticketing-based workflows in order to retrieve credentials, which notoriously adds unnecessary toil and hampers the productivity of staff.

»From key sprawl to scalable control with HashiCorp Boundary and Vault

Using SSH keys from an operational perspective is trivial. You add your public key to a host, and authorize and authenticate yourself using your public/private key pair. The problem with SSH key pairs, however, comes with scale. Users have to manage their public keys across all of their infrastructure, trusted access to a machine is defined by the individual key pair, and it’s not uncommon to find unknown public keys in the authorized_keys file of your servers. As you look to connect to more hosts, best practices would encourage SSH rekeying. Users often neglect this practice because it’s complex and time-consuming to manage dozens or hundreds of public keys for sets of SSH hosts, and as a result, these key pairs become usable permanently across your infrastructure.

SSH certificate-based authentication extends key-based authentication with digital signatures. The user’s authenticity is based on possession of a certificate signed by the trusted CA. Instead of having to provision and de-provision keys across your infrastructure, SSH certificates allow you to specify:

• How long a certificate is valid for
• Who will gain access to a target host
• Which users the certificate holder can log in as

Policy becomes centralized, and you avoid having to use the authorized_keys file. SSH certificates, unlike SSH key pairs, are short-lived and self-revoking, but there is still the overhead of having to manage the day-to-day operations of the CA and any system that manages the incoming requests for certificates.

The integration of HashiCorp Boundary and Vault, focusing on SSH credential management, introduces a robust workflow to dynamically create and manage SSH certificates at scale. Vault’s capability to generate SSH certificates on-demand adds an extra layer of security, reducing the risk associated with static credentials or manually generated certificates.

Boundary dynamically injects the certificates during runtime, ensuring fine-grained access control and reducing the potential for unauthorized access or credential exposure. This approach optimizes security, streamlines compliance, and simplifies administrative tasks, all within a centralized, scalable framework. It’s particularly beneficial for organizations operating in multi-cloud environments, offering a comprehensive, centralized solution to efficiently create, manage, and secure SSH certificates in a dynamic and highly secure manner.

If you’d like to see a demo of Vault and Boundary tailored toward your organization’s challenges, send us a note.