Vault deployments often begin with a small number of services retrieving secrets or issuing certificates. As adoption grows, more systems start depending on Vault for authentication, secret retrieval, and certificate management.
CI pipelines retrieve secrets during builds, machines request SSH certificates during provisioning, and applications dynamically issue PKI certificates.
As the number of dependent systems increases, concurrency grows as well. Understanding how Vault behaves under increasing concurrent workloads is important for capacity planning and infrastructure design.
To better understand these performance characteristics, we conducted a series of benchmarks simulating common production workloads across the KV, SSH, and PKI secrets engines.
»How we tested
Vault benchmark deployment architecture using HashiCorp Validated Design
All benchmarks were conducted using the HashiCorp Validated Design (HVD) deployment for AWS. Vault Enterprise (v1.17.3+ent) was deployed using integrated Raft storage in the AWS us-west-2 region. Workloads were generated using the k6 load testing tool, and system metrics were collected and pushed to Datadog for insights.
Tests followed a gradual ramp-up model to observe behavior under increasing concurrency. The load started with a single virtual user (VU) and increased in stages up to 500 VUs for KV workloads, while SSH and PKI tests ranged between 1 and approximately 200–300 VUs depending on the operation being evaluated.
Workloads simulated common production patterns across KV, SSH, and PKI secrets engines.
The results are intended to inform planning and tuning decisions, not to define universal performance limits for all Vault deployments.
»Key findings from the Vault benchmarks
The following sections summarize the most significant performance patterns observed during the benchmarks.
The benchmark results refer to key performance thresholds. The knee point is the concurrency where latency begins to outpace throughput growth. The saturation point indicates where throughput plateaus despite increasing concurrency, and the failure point marks where requests begin to time out or fail.
»1. KV reads consistently outperform writes under concurrency
KV read performance across payload sizes
The KV secrets engine tests revealed a consistent performance gap between read and write operations.
Across all workloads, read operations delivered higher throughput and greater stability under increasing concurrency. Benchmark results showed that reads were approximately 2.3 times faster than writes, and they remained resilient even as concurrency increased. Writes, on the other hand, reached stress points earlier and became the first operations to fail under heavy load.
Payload size further amplified this difference. For large payloads such as 1 MB secrets, write operations for the KV2 engine began failing between 50 and 100 virtual users, while reads continued operating at higher concurrency levels.
These results highlight an important operational characteristic of Vault: Write-heavy workloads place greater pressure on the Raft storage path, since write operations must be committed and replicated across nodes. As a result, writes tend to become the primary bottleneck when concurrency increases.
»2. Payload size has a significant impact on throughput
KV throughput efficiency across payload sizes
One of the clearest trends across the benchmarks was the effect of payload size on system performance.
Small secrets scaled well across all tests. For example, 1 KB reads remained highly scalable and did not reach saturation even at 500 virtual users. Larger payloads behaved very differently. Secrets sized at 100 KB and 1 MB reached knee and saturation points much earlier, limiting the number of concurrent operations the cluster could sustain.
Throughput measurements showed a dramatic decline as payload size increased. While 1 KB reads achieved thousands of requests per second, 1 MB reads dropped to single-digit requests per second under the same test conditions.
The root cause is the increased cost of processing and committing larger payloads. Larger secrets generate greater disk write activity, increase Raft replication overhead, and place additional pressure on the storage subsystem.
Additionally, KV2 introduces metadata overhead compared to KV1, which can impact throughput even when only a single version is stored.
These results reinforce a common best practice: Secret payload size has a direct impact on the scalability of Vault workloads.
»3. Memory and storage behavior strongly influence KV scalability
Memory utilization vs. concurrency during KV benchmark workloads
System metrics collected during the KV benchmarks revealed several patterns in resource utilization.
Memory usage increased significantly as concurrency grew, eventually reaching 100% utilization across nodes during tests with 500 concurrent users. This made memory saturation the most prominent constraint observed during the benchmarks.
CPU usage behaved differently. During KV workloads, CPU utilization remained relatively moderate, typically between 20% and 40%, indicating that raw compute capacity was not the primary limiting factor.
Disk behavior also played an important role. Large payload writes produced noticeable spikes in disk write activity, which correlated with increases in tail latency. Because Vault uses Raft integrated storage, write operations must be committed to disk before they are considered successful.
These patterns suggest that memory availability and storage performance have a stronger influence on throughput than CPU capacity in many Vault workloads.
»4. ED25519 improves scalability for SSH workloads
SSH signing performance comparison between RSA-2048 and ED25519
The SSH secrets engine benchmarks compared certificate signing performance between RSA-2048 and ED25519 cryptographic algorithms.
The results showed a clear difference in concurrency tolerance between the two algorithms. RSA-2048 operations reached their knee point at around 100 virtual users, while ED25519 extended this threshold to approximately 200 virtual users, effectively doubling the concurrency level before latency degradation began.
Saturation behavior followed a similar pattern. RSA leader nodes saturated at around 250 virtual users, whereas ED25519 workloads continued scaling beyond 200–300 virtual users without reaching full saturation during the tests.
Another notable observation was the system’s resilience under load. Across the SSH benchmarks, the cluster degraded gradually as concurrency increased rather than experiencing abrupt failures.
Node-level metrics provide additional insight into this behavior. For RSA workloads, CPU utilization peaked between 92% and 96% at 250 virtual users, whereas ED25519 workloads remained significantly lower at approximately 40–42%, even at comparable concurrency levels, reflecting their computational efficiency.
Memory utilization showed a different pattern across both algorithms. While overall usage remained moderate during peak load, memory levels did not fully return to baseline after test completion, indicating potential resource retention that could impact long-running workloads.
These results indicate that algorithm selection can influence scalability for high-volume SSH signing workloads.
»5. PKI workloads reach latency limits earlier than KV workloads
PKI certificate issuance latency vs. concurrency
PKI secrets engine operations involve cryptographic certificate generation and revocation, making them more computationally intensive than KV secret retrieval.
The benchmark results showed that certificate issuance latency began increasing sharply at around 25 concurrent users, which marked the knee point for the tested configuration. At this stage, throughput plateaued while latency increased rapidly.
The baseline latency for issuing a certificate with a single virtual user was approximately 560 milliseconds, and tests showed minimal performance differences across the evaluated algorithms, including RSA, ED25519, and ECDSA.
At higher concurrency levels, all operations maintained 100% success below 100 virtual users, but beyond that point, revocation operations began experiencing timeouts, and failure rates increased as concurrency approached 200 virtual users.
These results highlight that PKI workloads can reach latency limits earlier than KV workloads, particularly when certificate issuance and revocation operations are performed concurrently.
»Operational recommendations for platform teams
The benchmark observations highlight several practical considerations for teams operating Vault in high-concurrency environments.
»1. Keep secret payloads small
Payload size had the largest impact on throughput across the KV benchmarks. Payloads below 100 KB scaled well even at high concurrency levels, while larger payloads reached knee points earlier and experienced faster throughput degradation.
It is important to note that these KV benchmarks were conducted with audit logging disabled to isolate core engine performance. In production environments where audit logging is enabled, additional overhead may further amplify the impact of larger payloads. SSH and PKI were performed with audit enabled.
When possible, store large objects such as certificate bundles, configuration files, or binaries in external systems like object storage or databases and use Vault to store references or credentials needed to access those resources.
»2. Optimize storage performance
Write-heavy workloads depend heavily on disk performance because Raft storage requires write operations to be committed to disk. Using storage with high write throughput and fast fsync performance (such as high-performance SSD or NVMe) can help reduce latency and improve stability under load.
»3. Scale memory for high-concurrency workloads
During the KV benchmarks, memory utilization increased significantly at high concurrency levels, reaching full utilization at 500 concurrent users. Increasing node memory capacity can improve resilience when handling large numbers of concurrent requests.
»4. Choose efficient cryptographic algorithms for SSH workloads
The SSH benchmarks showed that ED25519 handled higher concurrency before latency degradation compared to RSA-2048. For environments that rely heavily on SSH certificate signing, algorithm selection can influence scalability.
»Final thoughts
Vault is designed to handle demanding workloads, but its performance characteristics depend heavily on workload type, concurrency levels, and infrastructure configuration.
For platform engineers, the key takeaway is that capacity planning should consider concurrency patterns, payload size, and storage performance alongside traditional CPU and memory metrics.
Understanding these characteristics helps teams build resilient Vault deployments as infrastructure scale increases.
»Learn more
For guidance on tuning Vault for optimal performance, including OS, storage, and configuration best practices, see the official Vault documentation.
