Refresh Secrets for Kubernetes Applications with Vault Agent
Learn the system signal and live reload methods for updating Kubernetes applications when secrets change. See an example via a Spring Boot application.
Some applications include a HashiCorp Vault client library in their code to retrieve and refresh secrets directly from the Vault API. But what happens if you cannot or do not want to include a library in your application? You might run different types of applications on a Kubernetes cluster and want to standardize how they refresh secrets without significantly refactoring application code. Rather than write code to retry and refresh secrets from the Vault API, you can instead run Vault Agent as a sidecar, which reduces the need for your application to directly connect to Vault.
This post shows you how to use Vault Agent to update secrets for an application on Kubernetes using a termination signal or live reload when a secret changes. By relying on Vault Agent pushing a command to reload, your application stays updated with new secrets regardless of the programming language or application framework. Without Vault Agent, you would have to refactor your application to pull new secrets from Vault with retry handling and connection reloading.
»Demo Application with Vault Agent
This post uses a demo application named “payments-app” that submits payments to a processor and records the payment in a database. The application requires a database username and password generated by the PostgreSQL secrets engine for Vault. The “payments-app” uses Spring Boot, a Java-based framework. However, you can apply the patterns in this post to other application frameworks in different programming languages.
The database username and password for the application has a maximum lease of four minutes for demonstration purposes. Vault Agent renews the lease for the username and password before they expire. For the “payments-app”, add Kubernetes annotations to inject Vault Agent as a sidecar. Vault Agent runs as another container and handles the retrieval of new database credentials.
apiVersion: apps/v1kind: Deploymentmetadata: name: payments-appspec: replicas: 1 selector: matchLabels: service: payments-app app: payments-app template: metadata: labels: service: payments-app app: payments-app annotations: vault.hashicorp.com/agent-inject: "true" vault.hashicorp.com/role: "payments-app" ## omitted for clarityNext, configure Vault Agent to cache a Vault token and secrets after initialization. Enabling the cache ensures that the sidecar contains the secret and Vault token and avoids reloading the application when it starts.
apiVersion: apps/v1kind: Deploymentmetadata: name: payments-appspec: ## omitted for clarity template: metadata: ## omitted for clarity annotations: ## omitted for clarity vault.hashicorp.com/agent-cache-enable: "true" ## omitted for clarityFinally, define a set of annotations for Vault Agent to create a file with the database credentials. Vault Agent retrieves the username and password and writes them to a file named /vault/secrets/database.properties in a shared volume mount. The suffix of the annotation should match the filename you want to create with the secrets.
apiVersion: apps/v1kind: Deploymentmetadata: name: payments-appspec: ## omitted for clarity template: metadata: annotations: ## omitted for clarity vault.hashicorp.com/agent-inject-secret-database.properties: "payments/database/creds/payments-app" vault.hashicorp.com/agent-inject-template-database.properties: | spring.datasource.url=jdbc:postgresql://payments-database:5432/payments {{- with secret "payments/database/creds/payments-app" }} spring.datasource.username={{ .Data.username }} spring.datasource.password={{ .Data.password }} {{- end }}If you have secrets at different Vault paths, you can add different annotations for each set of secrets. The suffix of each annotation should match the filename containing the secrets. For example, the payment processor’s credentials exist at a different Vault path, so you would add a second set of annotations to write the API username and password to /vault/secrets/processor.properties.
apiVersion: apps/v1kind: Deploymentmetadata: name: payments-appspec: ## omitted for clarity template: metadata: annotations: ## omitted for clarity vault.hashicorp.com/agent-inject-secret-database.properties: "payments/database/creds/payments-app" vault.hashicorp.com/agent-inject-template-database.properties: | spring.datasource.url=jdbc:postgresql://payments-database:5432/payments {{- with secret "payments/database/creds/payments-app" }} spring.datasource.username={{ .Data.username }} spring.datasource.password={{ .Data.password }} {{- end }} ## omitted for clarity vault.hashicorp.com/agent-inject-secret-processor.properties: "payments/secrets/data/processor" vault.hashicorp.com/agent-inject-template-processor.properties: | payment.processor.url=http://payments-processor:8080 {{- with secret "payments/secrets/processor" }} payment.processor.username={{ .Data.data.username }} payment.processor.password={{ .Data.data.password }} {{- end }}What happens when the database username and password change? Your application needs to reconnect to the database with the new username and password. As a result, Vault Agent needs to send a signal to your application to reload the new credentials.
»Update Secrets by Application Restart
One technique to update secrets for an application involves restarting the application instance with a termination signal like SIGTERM. When your application gets a SIGTERM, it handles the signal by disconnecting connections to external services and shuts down. Then, Kubernetes schedules a new application container. When you use this approach, make sure that you have multiple instances of your application available as the signal will shut down the application instance and disrupt your service.
You can configure Vault Agent to send a SIGTERM. To start, make sure your application responds to the SIGTERM by shutting down connections gracefully. For example, “payments-app” includes a Spring Boot property to gracefully shut down its web server after the application completes active requests. Set server.shutdown=graceful in the application’s application.properties file.
server.shutdown=graceful
Your application may require additional code to handle the signal and complete active requests.
To tell Vault Agent to issue the signal, add a SecurityContext for the “payments-app” to run as a specific user ID. Then set the vault.hashicorp.com/agent-run-as-same-user annotation to true in order to allow the Vault Agent to use the same user ID as the application. Vault Agent also needs access to the application container’s processes in order to issue a termination signal for reloading. Running as the application’s user ID and setting the shareProcessNamespace attribute allows Vault Agent to send the signal to the application process. Note that sharing process namespaces gives all containers in the pod visibility into each other’s processes and filesystems.
apiVersion: apps/v1kind: Deploymentmetadata: name: payments-appspec: ## omitted for clarity template: metadata: ## omitted for clarity annotations: ## omitted for clarity vault.hashicorp.com/agent-run-as-same-user: "true" spec: shareProcessNamespace: true containers: - name: payments-app ## omitted for clarity securityContext: runAsUser: 1000 runAsGroup: 3000Besides sharing process namespaces, you need to define the command Vault Agent sends to the application. Add the vault.hashicorp.com/agent-inject-command annotation with the suffix for the database.properties secret to the deployment and send the SIGTERM to the process ID of “payments-app”, which runs as a Java process.
apiVersion: apps/v1kind: Deploymentmetadata: name: payments-appspec: ## omitted for clarity template: metadata: ## omitted for clarity annotations: ## omitted for clarity vault.hashicorp.com/agent-inject-command-database.properties: | kill -TERM $(pidof java) ## omitted for clarityWhen Vault rotates the secret, the Vault Agent injector retrieves a new set of database credentials, writes them to the shared volume mount, and sends the termination signal to the application. The application container shuts down and gets rescheduled by Kubernetes. When a new application instance starts, it reconnects to the database with the updated credentials.
»Refresh Secrets through Live Reload
Although you can use a termination signal across many application frameworks, you may find it disruptive to restart the application container each time a secret changes. As a less disruptive alternative, some frameworks support the ability to reload an application upon configuration changes without shutting down the application. Other application frameworks support automatic live reload when properties change. If you use live reload, you do not have to wait for Kubernetes to reschedule the application container.
For example, Spring Boot offers an actuator module that supports operations use cases like observability. Invoking its refresh endpoint triggers an event that signals to any interested components that something in the application has changed and that they should reconfigure themselves accordingly. The examples in the repository cover the additional dependencies and code for a Spring Boot application to live reload.
»Configure Application
Install org.springframework.boot:spring-boot-starter-actuator in your dependencies. Update the application’s application.properties to import configuration from files created by the Vault Agent injector using the spring.config.import property.
Finally, enable the actuator endpoint for /refresh.
spring.config.import=file:${CONFIG_HOME:/vault/secrets}/database.properties,file:${CONFIG_HOME:/vault/secrets}/processor.properties
management.endpoints.web.exposure.include=refresh
Note the spring.config.import property references the CONFIG_HOME environment variable. You can use this environment variable to override the shared volume path to the Vault secrets. If you do not define this environment variable, Spring reads the configuration from the default volume path at /vault/secrets.
When the application starts, it gets the database credentials from /vault/secrets/database.properties and connects to the database. Each time you make an empty HTTP POST request to the /actuator/refresh endpoint, the actuator checks for configuration differences between the application’s environment and the file properties. If it finds a difference, Spring publishes a refresh event.
You can have Spring recreate a whole bean with the @RefreshScope annotation annotation. Place this Java annotation on any Spring component or bean provider method to create a proxy object. The proxy object listens for refresh events and recreates the internal instance.
Note that @RefreshScope will completely destroy and then recreate a given bean with no regard to its internal state. For additional fine-grained control over an object, use a bean of type ApplicationListener<RefreshScopeRefreshedEvent>. Interested components may pull down the updated configuration and manually refresh their own internal state.
Define a database connection DataSource that refreshes each time the properties used to configure it have changed. Inject Spring Boot’s DataSourceProperties instance and create a refreshable DataSource bean.
package com.hashicorpdevadvocates.paymentsapp; import org.springframework.boot.SpringApplication;import org.springframework.boot.autoconfigure.SpringBootApplication;import org.springframework.boot.autoconfigure.jdbc.DataSourceProperties;import org.springframework.boot.jdbc.DataSourceBuilder;import org.springframework.cloud.context.config.annotation.RefreshScope;import org.springframework.context.annotation.Bean; import javax.sql.DataSource; @SpringBootApplication// omitted for claritypublic class PaymentsApplication { public static void main(String[] args) { SpringApplication.run(PaymentsApplication.class, args); } @Bean @RefreshScope DataSource dataSource(DataSourceProperties properties) { return DataSourceBuilder.create() .url(properties.getUrl()) .username(properties.getUsername()) .password(properties.getPassword()) .build(); } // omitted for clarity }For other properties, like the payment processor’s credentials, you can use @ConfigurationProperties or @Value annotations to define custom properties and refresh them.
package com.hashicorpdevadvocates.paymentsapp; import lombok.Data;import org.springframework.boot.context.properties.ConfigurationProperties;import org.springframework.cloud.context.config.annotation.RefreshScope; @RefreshScope@ConfigurationProperties(prefix = "payment")@Dataclass PaymentAppProperties { private Processor processor = new Processor(); @Data static class Processor { private String url; private String username; private String password; }}Inject the PaymentAppProperties instance and create a refreshable PaymentProcessorClient bean, which the application uses to submit payments to the processor.
package com.hashicorpdevadvocates.paymentsapp; import org.springframework.boot.SpringApplication;import org.springframework.boot.autoconfigure.SpringBootApplication;import org.springframework.boot.context.properties.EnableConfigurationProperties;import org.springframework.boot.jdbc.DataSourceBuilder;import org.springframework.cloud.context.config.annotation.RefreshScope;import org.springframework.context.annotation.Bean; import javax.sql.DataSource; @SpringBootApplication@EnableConfigurationProperties(PaymentAppProperties.class)public class PaymentsApplication { public static void main(String[] args) { SpringApplication.run(PaymentsApplication.class, args); } // omitted for clarity @Bean @RefreshScope PaymentProcessorClient paymentProcessorClient(PaymentAppProperties properties) { return new PaymentProcessorClient( properties.getProcessor().getUrl(), properties.getProcessor().getUsername(), properties.getProcessor().getPassword()); } // omitted for clarity}Upon refresh, Spring Boot reloads the new values in the Spring Environment object, which sources configuration keys and values from the application’s external property files. Vault Agent can now use the refresh mechanism to reload objects and values in the application.
»Configure Vault Agent
To configure Vault Agent to refresh configuration using the Spring Boot actuator, add the vault.hashicorp.com/agent-inject-command annotation with the suffix for the database.properties secret to the deployment and include a command to send an HTTP POST request to the application’s /actuator/refresh endpoint.
apiVersion: apps/v1kind: Deploymentmetadata: name: payments-appspec: ## omitted for clarity template: metadata: ## omitted for clarity annotations: ## omitted for clarity vault.hashicorp.com/agent-inject-command-database.properties: | wget -qO- --header='Content-Type:application/json' --post-data='{}' http://127.0.0.1:8081/actuator/refreshVault Agent sends the refresh request to Spring Boot, which live reloads the database configuration without disrupting the web server. The live reload gracefully disconnects database sessions, reads in new configuration, and creates new database sessions while maintaining the application’s availability.
Note that the Spring Boot refresh endpoint could trigger disruptive changes to your application, such as creating new data sources. Secure individual actuator endpoints by restricting access, making them inaccessible to public traffic, and adding certificates.
»Conclusion
You can use Vault Agent to standardize the retrieval and refresh of secrets across different application frameworks running on Kubernetes. For applications that support a live reload, add a Vault Agent annotation to the deployment to make a refresh request to the application. If your applications do not support a live reload, use termination signals to restart the application.
For more detailed configuration information, review the demo application’s code repository. Use additional annotations to configure multiple secrets and settings for the Vault Agent sidecar injector.
If you use Spring and want to refresh configuration directly from secrets in Vault, check out the documentation for Vault as a backend for Spring Cloud Config Server. For Spring applications that need to access the Vault API directly, review the documentation for Spring Cloud Vault. The library supports the refreshing secrets and encrypting data using Vault’s Transit secrets engine.
Thank you to Josh Long, Spring Developer Advocate at Tanzu, a division of VMware, for educating and clarifying Spring Boot details for this post.
Questions about this post? Add them to the community forum!
Sign up for the latest HashiCorp news
More blog posts like this one
Introducing The Infrastructure Cloud
Do cloud right with The Infrastructure Cloud from HashiCorp. Unlock developer potential while controlling cloud costs and risk.
How secret scanning works
HCP Vault Radar conducts ongoing reconnaissance of unsecured secrets stored as plain text in code repositories as well as configuration, DevOps, and collaboration tools.
Secrets sync now available on Vault Enterprise to manage secrets sprawl
Secrets sync is a new feature in HashiCorp Vault that facilitates centralized management, governance, and control of secrets for multiple external secret managers.