Learn how to use the Prometheus Operator with the new Vault Secrets Operator for Kubernetes to monitor secrets in a Grafana dashboard.
HashiCorp recently proposed a new method to consume Vault secrets — the Vault Secrets Operator. By implementing a Kubernetes Operator, you can now synchronize your secrets from different Vault secrets engines like Key/Value, Database, or PKI to a Kubernetes secrets store. As a result, without any code application changes, you can now sync Vault secrets to Kubernetes Secrets.
As teams adopt the Vault Secrets Operator for Kubernetes, it becomes a critical component of their platforms and must be monitored closely. This post will walk you through how to use the Prometheus Operator and the Prometheus metrics endpoint provided in the Vault Secrets Operator. This allows users to be alerted to any potential issues with the system. You’ll also see how to integrate Prometheus output into a Grafana dashboard.
In this walkthrough, the Vault Secrets Operator starts out deployed in a dedicated namespace called vault-secrets-operator with Helm. This is in line with our Installing the Vault Secrets Operator documentation. The Prometheus Operator will be used with kube-prometheus to simplify integration. At the end of installation, your setup should look like this:
The Vault Secrets Operator is in charge of syncing secrets from HashiCorp Vault to Kubernetes Secrets and exposes metrics through the Prometheus endpoint. Those metrics are then scraped by the Prometheus Operator, which provides alerts and is used as a datasource for the Grafana dashboard.
The Vault Secrets Operator was built using Kubebuilder (an SDK to build Kubernetes APIs), which exposes a set of protected metrics by default. As mentioned in the Vault Secrets Operator announcement, there are a number of metrics available through an endpoint supported by Prometheus Operator.
The first step in the deployment process is to deploy kube-prometheus: a packaged method for deploying Prometheus and monitoring-related solutions like alertmanager for alerting and Grafana for dashboards.
Install kube-prometheus using Helm:
The code above deploys your Prometheus server with alertmanager and Grafana.
Next, forward port 9090 for a Prometheus service:
You should get access to the Prometheus UI, but note that the Vault Secrets Operator does not yet appear as a target for Prometheus scraping.
Without kube-prometheus, you would need to grant permissions to your Prometheus server so it can scrape the protected metrics. That process would involve creating a ClusterRoleBinding to bind the ClusterRole to the service account that your Prometheus server uses. But because we’re using kube-prometheus, this binding is already deployed.
When deploying the kube-prometheus from the Helm chart, several custom objects are already created in the Kubernetes cluster. One of them is the prometheus
object, which defines all configurations used by Prometheus (such as scraping endpoints, scraping interval, etc.). Here is the default config that gets deployed:
You can see that an object called kube-prometheus-kube-prome-prometheus
was created, and it contains all the configurations.
The Prometheus Operator offers multiple ways to define resources to be monitored. For the purposes of this blog post, we will use the serviceMonitorSelector:
Looking deeper, you can see that Prometheus is considering all ServiceMonitor
instances with the label: release:kube-prometheus
(by using the serviceMonitorSelector).
So, you will next create a ServiceMonitor
object for the Vault Secrets Operator. HashiCorp has a sample ServiceMonitor that you can use to start building your configuration, adding the label name: controller-manager-metrics-monitor
:
In the configuration file, you will also see that the metric endpoint is protected by a token (bearerTokenFile). This is because Kubebuilder embeds configurations that enforce authentication and authorization as a default value through an auth proxy deployed as a sidecar container in the manager pod.
Apply the following configuration:
The chart below summarizes the interaction between the Prometheus Operator and the Vault Secrets Operator:
You’re now finished with the Prometheus Operator configuration. Go back to the Prometheus UI, where “controller-manager” should be coming up as a target in the Targets section:
Next, create a Grafana dashboard to visualize the data coming out of the Vault Secrets Operator. To integrate it into your stack, you may want to expose the Grafana port using an ingress controller in production, but for the purposes of this blog post, you can forward the port using this command:
You should now have access to the Grafana interface by typing http://localhost:9091 into your browser. Log in with admin
as username and get the password from the Kubernetes secret store:
Once you’re logged in, you can import the dashboard by going into Dashboards / import section and using dashboard ID 18656.
The dashboard should look like this:
It’s important to know that the Vault Secrets Operator is exposing all the reference metrics from Kubebuilder SDK, and the documentation needed to get a better understanding of those metrics can be found in this list of Default Exported Metrics References.
This dashboard is split into three parts:
This dashboard is a sample. We encourage you to customize it with critical data more closely related to your business.
Kube-prometheus contains several examples of cluster alerting rules. You can find those rules in the UI using the /rules endpoint or through the command line with this command:
No rules are created initially to monitor your Vault secrets system.
For this demo, create this set of rules:
These rules add alerting on vaultconnection/vaultauthentication controller issues. If you are using the vaultpkisecret or vaultdynamicsecret controllers on different namespaces, you may want to make similar alerts for those as well.
As more users begin to adopt the Vault Secrets Operator, it will become a critical part of the production stack in charge of syncing secrets between a centralized HashiCorp Vault and Kubernetes secret store. This means that teams will need to closely monitor this operator.
The example workflow in this blog post is intended to help you get started building an observability solution for the Vault Secrets Operator based on the Prometheus Operator with Grafana for dashboarding and Alertmanager for alerting.
Learn about the ACME protocol for PKI, the common problems it solves, and why it should be part of your certificate management roadmap.
New HashiCorp Vault ecosystem integrations extend security use cases for customers.
With Vault and Boundary, HashiCorp makes its debut in Gartner’s Magic Quadrant for privileged access management.