Vault at GM Cruise: Securing Autonomous Vehicles and the Humans Who Build Them
Nov 19, 2018
Building frictionless security into the everyday workflow of engineers is a great challenge. Achieving this becomes increasingly difficult when the product that is being built is an autonomous robot.
Building security into the everyday workflow of engineers is a great challenge. In order to be effective as a security engineer, you must provide frictionless security. Achieving this becomes increasingly difficult when the product that is being built is an autonomous robot.
This talk will cover the measures being taken to ensure that the humans who are responsible for building and deploying autonomous vehicles can do so with built-in world-class security. Then, we'll discuss how the same systems that were built for humans, were adapted to work with robots.
- Brian NuszkowskiSecurity Engineer, Cruise Automation, GM Cruise
My name's Brian Nuszkowski. I work for a company here in San Francisco called Cruise Automation. You may have seen me earlier stumble onto the stage for approximately 1 minute and 50 seconds and ramble off a few sentences. Hopefully now I have more time to elaborate on some of the bullet points I was talking about.
I've tried my best to encapsulate the three themes in the presentation. I tried my best to keep each theme bundled together, but unfortunately, there may be some bleed-over where we're discussing one type of topic and it doesn't connect. I apologize in advance, but I'm really going to try my best to weave them all together and give a bigger picture.
Those three themes are: a philosophical discussion briefly, so if philosophy is not your thing, I'm sorry. We're going to go deep into some nuances of security and life, et cetera. Then we're going to talk about some scaling topics. Not so much scaling from a technical perspective, because lucky for me, someone else this afternoon is going to cover scaling Vault, so I got off easy. I don't have to go into all the, "Oh, here's how you do your N-whatever clusters." I believe it's right after this presentation, so when I'm done, you can go to the next. If you're on the Vault train today, you can continue to hop aboard. I believe it's the next session. Those are the expert guys, so you can probably trust them more than me.
Last, we'll go over some cool things that I think we're potentially working on. You may not find them so cool, disclaimer. Second disclaimer is that this is not a one-size-fits-all situation. This is more of a thought exercise of, "Hey, I like how he thought of this one particular component. It's not going to work for me." It probably won't, so just keep that perspective that some of the things you hear me say may or may not be applicable to you, but try to inspect each component of it to say, "Okay, this could potentially be used in my organization."
That's my only goal. If you could walk away with one small thing, I'd like you to walk away with maybe two or three things that would be beneficial, but if it's one small thing, that's my real goal here.
The dishwasher philosophy
First, we'll get into the philosophy component, what I call the dishwasher philosophy. Who here has a dishwasher at their home? Do I have to explain what this is? It's a cube. You put dirty dishes in. Water sprays. Your dishes come out clean. Who here has an office dishwasher that's an absolute disaster? You think it'd be really easy -- you throw a bowl in, the magic machine cleans it -- but it's not.
The problem is that oftentimes it's a simple concept, but it's executed poorly in environments where it's not well understood, or it's complicated, or it's rushed, et cetera, so you can draw those parallels to building your own Vault environment. If you're not thoughtful and considerate, and think of the external factors of the things you need to do, the security initiative of Vault can turn out poorly. Execution and planning is everything.
You come to this talk today and say, "You know what? I'm going to deploy this Vault cluster. Brian's going to give me all the answers, and I'll have to think about it once. It'll be like the Showtime Rotisserie & BBQ: I'll set it and forget it, and I'll go about my way and reach my compliance checkbox." That's not going to be the case.
Trying to get you to think about, "How will we do things thoughtfully? How will we do things flexibly, and things that can potentially scale with changes in the future?" I don't know about you, but the business I'm in, things change often. If we go with the one big Solution X, maybe we're not very flexible, we're very rigid, then surprise, two months later, when the next person comes in with the better idea, we can't adapt to it well and deliver a good security solution.
As luck would have it, our problem sets are pretty similar. Who here is responsible for running some type of application that runs on a computer? Good. So are we. The application in this case happens to be a car with some computers in it. That's it at its simplest element, but really, they're kind of the same thing.
I'm not here to tell you that we've been tucked away in a lab somewhere developing some new cryptographic primitives, like we're going to release something better than some key size with this type of cryptography. We didn't do that. What we want to do is leverage things that were flexible, that adopt industry best-standard primitives and put them in the environment, because we want to deliver solutions as quickly as possible.
If, for example, there's a group of developers that needs to deliver something, and we need to have a security component involved, we want to be able to deliver that solution quickly, and not be a roadblock and not be complex, the typical, "Oh, you cannot do that because we have to follow X, Y, Z."
Autonomous-vehicle security objectives
What are our security objectives for securing autonomous vehicles? We don't just have to think about the vehicle, we have to think about the whole picture. By itself, the vehicle goes, and it's secure, and that's a good thing. That's a good feeling, but there are all these peripheral systems that are around it interactively, and those are key too. It's looking at the whole picture. How do we secure the vehicle? How do we secure all the components, that are just as important, that connect to the vehicle and make this thing go? We want a healthy security ecosystem.
Introduce minimal friction
Our primary security objectives are to introduce minimal friction, so if we come up with a new process or implementation, it needs to be implemented as smoothly as possible, with as little friction as possible, because most security organizations don't have the best reputation, so we're trying to improve that by delivering easy-to-use, nice solutions.
Trust and protect
Obviously the other two bullet points are the information security industry's de facto: We want to trust and protect all the data. We want to trust who the data's coming from, that we can go through some type of process to vouch for the thing, trying to do whatever it's trying to do, and we want to protect it so no one else can see it.
We also want to be successful. That's our goal as a security organization, to achieve the objective.
Be ready to build
One and four are pretty important right now, but the last one, which I've bolded for you, so add some extra emphasis, is be ready to build. That means that you have this one tool, Vault. Be ready to identify gaps in the system, and be ready to develop tools for those.
Maybe if you're not a software engineer by training, but you've always wanted to be, or you have someone on your team who wants to shine in that component, you should leverage that and be ready to identify pieces that aren't there, and build those pieces. Whether or not that someone announces them onstage today or tomorrow, you should probably be ready to build that, plug it in, so you don't hold up the rest of the org. That's the number 1 piece of advice I can give.
How we've done stuff in the past: I have this file that exists in a code repository, and we have these standard complexities and primitives, like a cryptographic operation. Typically, up until now, if you haven't implemented Vault, you have these miscellaneous tools. Maybe you like to use OpenSSL command line to encrypt files. Maybe you like to use some Python wrapper. The point is, there are all these tools that are inconsistent, so Team A encrypts with GPG, Team B doesn't, so when you need to share a file, you're going to have to go to Stack Overflow and look it up.
You can be honest. You don't remember the flags to import a key and do all this stuff. It's a not very fun experience, and you don't want to admit it either. That's the number 1 thing security people, maybe even other people don't want to admit is, "I just don't quite understand that thing, because I haven't spent the most time with it."
The second step you should take is, just admit it. This is not my expertise. Who here has a PhD in math? Me either. Good. We're among friends. Guess what? I'm not going to be able to solve these complicated equations, so just own up to it. There are a ton of really smart, thoughtful, hard-working, well-educated people solving these problems for us, providing these extractions so that we can leverage them.
In this case, we have the standard cryptographic primitives, and we have some abstractions on top. Then what Vault gives us is an HTTP API. I'm pretty sure that most people here can use an HTTP library, or at least figure it out. That's a common language. Math was the common language before. It was a difficult language to learn, so we move up the stack a few pieces. Now we're talking HTTP, a language that we can both understand, so this immediately makes our environment much easier.
If we were to implement a security tool, let's say, some confusing binary protocol that is not easy to read and understand and well known, the security tool you implement might not be as successful. Think about that.
Set consumer expectations
Our organization, if I were to plot the growth, it'd be the standard line, linear. Everyone has a linear growth line. It's the typical thing. Our organization grows, and as it grows, initially, if you're lucky enough, like me, to incorporate these security philosophies off the bat, you can set that expectations for, in this case, authentication.
How often should a user authenticate, an end user or an engineer? Come up with realistic expectations, maybe 22 hours, maybe once a day, maybe that's not acceptable to an org, but the problem is, you're dealing with several humans, and you have to negotiate a social contract of, "I want the development team working on Product X to authenticate every 14 hours, to prove they are who they are, to access these secrets." It's a little bit harder to sell than, "I have a robot that expires every 2 hours. I can just develop software that A) makes it inspect its token and B) renew it automatically." Robots are the easy part about this equation. Humans are the hard part.
Initially, set the expectation of, "Hey, we're going to adopt Vault in our organization to work on this large project. As part of this project, you need to authenticate on X interval." This is where I'm starting to blend those three things I talked about earlier. I'm already doing it. I'm only 9 minutes in. You can leverage that thing that they do, like authenticating to Vault to get some kind of identity token to prove who they are, you can use that to leverage other things. Spoiler.
Now that we've set the expectation to our development group and end-user group that, "You're going to authenticate on a reasonable interval, and you're going to do 2FA and all that good stuff, the standard de facto thing." We move on to the onboarding strategy.
As you ride that linear graph of goodness, of feel-good company growth, up and to the right, you have the problem of getting people onboarded to Vault as well. While it might be as simple as, "Just use it," and people come up to you with the sentence that says, "I want to store a secret." Great. You have to have a good answer for that, and do you? "Oh yeah, just use Vault," and they'll go look at this Wiki page and it may or may not be very beneficial to them. They'll become disgruntled, and say, "You know what? Maybe I'm not going to use Vault. I'm going to go back to the OpenSSL command line, because it's easy for me to remember, and I don't have to go through this whole entire new process."
The onboarding strategy is extremely important, so I've developed these four onboarding questions that you may take and use in your organization. Do not give me credit.
Identify the data type that will be stored/manipulated by Vault
Someone comes to you and says, "Hey, I'd like to use Vault." Identify the data type they want to store. Is it as simple as, "I need to just store some string. I need to retrieve it later," or dig deeper in the weeds? I'm about to bleed over to another topic, but, "Hey, I have this signing operation that takes place using this proprietary thing. It's kind of standard, but kind of not. And we want to implement this into Vault." Gather that information first.
Identify who will administratively manage this data
Let's identify who's gonna administratively manage this data. That's always the important thing. People mostly think about who's gonna consume it, but it's also important to identify who's gonna manage the data.
Identify and document who (or what) will have access to the data
From day one you may populate some secrets and a bunch of users can then use it. If you don't clearly define who owns that, who's responsible for maintaining that, and set up those policy and authentication boundaries from the get go, you're gonna have a hard time. So after you're like "Okay, we want to store this string that's called Awesome Password, the most basic thing you can possibly do. This Team X is going to support it. We're cool. Now who's gonna have access to this data?" Is it going to be a service account? Is it an application? Is it an end user? 'Cause that can then change your question to number 4.
Identify and implement adequate authentication and retrieval
You have to identify and implement an adequate authentication method. If it's humans who need access to the data, great. You've already set that expectation that every developer's gonna come in, and they're gonna get the nice cup of coffee. They're gonna read Hacker News for 23 minutes. Then, after that you can authenticate to Vault, get your token. And if that's the case here? Cool, you're good.
But, if it's more complex like, "Hey, I've a cloud instance." Whether it's either platform that needs to authenticate, then you need to identify that and set that up. Once you've identified that, if you have a solution, you deliver it. If not, you go find a solution, and you implement it. Luckily, there are a million different authentication providers that have already been built, and you can use those.
Those are cool things. People are aware, "Okay, I gotta auth the Vault once a day. Here's the things I can do to get it onboard." Another disclaimer: This is not perfect, but it's better. While this still may be confusing to some people, take this and find another dimension. Maybe number 2 will always be your security at work. So don't worry. Now it's a three-step process for you, so you can move on.
Your adoption has continued to ride that awesome high curve, and at this point utilization is through the roof. Everyone's happy. Things are super secure. Now comes the next deep-level topic: How do you choose the path structure? How do you choose where secrets are stored? Which team manages the path, the concepts?
Don't dicate the path, encourage it
Some people often come to a security org, and they say, "Tell me what to do." They get into this case where they don't know what to do. The fear of admission of guilt. And say, "Well, do X," even though X may not be the best solution. Don't dictate the path, you should encourage it. As a team of developers over here, and developers here, what is the best thing that you can come up with that can support all of your applications? Instead of me as a security engineer saying, "You need to use slash secret, slash A, slash B, slash C." That's pretty rigid, because you don't wanna say, "Hey, I don't really know your workflow. Dictate it to me, and I'll help implement it." That's been a pretty valuable component, at least in our implementation.
We're almost to the top of the curve now, so let's wait for the crash. There is no crash. Sorry.
Managing policy and configurations
Let's say you have all these users onboard it, everyone's authenticating, all's well. In this situation, who here works for an organization where everything is going as well as it is onstage here? Yeah? Anybody? We're hiring at Cruise Automation, so be sure to visit our website.
Back to the topic. Managing policy and configurations, it's pretty important. What we do from a policy-and-manager perspective, we build out a physical representation, in a file structure type way, of what the Vault configuration looks like. If you haven't heard onstage at the keynote, infrastructure as code is everything. We thought, "Why would we configure an application like Vault, and not use configuration to define what Vault looks like?" If we did everything manually, and it's not sustainable?
We wrote a custom app that traverses a tree, a file-system tree, that's similar to what the Vault API looks like. You define those values there. This does two things. A) it gives you a way to track the changes. B) it gives security operators a more thorough understanding of the internals, and path links, and Vault, and API, the schema.
Maybe you've used a higher-level abstraction command line the whole time with Vault and you weren't quite sure of the path at which things were stored and retrieved. When someone comes to ask you a hard question, "Oh, I'm implementing my own Vault client, what's the path to do XYZ?", it's a double positive: A) You keep track of your config and B) you become more familiar with the internals.
What would Vault be without proper end-to-end testing? Say your company is gonna build a bunch of robots that will also talk to Vault. In the process, you check some basic health metric, you're good to go. Typically, once the adoption increases on your Vault cluster, it's gonna become a really critical path of all these applications. That begins to worry some people, but mainly the people who have to maintain that cluster. And it makes some people happy. The people who are in charge of compliance and whatnot. Maybe those things crisscross. I'm getting to that problem again.
I highly recommend identifying what are the four or five common things that your application or your infrastructure's going to do that will contact Vault, and identify a way to do that. Let's say your typical workflow is to authenticate, write, read, and delete the secret. You want to build some process or testing framework that exists external to Vault, and goes and does this thing all the time. When it's done, it asks you: Was it successful in each step? And what was the timeline for each of those requests?
This is extremely beneficial for two things. If there's a trending problem that's starting to occur, you can identify that before someone has to come into your Slack channel and make security look even worse. Also, you can identify usage over time. Those are two things you can do.
Policy at scale
Next we'll talk about how we deal with policy at scale. Let's say you have a group of application servers. Typically the standard is: Application A comes online, you write a policy for Application A. Application A is spread across 500 servers, they all utilize the same permissions path, and they're good to go. Give 'em the policy, it checks it as normal. In this case, we're spinning up hundreds of robots a day. What I would be required to do as a security operator, is say: Okay, I need to either build a webhook or someone puts a ticket in, and I'm on vacation. Now robots 700-2,000 don't get created, 'cause usually it's this process of, I have to run my
create _policy.py script that generates some number, and then goes through and creates all these policies.
If you go back to the very first thing of minimal friction, this is the opposite, it's the maximum. As these new things roll off the assembly line, these robots, and you're required to do this manual thing to scale this policy out, it's not gonna work well. Maybe for the first couple hundred, 'cause you're really good at knowing when these things come in, and you can identify, "Hey, we're gonna build the new fleet of robots, so I'm gonna make vehicle 400-500 today, so I'll know."
Sometimes people say, "Well, offload, and I'll just make all 1 million policies today." But then you have 1 million policy objects to keep track of. Anything could change in those 1 million, and it's a much harder thing to hold onto and analyze.
As Vault Enterprise customers, we use real-time Sentinel analysis of a request to minimize one policy. One policy for many things. Before, it was a one-to-one in a many type aspect, and now it becomes one-to-many in reverse order. Now this robot can make a request. Sentinel will analyze that request, and says, "Let me look at some attributes in your request. Does it match up with the request you're trying to make? Cool. All is good. You're good to go."
Now where your coworker who wrote
create_policy.py, it's obsolete, 'cause you have this new cool, automated thing that takes care of it. Are there some problems with this? Most definitely. My tagline is always: Not perfect, but better. Who would want to create all those policies, then you go on vacation, and you didn't create all the policies, and you're in trouble. Or you want to be on vacation and be like, "Oh yeah, I created this awesome Sentinel policy, and now as we scale, I don't have to do anything, and I look awesome." That's what I'm here to help you do.
Another thing we implement is SSH at scale. What that looks like is, there are two ways the typical SSH paradigm works, whether this is your SSH into a robot, or a common fleet of systems that expand and contract. You have this person down here on the computer, an operator or an administrator, and they need to access any number of systems. The typical paradigm is, you have a bastion host that funnels the access, and it goes out to X number of hosts in your environment.
A pretty common practice is, "I have a user name and password," or "I have an SSH key that someone has to disperse to all these 1 million nodes. And I have a push notification for a two-factor provider." That's cool, but the problem with that is, especially at scale, as users are added and removed, you have to manage all that external data in this host of clusters. If it's a virtual machine, or even a robot out in the world, let's say we're not building a driving car, we're building a drone or something like that. You're telling me that every time I have to change access to that, I have to bring the fleet in and update someone's public key. It's not sustainable.
A more sustainable model is what I call the SSH EasyPass. For anyone who's ever been to Cedar Point, the world's greatest amusement park, on the East Coast, the concept of the EasyPass is: You pay a little bit extra, and you can cut in front of everybody. Kind of similar here, but you pay extra from the perspective of, you've invested the users into adopting this Vault workflow. In the morning, when they authenticate, which they're used to, they have this token that can let them do things like SSH now.
They have these built-in security primitives and they don't even know it, because this one thing happened early on. Keep thinking about how to tie those things together. I don't have to dig into this further, because there is an SSH-at-scale talk, maybe tomorrow. If you're interested in SSH at scale with Vault, you should go to the other talk, and not listen to me. They'll be much more insightful than that last slide.
There are two ways to think about certificate issuance. There's the way in which you use a public-certificate authority. Who here has the one person who's a cert person? Like "I need a cert for this new thing we're spinning up." You put a ticket into the cert guy and now you have to figure out, "Okay, I don't remember the OpenSSL command to do the proper certificate request." So you mess around with that. You send it to the cert guy, he gets it back to you whenever, that's cool. Now with the PKI backend, we can either leverage that using a third-party CA and doing some magic, that's expensive. Or you can also build your own certificate authority, which maybe you've been afraid to do so before and now you can implement this.
This is beneficial for both humans and robots. For robots it's easy, and for humans, it's also easy but maybe not as easy. From the robot perspective, or the system perspective, we've already authenticated to Vault, we have this token, and now all we have to do is make a request that says, "Give me a certificate." We've already had that trust chain implemented and that certificate is issued. Same with the user. You know what, don't worry about
openssl-req command, you don't need to worry about that. Make this simple API call and now you can have a certificate. You don't even have to make the CSR. Just say "Give me the name, and I'll give you something back."
We're going back to minimal friction. It's a bonus that it works for people, but it's great that it works for robots because you don't have to worry about having to do all this manual work on a robot as well.
Cryptography as a service
Another important thing that we implement is cryptography as a service. Armon talked about the transit backend today, which is extremely useful. You all have it now. There are several private keys lingering in some repository and you're going to change them one day. I know you will. You check those out and maybe they're password protected and the password is stored in a password manager, and they get the password from the password manager and they run the command line, they decrypt it, they do their thing.
And that's business as usual. I'm not here to tell you all about the transit backend. I'm here to tell you about, think of the things you can do to make that transition process easier. You've identified all these keys in your environment that these systems and people have implemented right now. Step 1 is, "Hey, maybe I'll just use those existing keys for awhile, but bring them into the environment." So get ready to build. So you probably have to build a tool to do that.
It's kind of interesting as a phase 1, a compromise. Rather than say "We scanned the repo and we want all the keys gone. We want them all gone now." It's typically not sustainable and it'll probably give you a bad reputation. So we say, "Alright, as a peace brokering of a phase 1, we'll take all the existing keys and we'll move into this new abstract primitive store." Okay. And they start to use it and become comfortable and maybe a few weeks later when they're having such a nice time and such a nice experience using this, using Vault, say, "Now we're going to switch the keys." And it's not so difficult because those basic primitives are implemented.
Think about things like that. It may not feel like the best to use the old key, but in the long run it's going to be the most impactful. I'm crossing the themes again. Sorry about that.
Now that we've been successful, we have these robot cars riding around on the street, we have all these platforms, we have all these applications and as time has gone on and we ride that curve, we're connecting them all together. And we realized that we are slowly becoming the global trust broker, whether we realized it or not. You or I or anybody in this room may integrate platform provider A, B or C with Vault. Application A is with Vault, and now this other thing which is a robot that's rolling around several cities, has now integrated with Vault.
Now you can leverage that platform to share information, which you couldn't before. And maybe on your radar was "I need to build a strategy to let all these applications authenticate and authorize to each other." You have this primitive you should already be advertising across the business to use, and now you can use that. Now my Cloud Provider A needs a secret for a Cloud Provider B because we haven't migrated that data over. Guess what? In the past you'd have to generate some keys, copy them over or copy them into something, some really hard process, and now we can just leverage the existing identity priority we have. In this case is, up one level of Vault and say, "Cool, you give me that instead. I won't worry about it."
Now you are a global authentication broker, which is scary, it's kind of like all your eggs in one basket. Not perfect, but better. In addition, you can do things like cross-service talk. This is a way we use solve a problem of, "We have this rolling computer and we need to talk to multiple services and these multiple services are also using Vault." A little redundant in the last slide, but it clicked that, "There's always been this policy and stuff around how this Service A authenticates to this central place, and now this new thing does, why don't we leverage that?" Build a few little complexities on top of it. Now we have this integrated crosstalk that's both secure and manageable.
Authentication is a form of "I give you a piece of information and you match a string." You compare and contrast. If equal, then authenticate it. Typically, in the real world, this is easy for Cloud Provider A. We use their underlying identity-management store and we leverage them and say, "Hey, X is this, can you prove it's that?" Typically we're gonna trust them to make that decision and give us back the result. We can do the same with Cloud Provider A, Cloud Provider B, LDAP, etc.
Now we have this thing rolling around. It's a robot. How do we authenticate it? Kind of the same way. You can take these basic primitives about it and ways to improve it. We wrote a Vault plugin. The last 10 slides of this will be dedicated to talking about how awesome Vault plugins are. It's like, write a plugin for it. If you have any plugin writers in here … any plugin authors for Vault? Yeah. So if you haven't, I recommend checking it out because it's really cool.
In this case, we have all this information in random places about a robot. We can categorize that together. Use other basic security primitives. Hope you see the theme here is that we pick things that have been proven and trusted and we leverage those to implement the most basic workflow. We're not here to write a new encryption standard, we're here to use things that exist, implement it, make the workflow easy, frictionless. That's how we authenticate new and custom things.
I have 2.5 minutes left. I'll go through some cool things that we're working on, like our labs type projects. The first one was code named Indy, so it's a Vault client, but for servers. We saw this problem that people had applications or servers and they needed to authenticate to Vault. Typically the user experience is, you have this command line client you download, you have to do some configuration, it's a little bit different, but that's not a great experience for a robot. So we said, what would a robot like to do to authenticate? What the robot would like to do is just run this binary, identify all these custom attributes automatically, and authenticate it.
We worked on this project. It does just that. You give this small binary to the computer and it can authenticate to Vault without having to worry about "Where do I download the Vault client from?" Or "Is it trusted? Do I to have this whole extra process?"
As a security engineer, something as simple as "You need to download a binary to authenticate to the system" becomes difficult because all these extra 50 questions come. I got to download it. It comes over the internet. Now I have to teach end users the workload to validate the binary. If you already have an existing framework to do that, you can build this small application like we did, but it's like a client for servers. And you may say, "But Brian, Vault just released this," and yeah, that's the risk we take. Sometimes we know something better may come along, but there's a gap. We identify it, we build it, we move on. And if something better comes along, we have no problems with scrapping it and replacing it.
Another cool thing we just started working on is Terraform Enterprise Auth at the lowest level. The old workflow was, you had these people with their laptops that have already done this awesome Vault workflow, everyone's got a token. They leverage that token to decrypt stuff and send it to the cloud. If you're moving to Terraform Enterprise, what are you going to do? Have people copy and paste their keys into some some job, and have to worry about it. You had this great security solution. The point I'm trying to make is, you can develop a solution that works well. You add an enhancement that can change the structure of that, and now you have to rethink that. So be prepared to do so. The theme of "I'm going to install Vault and I'm going to store all the secrets and we're going to be awesome forever and I can maybe work like 5 hours a week," that's not a thing.
There's a plugin for that. As I said, my common theme is really identify, so we have an application that has all these subtasks and those tasks are different and need separate sets of secrets. What we do is identify each task. Is there a way for Vault to talk back to that service and trust who it is? We write the plugin, we delegate the trust to someone else, we trust them, we issue the token.
With zero minutes left, just to review the bullet points. Our success story with implementation is: Introduce minimal friction, trust, protect, be successful, and most importantly, be ready to build these custom solutions to make people happy and secure at the same time. Thank you.