Security fundamentals at scale with HashiCorp Vault and Sentinel

In large installations, traditional access control lists (ACLs) can become unwieldy and hard to manage. Ultimately, this can often lead to security vulnerabilities, due to unavoidable errors.

HashiCorp Vault's new Sentinel features make access control far more flexible and easier to manage. Vault now implements:

• multi-tier, defense-in-depth access control
• role-governing policies
• route tokens
• conflict-free composability

As a result, Sentinel's improved experience leads to improved security.

Transcript

Originally, the title of this talk was Vault and the Super Secret Technology. Which, if you read the description of the talk, it makes a lot of sense, right? Someone in marketing changed it, somewhere along the way. So if you came here for Security and Fundamentals at Scale with Vault, I'm really sorry. But the actual talk is Vault and Sentinel.

OK, let's have some real talk here. I am so excited about this. So, so, so excited. When I first saw Sentinel, actually running, I think it was about early May, [00:00:30] I immediately just rushed and did like a proof of concept, getting it integrated into Vault. And it was so nice. I've been wanting to have this kind of flexibility for so long in Vault, and ever since I did that I've just been waiting for four months, telling people "Yeah, things are coming. Yes, this is better, this will be better soon, things are coming." And sort pushing people off. I can keep a secret, that's why I work on Vault, but it's been really hard to hold onto this for such a long time.

So, why am I so excited about this? It's because [00:01:00] of Vault's ACLs. So let's get ACL-imated with Vault's ACL system. Okay, so Vault's ACL are default deny. They give you privileges in the form of capabilities, so they're very CRUDY-y mostly. There is Create, Updated, Delete, List, Sudo and Explicit Deny and some other, optional, restrictions. And they're in JSON-Compatible HCL. Basically there's a path, [inaudible 00:01:26] you sign it, that's a very, very basic example, you want secret/foo capabilities, [00:01:30] create, read, list. So whoever is there can create, can read, can list, that stuff.

And these are merged, so looking at these two path statements, the overall capability grant is a create, read, update, delete and list. Sorry about that, I've been talking for about three days and I did a full day training in my voice is pretty shot at this point so I apologize. I'll be sipping water. So, we are going to come back to the fact they merge because this is something that actually ends up being pretty difficult.

So the current policy model is pretty simple [00:02:00] you have one or more ACL path statements, they're placed into a policy, you take one or more policies, you attach them to a token or identity info, which I'll talk a little bit more about later. And at request time the policies are combined, so we take the policies as they exist at request time in their current state, we don't do any pre-done evaluation, we look at it at that point, and we combine it into this ACL object which is used Check Access.

Here's the problem. The larger the installation of vault, the more likely that some characteristics [00:02:30] of ACL's become like serious pain points. So one is HCL and JSON is the language, and I'll go into more detail on these. Next is the path-based workflow and third is merging. So ACL's are HCL, HCL is JSON compatible, you can use either the ACL syntax or JSON, both are acceptable. JSON is really simple, it's really compatible, which is one of the really nice things about it, I really love JSON. But, it's really a data interchange configuration language and it's not very expressive, right?

So you can't... there are plenty [00:03:00] of times probably, if you've worked with JSON, where you're saying, I really want to do this kind of complex thing and I have to figure out what is the right set of arrays, maps and other things to shoehorn into what JSON does. And you can't do introspection, I mean, you can do this with a lot of code outside of it, but there's nothing native to do introspection, interpolation, logic et cetera. And there is hash curve interpellation language, which [inaudible 00:03:23] that came along much later, it's also compared Sentinel, quite limited.

So we've added more things, a more, [00:03:30] we've added more into ACL's. So, over time, we added things like parameters, denied parameters, which themselves are kind of limited again because of what we can do within this JSON scope. So not as flexible as they could be. So, we've sort of kind of extended the ACL, capabilities over time but they're not as good as we would like.

They're specified as grants and restrictions on paths. So, originally, vault, if you listen to keynote yesterday then they said that was really kind of scratching an internal [00:04:00] itch, we needed a way to sort things securely inside Hashi Corp. And that was layered on top of console and so the actual, the generic or K/V secret that you get if you start a dev server, you start a vault server for the first time, the actual code is in a file called logical pass through. It's called pass through because it literally is just passing through what you give it, straight into console but in encrypted form.

And so the origin of the path based workflow, which is really nice in a lot of ways because it lets you organize things you via these HP paths and that [00:04:30] actually has a lot of benefits, but it was this encrypted K/V pass through. So it's not an unreasonable choice for protecting API's because API's are defined by paths as well as the request objects, and headers, and things you give to it which can modify behavior. So it's worked pretty well overall and if you've used the vault and thought, man a lot of this API is just super weird, a lot of it comes down to the ACL system. We want to be able to say, if this is something that people are probably going to want to restrict, then they they need to be able to write an ACL for it which means they need to be able to [00:05:00] identify that is a path or a prefix of new.

So here's the thing. Path is a single component of request, right? There's lots of... more things, there's IP addresses, there's headers, there's request bodies, there's even things like when they come in, what time is it? All sorts of things about a request that we can't really do anything with if you're just looking at a path. So basically, the only way you can deal with this right now involves... you can say, I can have a path [00:05:30] of star and you apply that to every single token that you have, but every time you want to do something that isn't just looking at a specific path you have to use a path of star and that just adds up to like lots and lots of stuff.

So, in theory, ACL's are composable because paths are merged. So I mentioned before, I showed that example of the two statements and the total set of capabilities was create, read, update, delete, and list because the paths are merged together. But, what we found is that the more kinds of things we want to put into ACL's, the harder it is to [00:06:00] have correct merge logic. There's potential for conflicts, like, what do we do if this policy says that this is allowed and that one says that it's not allowed? [inaudible 00:06:09] default to not allowed.

But it becomes much more complicated and when you want to have like arbitrarily complex structures in JSON and in ACL's, that becomes really difficult. And we found this out when we were doing a lot of parameters and denied parameters where we can support some things but other things are kind of... we don't support maps. But there are some things involved that it will take [00:06:30] in a map as request data.

So we do have some enhancement ideas for ACL's and we sort of brainstormed over time but these properties continue make these enhancements difficult. Any enhancement that we make, we have to know how it's going to merge together, and we have to come up with what does the syntax look like, how do you express it. And so it's much more difficult to do what we want.

All right, and then along comes Sentinel. Sentinel, Sentinel, [00:07:00] Sentinel, Sentinel... I told you I was really excited about this right? So the new Vault Policy Model. So it's a multi-tiered, defense in depth approach access control. We like defense-in-depth in vault, that's one of the things we do, we espouse it, we tell everyone to do it. So we have ACL's, we still have them, they're the normal kind of course-grained ACL's that you know and love.

Those aren't going anywhere. We have role governing policies. So these are Sentinel policies that are attached to tokens and identity information. And [00:07:30] then we have end point governing policies, so these are policies that are Sentinel, that are attached to prefixes within vault, including most unauthenticated paths. Just means you can start having... start talking about and describing in these policies things that you want to do on paths that aren't even being hit up with a token.

So it looks kind of like this. So we have things that are request based so these are the ACL's and the role governing policies. Then we have end point based ones, and here, what this is showing, and I'll [00:08:00] talk about this a bit more, is you can have multiple different end point governing policies that all get evaluated in the context of a single request. And if you have seen what consul's doing, consul's doing it initially at least, doing in line Sentinel policies. And so why aren't we doing that? It's because of path merging. So actually I have a quick question, because this is a something that we debate often within the Vault team. So who here uses vault? Okay.

Keep our hands up, put them down... okay. Put your hand down if you [00:08:30] if you don't use path merging. So, basically, sorry, I'm making this more complicated. I three[inaudible 00:08:37] three and then somewhere along the way just forgot what the second question was going to be. So basically just if you write vault policies and you merge paths together as you write vault policies, put your hand up, let me see. Okay, yeah that maps out to about what I expect. So we have this thing that we can't get rid of because people do use it, but at the same time not many people use it and it makes our lives much more complicated.

So one of the ways, which makes it complicated [00:09:00] is, if we want to think about how would we merge Sentinel policies, what's the right thing to do, is it all the rules, is it to end the rules. And whenever we try to think through, what about if a person wanted to do this in Sentinel, how would you describe them ACL's in a way that would merge the Sentinel policies together in the way that you always want. So it's very uncertain and when you're in security you don't like uncertainty. You know, I think in some ways vault is like the least magical of the Hash Corp products and that's very much on purpose, so we don't do it [00:09:30] like that.

Okay, so we had defense-in-depth. So basically here's the policy in full now. So, if the request is an unauthenticated we only do EGP's because we don't have a token and so skip to step four. Step Two, if ACL's are compiled and evaluated as normal you still have to have an ACL that grants you capabilities so you have to be able to go through that sort of core screen check and say yes, you are allowed to access this path with these capabilities. And if you do any sort of [inaudible 00:09:57] stuff that still takes effect. And [00:10:00] that's even it even if you just say you know what, I only want to use Sentinel, then you still have to have a single ACL policy that has star and gives every single capability to everybody and I don't recommend it, but you can do it.

Okay, step three. Role governing policies attached to took token or identity information are evaluated and any failure fills the request. So we use normal Sentinel evaluation, advisory soft mandatory, hard mandatory. So if it's soft mandatory or hard mandatory filled and overrides [00:10:30] are not specified for soft, then it fails at that point. Once we've got to that point we go to EGP's and we say, all right, look at the prefix involved. So you know if it's secret foo and you have one at secret foo, secret and star, as we saw before, that we actually look at each of those prefixes and each of those paths.

So EGP is from, basically, an exact match in any glob that's all the way up the tree, all get evaluated. And so that's actually nice because you can sort of layer [00:11:00] how you're doing these things, you can say, you know what, I have this sort of global policy that I want to define, I'm going to put it star. I want every single request to be beholden to this. Then under secret that I have another set of policies and then I'm going to give some other... a minor set of extra restrictions on this one set of keys, or two set of keys or something. And you just put those on at the various levels and they all get evaluated.

So unlike ACL's, as I just said, each [00:11:30] piece uses prefix walk, so in this example all EGP's at all three paths will be evaluated if it's secret/foo. So we have the exact match and then we have, going up the tree, it's done internally as a radix tree, going up the tree anywhere, anything that's a glob matches. Some paths don't, and won't support EGP's so, as an example, generate route is already your path of last resort right? You generate route if you revoked all tokens involved accidentally, you take a quorum of [00:12:00] unsealed key holders all together and you generate a route token, you can get back in business. If you had EGP's on that, that said you can never do this, then that would be useless so a generate route won't support it.

And speaking, which, this is an FYI, just like with ACL's, route tokens skip RGP's and EGP's, you should never be having live route tokens for anything other than exact things that you're doing right then. You should always have multiple eyes on route tokens. And when I give trainings and I talk about route tokens and kind of managing them, I [00:12:30] always say treat route tokens like they're the ninja in a movie where like the baddies are surrounding the Ninja and do all the things that the baddies don't do. So, like the baddies go away and leave the token alone, or the ninja alone, and they'll have one person there that's easily overcome right? You just have multiple eyes on it, revoke it as soon as possible et cetera.

So, while it's going Vault won't check properties of the request and, or user into Sentinel and various named spaces. So right now, what exists, are requests, so that's information of the request itself, what is the path, [00:13:00] what is the operation type. Is it an update, is it read, What was the raw data that was brought in. Other information about it, was a policy of read requested, is it unauthenticated and so on.

We have tokens, that's information about the token being used, creation time, what policies are attached, What's the current TTL, Things like that. Identity, so this is identity entities and related data and MFA, so access information about MFA validations and we're going to talk about those things a [00:13:30] little bit more.

And if you're doing an unauthenticated item, you obviously won't have a token right? There won't be anything there because we don't have a token attached to the request. So some of those things are only valid for RGP's and some that will be in both RGP's and EGP's. Okay, so enough yammering about all this stuff. Let's show it to you.

So these are all real Sentinel policies that work to do things. So this is [00:14:00] an example... this a very, very basic example of allowing only specific entities or groups. This is using false identities system, which debuted in 08 in Enterprise and is coming to Open Source in 09. So this is showing an example of, if an identity entity name is "Jeff" for my ideas whatever, or one of the groups that I name is "Sysops" to dot id, then I am allowed right? That passes. We do recommend, if you're... whenever you're using like a name vs id, use id whenever possible. [00:14:30] Id does not change, it's not reused whereas name, someone can delete an entity out, create a new entity with the same name and your policy will still match. So always use id's if you think that might be possible

All right, this is a nicer example. So this is multi-factor authentication and CIDR check on login. So I'm going to start the bottom. So, insert at the bottom, this thing here. When request path, so [00:15:00] main equals rule, when request path is "auth/ldap/login". So Sentinel policies must pass right? But it creates an issue because Vault lives in a default-deny world, that's what we've always done, that's what we like. oS how to ensure policies pass without saying true and finding restrictions? Because, at a base level, you have to say true and then you just restrict what's there.

So we have this precondition mechanism that describes the conditions under which the rules should be evaluated so the defaults are always so there's no precondition that it will get evaluated. And then [inaudible 00:15:30] success [00:15:30] criteria. So it's actually not that different from ACL's right? ACL's you have a path, that's your precondition and then you talk about the capabilities you should get on that path. And Sentinel it can actually be arbitrarily complex sequence, so you'll see examples later where that precondition is actually under the rule and the evaluation of that rule, true or false, says whether or not the precondition is valuated.

And what if no preconditions match? Its still default-deny because you have had to have been granted capabilities on path via [00:16:00] ACL's. So, if no Sentinel policies match at all, you still have to have that core screened ACL permission. Let's go back to this. So, the next part I'm going to highlight is this "auth/ldap/login". So this is an end point governing policy on a login path. So full featured, request-based access control, and it's little bit contrived, normally you put the EGP at the endpoint itself, this is just for this example I wanted to show something here. Normally you'd say, I'm going to just put it on that path instead [00:16:30] of evaluating it everywhere, just for performance reasons.

So the next bit is this, so on the bottom rule you can see ping valid insider checks, I'm looking at the ping valid rule now, you can see mfa.methods.ping.valid. So, the identity-based MFA system that we debuted didn't support login paths, now you know why. It was waiting on Sentinel and it was waiting for us to get this capability to do these endpoint based paths, sorry, endpoint based policies.

[00:17:00] So now we have that we can do some really cool things with it. So in many cases no preconfiguration is necessary for users and the reason is that what we've done is we've built this mechanism where we go to an auth backend and we say, alright, if given this request data, if this was successful what would the unique user name be? And then we take that back and we go to the MFA check and we say, we don't have access to anything outside of just this name but in many cases that name will match to some name that's also in your MFA providers. So it'll be... [00:17:30] it might be that name at example dot com right? And so, in many cases, you don't even need to preconfigure you just configure the MFA method, you put a Sentinal policy on the path and you can get automatic MFA for logins without any other preconfiguration.

For those that require any sort of massaging of that or using metadata from a need to do that, then that's still possible too, you just have to go in and hit up some endpoints and preconfigure those identities. MFA runs for login. [00:18:00] So the reason that's nice is, if MFA fails, we don't need to revoke or clean up anything, we don't have to revoke a token, we don't have to potentially hit up a third party API to do anything there. And it's harder to brute force, right? You can't get to the point where you're trying to brute force a password if you are failing the MFA check in the first place. So we do them a favor on login.

So this is the whole thing together, you can see we're doing this check and [inaudible 00:18:28] did something not valid. Simply [00:18:30] running that check right there actually triggers validation and we memorize the results. So, if you have multiple policies that are doing a check on ping, ping by the way is the just the name of the method it is one of the ping identities one of the MFA supported methods but there it's just the name. So basically, when that's checked, we normalize it and so if at any other time during that request, if we get a validation check for that MFA method, then we actually just use that. We use that result.

And that also applies... you can do you can trigger MFA checks from ACL's and those are [00:19:00] shared between the ACL's and Sentinel policies. And last thing there that I didn't get to before, is just that we also can have this CIDR check. So there's a sockaddr as one of the standard imports and we can say, if the remote address is within... it must be within this private IP range, if we say, nobody should be trying to log into vault if they're not coming from our corporate network.

Okay, so the next thing we're going to talk about is a break glass case. So suppose that you just found out that a bunch of tokens have leaked out, you don't know which ones [00:19:30] have been compromised, you don't know where they might have gone, you don't know what they might have been used for, but you want to make sure that they can't be used right now and give yourself some time to perform forensics. So with the existing system, you can say, I'm going to deny all requests except group, so you just do like Star and just denial. And you can revoke all tokens.

Two problems. One is that, if you're denying all requests then it means that nobody else can actually get a new, non-compromised token and actually do things [00:20:00] while that policy's in effect if that's applied to all of your tokens. Calls to revoke all tokens, but if you find out that there is actually only just one token that leaked, and you can revoke that one token, now you have a whole bunch of tokens you've revoked, they're no longer valid, that might actually have leases attach them that you want to keep. And that can take a while. When you revoke a token it revokes all seeded leases and so all of those leases have to potentially go up to third party databases or third party providers and say, destroy this user, destroy [00:20:30] this user, destroy this user. So that can take a while if you're doing hundreds or thousands of tokens.

So, Sentinel provides a different way forward and it's really simple, really nice. The following would be an EGP that you just set on star. Really, really simple. So the rule says, when the request is not an unauthenticated, the token creation time has to be after that point in time at which you discovered the compromise. [00:21:00] Really simple. Some of the syntax there, there are a couple of features coming in Sentinel that will make that a little bit easier. Right now this is literally just saying, in the time package, when you do a load it creates, what we call, a time space and we're getting the Unix value in checking out against the Unix value of that fixed string.

And because of that request is unauthenticated, then it means that somebody else is coming in and saying, you know what, we want to block all these previous tokens because we don't know if they've been disclosed. Someone could come in, authenticate fresh their credentials, get a new vault token [00:21:30] and it's not going to be subject to these restrictions. So you've denied all previously existing tokens while still allowing people to get new tokens continue on with their day.

Okay, so my final example here is delegating EGP policy management. So we get asked about this all the time and we want to delegate policies. We need to delegate policies. And the main reason that it hasn't occurred before now is because, it's really, [00:22:00] really hard to figure out how to do that with the ACL specification that we have. It's pretty much impossible, any sort of flexibility and expressivity that you may want in terms of the things that, that policy is allowed to define, the powers of it's allowed to access, are things that would either have to be worked into like totally new API that take totally different sets of parameters, just write the same kind of policy. Or have incredibly complex syntax and structures. [00:22:30] Just really super difficult to express with JSON and HTL.

So this policy is long, I'm going to show it in two parts, talk through. So this first part is using the preconditions to narrow the scope, to just policy writings, So we're saying, we don't even want to check this if it's not on a path where policy is being written. So in this case, it's not necessarily contrivance, you could do this as an EGP that you say, I want to set it on this path but you could also set it on tokens. So if you want to say, [00:23:00] this is actually an RGP and I'm just going to put on the tokens of the people that are authorized to do this, then you can set it there. And the advantage of doing it that way as you're not forcing every request to run through the Sentinel check, you're only forcing it to run through when it's these particular people you want to delegate to.

Alright, so the first part starts up with Main. And you can see here that the precondition is a rule itself. So all that has happen is that precondition just has to turn into a bullion, true or false. So [00:23:30] we look at the request operation, see if it's create or update, and then we look for a prefix, a request path that says policy EGP. So one that matches, so when someone's trying to write in EGP, then the next that matches... the next thing we do is we look for a prefix on the request path of this policy, EGP Team A. So here, in the example, this is a Team A that I want to give access to. I want to say, you can write any policy that you want as long as it starts with team a dash so they know who did it.

[00:24:00] Before going on, actually can you go back one slide? So I'm not sure... don't remember if I talk about this later but, this is using a bunch of... has a prefix, strings that has prefix. And those strings package and it has some containers and it has among other things. Sentinel has matches as an operator and matches is cool. It's a Reg X. if you're in security you don't like Reg X's they're very easy to get wrong, Reg X engines have been the source [00:24:30] of much pain, security wise over time. They can be very hard to reason about when they get very big, so when all you need to do something simple then say, here's an explicit, I want check for prefix, I don't want to just do you like Reg X that I think does what I think it does and maybe it doesn't.

Okay, so the second part is a function that verifies the paths set. So this is a little bit longer so I'll go through it. So, [00:25:00] the first part there is saying, I'm going to check that there is request data, and I'm going to check that the request data has a key name paths, because in the in this policy EGP, then paths is where you set the paths that it should apply to. So I'm going to check the paths is actually populated, if not, return false. And for each path verify that it's in the allowed list so we're splitting the request paths, we accept both comma, separated values and JSON arrays. So basically depending on if you doing [00:25:30] it on the command line or over the API, you can do both.

So first we make sure that it's split so we can loop through, and we sanitize and say get rid of any leading slash and then we just do this real basic thing that says, okay, if it doesn't have this prefix dev-kv or doesn't have that prefix prod-kv/teama, it's disallowed. So what we've done here is we've said, and by the way this is a little bit of contrived example to, it does work but of course don't run dev and prod on the same server, just check and run right? [00:26:00] So we had a, let's see, there was a customer that did run dev, Q and prod on the same server and ran into problems. And I can't say more that but they ran into big problems. When something in dev went haywire.

So don't do that. And so here was saying, okay, for every path that comes in, the path that, that EGP is applied to has to has to start with one of those two things. So you can imagine, here's a dev-kv, [00:26:30] here's prod-dv [inaudible 00:26:31] of the kv backend, which is what we renamed generic to. And then, within those two things, they can delegate policy under Team A. So at that point anyone on Team A that you give this policy to, can go in and say, you know what, I want to control... I'm being put in charge of dev-kv/teama/sysops and dev-tv/teama/devs. And in fact there's really no reason why the person in charge of dev-kv/teama can't then write an EGP that applies one level down, so that other [00:27:00] people then have, based on like say, identity group information, can then carve it out even further down along the way.

So this is really nice. This is something people have been asking for, for a long time and it's just waiting for us to get to a point where we had this nice expressivity in policy thing with which to do this. So I think I just talked through all that anyway, there it is, but... so trying to do... so this talk about strings that has prefix but doing this in JSON, we ran into this doing a lot [00:27:30] of parameters and non-parameters, where we were trying to think, okay, within a lot of parameters and I non-parameters, if you don't want to allow Reg X's do we allow globs, do we allow them at the end, the beginning.

What happens if someone wants a literal glob, do we do like two globs together, this literal glob. And it became a very sad to try to reason about. And it's quite difficult. So Reg X is going to work but often not cerebral for the reasons I laid out. So we have this nice package that does it and those [00:28:00] packages are going to grow over time. So that's the whole thing again, this data match function.4

Alright, so last thing here. ACL's, Sentinel or both? So Sentinel is slower than ACL's, milliseconds not microseconds. The Vault team we were sort of surprised to hear that one of these people that were organizations using it, were saying that they have these really tight time requirements for vault requests and vault was meeting them but they wanted to do some [00:28:30] really fancy stuff and they were concerned that vault might not be able to meet it. And we said what are your timing requirements, thinking it be like order of milliseconds, they said, oh I mean right now it's handling things that you know about twenty microseconds. Really, wait, what? So we didn't actually know to do that.

But Sentinel, we do know is slower. I mean it's a whole language that has to do a lot of interpellation of the various kinds of parameters, various kinds of data, especially when considering that you can call in from there and then call into other functions involved. So it's slower. So if you need [00:29:00] super, super fast time, critical operations you should stick to ACL's, most cases involved this won't be important. But at the same time if there are time critical operations involved, it's really only one concern so identity look ups, slow identity is super useful but if you're doing super high [inaudible 00:29:18] transit stuff, encryption is service, you don't really need it and probably shouldn't use it.

Sentinel benchmark it for your each case. As always it depends on what you're trying to do, [00:29:30] what your time requirements are. And the final thing is, it uses as few ACL statements possible, even if you're just limited to ACL's, as I said before, it has to merge things together, has to build up this ACL object. So the more paths you have, the more things it has to this object and then evaluate against. So generally speaking, time critical operations, the more simply you can keep it, the better. But if you don't have super tight timing, Sentinel's super, super, super nice

So, that's it. [00:30:00] I'll be round for questions. Like I said before, I've been the lead on vault for a little over two years now since just after 02. This certainly the top three most exciting things that I've ever worked on, involved for sure. It might even be number one, I'm just so thrilled about this and the ways that we can describe policies now and things we can do with it so. I hope everyone enjoys it.