Protecting Workflows and Secrets

Most major data breaches in the last 10 years have exploited vulnerabilities in applications and data stores to circumvent cryptographic protections. Tokenization has been a common way to provide that additional layer of security, but its implementation is not without challenges. This talk will provide an overview of Format Preserving Encryption (or FPE), a form of cryptography meant to address common challenges with tokenization, and will look at common use cases.

This session also includes a Q&A session with Vault engineer Jeff Mitchell as well as the speaker, Andy Manoske. You can also check out Andy's talk from earlier this year Post-Quantum Cryptography and HashiCorp Vault.

Transcript

Andy Manoske:

Hello. My name is Andy Manoske, and I'm the Vault principal product manager at HashiCorp. We're going to talk about protecting workflows and secrets.

I joined HashiCorp in 2016, at a stage when we were starting to begin to sell Vault Enterprise.

Part of the reason why I did that is that I've spent most of my career in and around encryption and cryptography and how we protect secrets. Here at HashiCorp, as a principal product manager, I focus on how we build products and think about platform security and cryptography throughout the rest of the HashiCorp secure products group.

I formerly led product security encryption at NetApp and worked on the research side at AlienVault, where I supported the lab team and led development, something called Open Threat Exchange, which is a popular way that security researchers and teams try to collaborate over open-source information to think about how we protect sensitive information as well as interrogate cyber-attacks.

The product that I worked on at Open Threat Exchange, or OTX, was one of the tools that was used to attribute North Korea in the state-sponsored cyber-attacks against Sony Pictures in 2014.

That highlights the fact that over the course of my career, I've been trying to focus on how we protect secrets and, in particular, how we make it easier to protect secrets.

There's this quote that always strikes me from one of our Vault Enterprise users: In her opinion, the best security tools are the ones that you actually use.

This rings true since, in 2020, we have a number of different ways to protect information, using very complex mathematics, using very complex tools that stop adversaries as they're in the process of attacking a system.

But there seems to always be this trade-off that hasn't changed since the very beginning of information security cryptography: As we apply security tools and security protections, there's a cost to usability and to workflow.

This is something that we're really keen to focus on with Vault, because as we introduce new features into Vault, they're really focused on protecting secrets and defending information.

But we need to do so in a way that doesn't hinder the ability of practitioners, developers, and really anyone that's a constituent Vault user, whether they know it or not, to be able to conduct the workflows that they are engaged in.

Let's step back for a moment and talk about how we and the Vault team think about the world.

The Complexity of the World Is a Challenge

We take an adversary modeling approach to thinking about how we build Vault and Vault Enterprise. What I mean by that is that we try to step back from it and think, What is going on in the world? And in particular, what is going on to motivate adversaries?

Unfortunately, the world is in a very complex situation, and not just because of what's going on from COVID-19, but certainly that does not help things. Even without the pandemic, there is a significant amount of complexity that's going on in enterprise IT and enterprise infrastructure.

That complexity is challenging, because if you've spent time in information security, you know that complexity is the enemy of security. The more complex the system is, the more likely that someone can circumvent that system or exploit aspects of that system to be able to launch complex cyber-attacks that can circumvent cryptography.

This year, the number of passwords that are used by machines and humans is expected to exceed 300 billion.

That highlights the fact that security infrastructures in 2020 are very complicated. Especially when you look at where the enterprise is, and we define "the enterprise" in HashiCorp as the Global 2,000.

These kinds of users and environments typically have multiple or even redundant infrastructures of security, redundant firewalls, redundant types of defenses. And on one end, this makes sense. There's a model that we frequently talk about in security called "defense in depth": Let's think about your infrastructure like it was a castle.

I have different concentric walls that are built in such a way that, if someone breaches the first wall, they can get in, but they're faced with another wall that they have to breach. This makes a lot of sense, and defense in depth is, generally speaking, a good idea.

But the complexity, and especially in terms of creating lots of different types of credentials or other secrets that I need to protect, is a significant challenge.

Adversaries Are Sophisticated and Out to Get You

The second motivating factor that complicates things is adversaries. Thinking that I am facing a bunch of kids that are breaking into me for fun, sort of like the Hackers movie-style approach to cybersecurity, does not really apply here.

When we look at The Global 2,000, 71% of breaches were financially motivated, and 25% were motivated ultimately by espionage.

So the modern enterprise is not facing Dade "Zero Cool" Murphy and his friends. They're facing organized crime. They're facing state-sponsored adversaries that are ultimately there to commit espionage on privileged information. They're facing phishing rings, mercenary hackers hired by governments, etc.

You're not facing simple adversaries; you're facing complex adversaries. And these advanced adversaries are ultimately motivated by the need to conduct very damaging attacks.

Complexity Plus Adversaries Equals Breaches

Ultimately, the consequences of these damaging attacks are data breaches. We've all heard about data breaches, and especially within the European Union, there are consequences for data breaches in the form of the GDPR.

But when we break down and analyze the statistics of what happens in most data breaches, especially in the last year, we see some very troubling trends. First, when it comes to 2019, there were around 4,000 data breaches that were at least logged.

There probably was a significant amount more that were not logged, but this is a significant challenge for the Global 2,000. Around 4.1 billion-plus records, at least, were exposed. Remember, with GDPR, every time I expose a record, I'm committing an infraction. And the implications of that infraction, depending upon what your supervisory authority is within the European Union, can be very significant. With 4.1 billion records exposed—this has the potential to cripple some companies.

72% of the targets were enterprises. This makes sense, right? When we look at adversary modeling, the adversaries we're facing are not necessarily kids poking around and blowing up websites for fun.

They are motivated, professional cyber-attackers, hackers who are trying to steal information for espionage or for financial reasons, to sell on the dark web, etc. And of the 4,000 organizations breached last year, 58% had resulting attacks.

So, if your data was breached online, you have a nontrivial possibility that it was consumed and used to commit identity theft, to commit fraud, etc. And this really highlights that cyber-attacks are not a victimless crime. This is a very challenging sort of enterprise that is going to get only worse for the Global 2,000.

Complicating things, as I move toward digital adoption, I incur more complexity and might be forced to confront really advanced adversaries online who are after my information. But I might lack the capabilities to defend myself in this time of COVID-19 because of lockdowns and restrictions on in-person contact, in-person business, etc.

How Do We Deal with This Problem?

In Vault, we typically look at how we protect data in 2 ways.

First, though, what is a secret? You hear this term a lot when it comes to Vault. A secret is anything you don't want to be public. A secret can be, for example, a phone number, a credit card number, a tax ID number, a Social Security number, a FEIN, etc. These are all pieces of information that can be effectively referenceable by a string.

But these are all sensitive things that you don't want to get out. They could be things that you don't want to get out because of regulations like GDPR. They could be things that you don't want to get out because of reputational risk reasons or because it could lead to another person conducting a cyber-attack or anything like that.

It's just something that you don't want to be public. And this is really important, because when we step back and look at why Vault was created, one of the reasons was to make sure that you don't have to face this dilemma of: What secret is secret enough that I have to protect it? What secret is so important that I have to weigh the challenging cost of "How do I protect this information?" versus "How hard is it to deploy a system to protect it?"

With Vault, the goal is for you to never have to make that judgment.

You can just protect it no matter what and never have to deal with the complexity of deploying cryptography, etc., unless you want to. The most basic way to protect any of those secrets is what we call the KV secrets engine, a key-value secrets engine. There's another way to protect secrets, which is to make sure that that secret never leaves Vault.

When I protect something in the KV secrets engine and I use features within Vault like dynamic secrets, this doesn't necessarily require that I, for example, pull it out of Vault. When I leverage the ability to access these secrets, I'm getting a token that allows me to prove that I am who I say I am, that I authenticated myself properly, etc. I can go in and request that secret. And it only gives me access for a very short, bespoke period of time, something we call "dynamic secrets" within Vault.

Keeping Secrets Away from Users

But let's say that I don't want users to access the secret at all. This highlights a lot of trends that we're seeing in security around zero trust, around the ability for a user to ultimately hold on to data that isn't sensitive by nature or something that couldn't be used to privilege-escalate or otherwise compromise the infrastructure.

In Vault we have this thing called been called "orchestrating security workflow." There are secrets engines within Vault, mechanisms for you to be able to use a secrets store within Vault to conduct an operation where the user is not getting access to that secret, but Vault is handling what the user was intending to do.

Here's a good example. Within Vault, I have a KV secrets engine that can store something like an AWS secret key. An AWS secret key could be used by an application to be able to call something within the AWS API. Maybe I want to deploy an EC2 instance.

But maybe I don't want to give the user access to my AWS secret key. What if I'm concerned that there's an adversary waiting there? Something on the user side, to be able to leverage the access to that information. However instantial it may be, to be able to access something within the infrastructure.

Unfortunately, in the last 10 years, we've seen attacks like this. We've seen attacks where advanced malware snuck onto systems. And that malware is able to use keyloggers or other kinds of mechanisms to circumvent the cryptography of sensitive secrets management suites, to be able to see what the user is seeing and steal it.

Again, we're not facing teenagers with hacking kits necessarily. We're facing complex adversaries, with the ability to pull off some of these complex cyber-attacks. So it might be better to allow us to make sure that users don't hold on to any sensitive information at all.

Instead of saying, "Let's give you access to that secret key for my entire infrastructure," how about I give you a credential. A credential that is dynamic, that is created just for this time. That's going to only live for something like 30 seconds. And if someone steals it, it won't give them undue access to the infrastructure.

You can't conduct privilege escalation or use something that's been misconfigured within the access control rights to escalate what you can do as an adversary.

Which is a complex mechanism, but something that we've seen throughout many of the large data breaches in the last 10 years.

We have secrets engines within Vault, like the database secrets engine or the AWS secrets engine, that allow you to create these dynamic credentials. Say I wanted to call something within the AWS API. There's an AWS secrets engine that is specifically created for the purpose of creating IAM roles.

So a user or developer can then do whatever they need to within their infrastructure. And you, as the operator who is managing Vault, can ensure that workflow only is provisioned for a specific user, only is conducted for a specific kind of task.

For example, they can't call things that they shouldn't be able to call because the IAM role doesn't have the rights to be able to do that.

This is the dichotomy that we see in Vault today: either protecting a secret, where I have a secret completely stored within Vault and I'm gating users' just-in-time access to that secret, or I'm orchestrating a workflow.

I don't access that workflow at all. I don't have users accessing any of the secrets at all. It's an opaque process where I get a nonsensitive piece of information, because it dies quickly after that or it's explicitly scoped to a specific role or task, or ideally both.

What Happens to Data That Doesn't Live in Vault

This is one of the challenges that, over the course of the last 5 years or so, we've been trying to contend with. Vault is very good, traditionally, at protecting data that lives within it. But as I said before, the best security tools are the ones that you actually use.

So we think about how to protect external secrets, secrets that are not going to necessarily be stored in Vault, because they're too big or complex, or the workflow can't be evenly orchestrated.

We need to solve that problem.

Let me give you a few examples of external workflows where I'm not going to store everything necessarily within Vault or whose orchestration is too complicated to fit within the context of one specific secrets engine.

Say I want to protect hardware keys for hard drives; this is something that is typically done through full-disk encryption.

"Full-disk encryption," or FDE, is the ability for an enterprise-scale array or even laptops, if you're using BitLocker or something like VeraCrypt, to be able to encode an encryption process when mounting a drive. Then, if you were to remove that drive from that computer and plug it into another computer, it doesn't have the ability to access it.

Enterprise storage arrays are typically bundled so that there are dozens if not hundreds of hard drives. If I want to manage all that information at scale, we're talking about a lot of keys and a lot of encryption.

The challenge with that is managing where all those keys live. I could write a plugin into Vault that does it for one instance of it, but there are dozens of different types of ways to interact with full-disk encryption suites.

I can either interact with them at the software level within the computer, like how BitLocker works. And then each of the enterprise storage vendors, the Dells EMCs of the world, the NetApps, the HPs, etc., all of them have different ways for you to be able to access how I store those keys in the cloud or on-prem or a combination of both.

That's complicated. And that's a workflow where I need to figure out one way to protect all that that doesn't involve creating dozens of different types of secrets engines specifically for each vendor.

Or what if I want to protect credit card numbers in databases? There are a lot of regulations around that, as I'm sure many of the people watching understand. There's PCI DSS, which is a regulation suite that focuses heavily on protecting all the credentials and information that comes out of a credit card.

If I need to protect all of that information with strong cryptography, that's great. But every time I deploy strong cryptography, I need to deploy even stronger key management.

What happens if I have hundreds of credit card numbers? I have a lot of data that's trapped within a credit card. I have the stripe data. I have the CAN information, the account number that's linked to it.

For example, if I have a debit card, that account never could be a bank account. A credit card is a credit account, etc. These kinds of challenges are very complex to deal with. They highlight that explosion of secrets that I need to manage.

As we say within the infosec community, it's a lot of data. We need to protect a lot of information for each credit card.

What happens if I have thousands, hundreds of thousands, millions of credit cards?

I have a combinatorial explosion of data that I need to manage. Am I going to store all of that in Vault? Maybe, maybe not. Maybe there are times I need to have some of that data protected within a database. It's more resident in my application, because when someone swipes their credit card in a retail store, they don't want to wait there for 10 to 15 seconds, just for an initial transaction to occur.

There are a number of similar types of data that need to be protected outside of Vault because they can't incur the impact of storing that data within Vault or because the scale and size of that data can't be stored and involves storage backend mechanisms.

Vault is a very efficient system. One of my favorite examples of this kind of efficiency within Vault is the fact that we have a Vault Enterprise user who is replicating data that needs to be at the sub-3ms range globally.

They have interactions where Vault is serving as the cryptographic backend for some very serious financial services applications, and Vault cannot fail, because if it does, it takes out the world economy.

That's challenging. They manage a lot of information to that mechanism, but simultaneously the same underlying subsystems protect a user where they don't care about speed. Instead, they care about the data being permissible and accessible and being the same in a datacenter in New Zealand and a constellation of satellites that fly over.

In that model, a 500ms round-trip delay is a pretty good day. If it was the other user, it wouldn't make sense. But the fact that you can have the same subsystems for light and heavy users highlight scalable Vault is.

But there are limits to that scale in the terms of the storage that Vault needs to back up.

If I need to have a system that backs up that information so that it's referenceable and secure and also accessible to users, sometimes I need to store those in external databases instead of Vault.

Again, in both of these use cases, we highlight challenges with external workflows, where there might be an orchestration suite that needs to happen across a lot of other different types of application platforms. Or maybe I just need to store too much data.

The data that is outside the scope of Vault is data that just grows too fast or that needs to be located external to Vault for the purposes of application referenceability or speed. Protecting these workflows has been a focus of the Vault team for around 2½ years. And the result of this is the Advanced Data Protection suite that you see within Vault Enterprise.

The Advanced Data Protection suite, or ADP, is a module that focuses on protecting these external secrets and workflows. I need to have Vault go out and intelligently work with a workflow or keep developers or users from being compromised while enabling them to act with security without having a deep background in cryptography or ultimately code in a lot of security measures themselves.

Let's talk about things like full-disk encryption and those kinds of workflows where Vault needs to serve as an extra key manager, but you don't want dozens of different types of bespoke secrets agents to manage.

For these types of enterprise protocols, we leverage the KMIP protocol, or the Key Management Interoperability Protocol, to ensure that, if we're doing something like full-disk encryption, Vault does it the same for all vendors.

KMIP was created about 15 years ago to allow traditional hardware security modules to conduct these kinds of external workflows, where they don't want to authoritatively interject themselves in an aggressive way that would compromise the ability of applications to leverage that infrastructure.

But at the same time, if I am NetApp or Dell EMC, I don't want to be in the business of trying to build bespoke key managers. I respect that you have some kind of security management infrastructure on the outside, and instead I want you to leverage that.

It's a way for them to all work together, regardless of the different ways that different vendors think about storage.

Same thing as with Transparent Database Encryption. TDE is a mechanism that allows you to have a database protect itself by encryption and manage those keys off-site in some other location.

There are a lot of different databases in the world: MySQL, Microsoft SQL, pretty much everything that you can think of that is web-scale cool and nonrelational.

They all have different ways of implementing encryption. But the one thing that they all agree on is the fact that they need to have a key manager somewhere to protect this information.

I might be MongoDB Enterprise or I might be MySQL. It doesn't matter. I know that I need to store this key off-site.

We came up with a good protocol for us to allow Vault to exist as a KMIP server. This is part of the ADP platform.

We want you to be able to build a functionality that ultimately doesn't require you to authoritatively interject this existing enterprise protocol or enterprise workflow. Because, again, you want secrets and security management tools that you actually use.

Otherwise, it's all for naught.

Cryptography and Tokenization

That's good from the protocol perspective, but what if there isn't a protocol, no mechanism for you to be able to automate that secrets management process for your existing workflow? And what if we run into that credit card number problem, where I've got a lot of data that needs to be protected but to reside outside of Vault?

That's where we go back to the drawing board and see how cryptography in general tries to address this problem. In cryptography, one of the primary ways to interject this is tokenization. That is the first approach that the Vault team took a look at. We were trying to figure out how to protect external secrets that live outside of Vault.

Tokenization is a process where you generate a random or otherwise one-way, referenceable identifier for information: a credit card number, a Social Security number, pretty much anything referenceable as a string. I can identify it in a one-way transform. And I protect the mechanism that's used to be able to ensure that a user or application can say, "Here is my unique identifier." I want to go back to a system that's external to me and say, "Hey, system, here's an identifier; here's my credential or identity. Give me back information."

The industry is typically trying to challenge that with a cryptographic hash, as a way to implement this kind of system. A cryptographic hash is a bit different from what you might see as a programmer with a normal hash algorithm. It's not necessarily because encryption goes on within it.

Instead, the "cryptographic" refers to the fact that there are as few collisions as possible. At least, with any kind of known constraints of data.

A collision is the ability for me to hash one set of data. If I was to hash "Andy" and get back 123, and then I hashed "Manoske” and got back 123, that would be a collision. I have 2 identifiers that resolve to the same kind of hash value.

Why is that bad?

On the Vault team, we don't think with only the focus of "How do I build cool security tools?"

We think about this the way an adversary thinks. If I'm going to attack a system like tokenization, I'm going to employ some kind of mechanism to be able to ensure that I can go around the security, something called a side-channel attack.

A common side-channel attack on tokenization, other than just breaking the system or exploiting a vulnerability in it, is to run what they call a "rainbow table attack."

A rainbow table attack is where I have a system of pre-computed hashes and I steal some kind of credential or assault or an identifier for the hashes. I run checks and see, "Hey, if I've stolen a cryptographic hash or if I know what this tokenized value is, I'm going to go see what it maps to." If I know the algorithm and some other key detail, I can probably search that list of common passwords or common identifiers, etc.

If I'm doing tokenization, it's really important that I have some kind of cryptographic hash algorithm or other kind of mechanism to be able to ensure that searching for a unique identifier is as difficult as possible.

Once I've done that, assuming that I've deployed tokenization, I store that secret within an external database. And then I protect the hash input table. This means that I'm holding on to what's called a "one-time pad encryption."

I've got this table that is hopefully non-referenceable, or that is not reversible in the case of tokenization, and I protect it. I really hope that an adversary isn't able to breach my system to take a look at that table, because otherwise they've got to contend with mathematics. They've got to contend with the difficulty of searching for possible inputs to that.

We can make that really hard using math. That highlights one of the things that we always want to do with Vault.

We always want to push people toward cryptography, so that they have to fight that, as opposed to fighting the entire system. Because unless they've gotten sufficiently advanced computers, that's going to take longer than the heat death of the universe to find that.

Most tokenization suites want you to do the same thing.

Problems with Tokenization

But there are challenges to tokenization. That's one of the reasons why our first approach to this is not going to be tokenization.

The most important challenge is that the data is not preserved when you tokenize something. If I'm properly tokenizing a value, that data is not going to retain the same format. "ABC-123" might come out as gobbledygook.

That challenge arises because there shouldn't be anything deterministic about how "ABC-123" is resolved, so that it's difficult for an adversary to run a rainbow table attack against it. We want to make sure that it's really challenging to go around the cryptography.

I'm going to generate a fully random output from that data.

Unfortunately, that means that I'm going to inherently destroy the underlying formatting of that data. Format preservation goes out the window when we talk about tokenization, and that's bad from our perspective.

At least from what we're trying to achieve, because when we want to build systems, we want to build a system that we're going to use, and if we're going to make it more complicated on top of the complexity we're already facing, that's not necessarily a good thing.

If we're going to go build tokenization, we're going to do so as the second ideal here, because we don't want to include and introduce more complexity in your system. We want your developers to be able to make your data management as simple as possible, even if you are using Vault as a mechanism to protect that data.

Space complexity is a big problem. When I am protecting data with tokenization, I am building a one-time pad effectively of all the mappings of that data.

I need to know how "ABC-123" becomes "#S)l_u!"." And if that's the case, that means that I'm copying that data effectively.

Think about one of the challenges that we were facing before, that the data is too big or was scaling too quickly or for some other reason—performance, etc.—can't live within Vault. If the data can't live within Vault, it's probably going to triple the space complexity challenges that we're going to face.

This makes tokenization a non-ideal first approach to this problem, because the challenge with tokenization is that it trips over the same kind of things that we don't want to trip over in the first place.

Which is, we make it harder potentially for users to leverage that information. And we challenge the complexity associated with your existing system by making it harder for you to protect that token table.

Instead of Tokenization, FPE

So what did we do instead? Our first approach is format-preserving encryption, or FPE. Within the secrets engine that was released in Vault 1.4, the transformed secrets engine, you can leverage FPE to protect data and also make sure that the data is encrypted.

Why did we choose FPE? First, FPE was the first mechanism that we chose to preserve this data because it preserves the data types, what we call data type preservation.

We don't distort the underlying formatting when we protect that data. For example, if you're looking at this input here, I can make sure that there are dashes in the right places. I can make sure that there are 4 digits for each kind of style input. And the output looks like something that is the input. It looks like the same data type, and that's really important.

We expect that in this kind of set of protections, we're going to store this date outside in a potentially format-limited or format-preserving data structure.

A really good example is the database. If I'm going to store in a database, there's a schema associated with the database. If I want to store that data in a way that is compliant with the schema of that database, I need to protect that information in such a way that it's going to look like that data information.

And that's great from a data management perspective. If I'm using a relational database where schema is very important, this is a very valuable kind of feature.

But it's also very valuable in other aspects. For example, if I'm using a file system and I don't want to make the size of the data variable such that it could compromise the size of a file, that's also very important. If I want to limit the size of it, etc., this is really important.

We also see format preservation and data type preservation as very important when it comes to managing text and character sets that are outside of English. So UTF-8, which is the common character set that we use within Vault, and also within most parts of the world outside of simplified Chinese and Japanese kanji.

This is a character set where I might accidentally violate not necessarily the schema of that data or the size of that data, but I might make something look like it's not supposed to.

I might try to write something accidentally in, let's say, hiragana that might come out in such a way that it compromises an application's ability to read that data. My application is expecting to get only, let's say, English text, and suddenly I'm dealing with UTF-8 identifiers that are not English. That could be a challenge.

So that data type preservation is very important, not necessarily just for scale or size of the schema associated with the database, but also for an application's ability to consume that information, to store that information, and ultimately when the time comes to swap it.

The second reason we chose FPE was that it changes the scope of the problem. You're no longer focusing on, How do I protect that kind of data, that one-time pad, that association of random tokens and identifiers?

That is a worthwhile problem to take on. But instead we focus on this as a key management problem, which is something that Vault is very good at already. And in focusing FPE into a key management problem, we ensure that we're not storing a copy of that data.

We instantly cut down on a lot of data that we have to store, and we ensure that there's less data that you have to manage.

Again, the whole point of one of the things that we're challenged with is the fact that data is complex. My infrastructure is complex and just getting more and more complex. So we want to reduce the amount of information that you have to protect or manage.

And FPE is very good at that because we can protect the data that lives within Vault in such a way that it's just like another key management problem to Vault. And you can ensure that you're not copying and placing more copies of that secrets data outside or managing another external system store to data.

How do we do that? We use the AES FF3-1 algorithm. Let's dive into what FF3-1 is.

There is a document called the SP 800-38G series, which is the special publication for security documents that is released by the US National Institute of Science and Technology in collaboration with the Canadian Centre for Cyber Security. This is basically a bunch of academic documents that guide users and security professionals on how they should implement protections.

SP 800-38G is a document that focuses specifically on how to implement format-preserving encryption and AES FF3-1 is a specific encryption algorithm that is mentioned in that document and analyzed as a mechanism to do that.

How does AES provide this capability for you to encrypt data while at the same time preserving its specific alphabet, which is those characters that we talked about? Whether it's Latin characters, Japanese hiragana, Arabic numerals, etc.

It can reserve that character set and it can also preserve the formatting. It's a certain number of characters, and then I'm going to include a dash or a pound sign, etc. It does this through a combination of Feistel networks and AES XTS transforms.

A Feistel network is basically a divide-and-conquer problem, where I'm going to divide up the plaintext that I want to encrypt, in this case into repeatedly half solutions.

If you study computer science, you might've seen something like merge sort, where I'm breaking everything up, sorting things, then merging them all back together. Think of it the same way.

A Feistel network is basically something where I'm going to break up everything that I'm going to encrypt with a line of text and then, once I reduce it to its atomic subsets, I'm going to perform a series of transforms on them.

In this, we are performing the AES XTS transforms. This is really important because this allows us to create transforms on potentially very small input sizes for protection. Why is that important? Because when we think about data that we're trying to protect, sometimes we might have a small input size. For example, let's say that I have andy.com as an email server and my username is Andy.

Let's say that my email is andy@andy.com. That's not a huge string for me to encrypt or tokenize. So, if that's the case, I need to be able to make sure that when I input that string that it's protected. AES FF3-1 is actually pretty good at this.

By using these specific types of AES transforms, what I'm doing is I'm dividing up the Unicode representation of that in such a way that I can provide a lot of security and I can protect that entire Unicode string with these well-known transform sets in a way that is algorithmically very fast.

The combination of Feistel networks and AES XTS transforms allows me to very quickly divide and conquer a string of inputs that would be going into this, even if that string is small, because we're using the UTF-8 recognition of that string, as opposed to just andy@andy.com.

I can then inline with the encryption process, substituted in characters, that would be ultimately defined previously or, if I know that I'm about to trip over something that has been defined as part of my string, like for example the @ sign or the .com sign, I can ignore that input.

To decrypt an AES FF3-1 string requires me to be able to use a key of tweak values. Tweak values are basically guides for how, when I go back to that string and I subdivide it again, to be able to decode that information, to walk my way down the potential options to be able to guide that series of transforms.

When you hear the term "tweak" with AES FF3-1, the best way to think of it is like a key, but it's a little bit different than most keys because it's not just a big block of text. It's effectively a guide for how the system walks back the information to be able to decode it.

The combination of these 2 kinds of mechanisms ensures that I can protect very small input sizes and I can manage a lot of keys on them, because when I'm storing that data in Vault, I'm not storing a copy of andy@andy.com or its encoded copy. I'm storing just the key, and specifically the tweak value other mechanisms necessary to decode that key.

Let's think about how we use format-preserving encryption in practice. French NIRs (numéros d'inscription au répertoire) use identifiers that in the US are similar to data that we call "Social Security numbers." If I need to manage this NIR data, I'm going to have to manage them pursuant to GDPR. And the challenges associated with managing French NIRs is that if someone gets access to this information, I immediately trip GDPR.

And if my supervisory authority is in France, they are not going to be happy about that. Because I have now just committed a serious data breach. The complications of that could cripple my company financially.

So how do we protect this data in such a way that it's going to live outside of Vault using the transformed secrets engine but retain the same format, retain the same character length, retain the same alphabet, etc.?

How I'd do that is, first, I'd mount an instance of the transformed secrets engine, I would then encode the French NIRs or SSNs within Vault, and then store them in an external database. And then, when an application needs to retrieve that data, they don't need to talk to Vault. They go to the database, and then they use Vault to decode that information.

This is very important: Vault is not storing the FPE transform, a copy of that data. Vault is only storing the mechanisms to decode that data.

The application is just saying, "Hey, Vault, I've got this piece of information. It looks like an NIR, but I don't know what it is because I'm going to assume that it's encoded. Can you decode this for me?"

And Vault says, "Hello, application. I've identified that you are who you say you are. I have passed you through all the required mechanisms that are necessary to ensure that you have the right privileges to access this, your Vault and your ACL privileges."

I'm talking purely about the open-source capabilities, and given that this is an Enterprise feature and also given the Sentinel features that are associated with you.

So making sure that if you have an endpoint grading policy for this feature, if you've validated the requirements of that but also the role that Vault has associated with the identifier, everything is on the up and up. I check it off and we're good to go. And we decode that secret.

The Value of Storing Tweak Values, Not Secrets

Again, Vault, isn't storing a copy of that secret. We're only storing the tweak value.

That's valuable for 2 reasons. One, the connection between the application and the database can be very fast. So I can focus all my time as an operator on optimizing the connection with the application and Vault for that specific kind of one-time transition. It's a lot easier to manage than managing constant or even ephemeral connections between the different applications.

You can absolutely do that too. In fact, our colleagues in the Consul team are very good at that.

But if I don't want to leverage a service mesh or something like that, this is a very good way of making management of networking associated with applications and databases easier. You're using the existing networking that you have.

Second, I'm not complicating the storage mechanisms of Vault. I'm not necessarily storing a massive amount of data.

I'm storing potentially millions of keys, but that's OK. Vault has already built the store and is managing millions of keys. I'm not storing millions of bindings or a large file that could be sitting somewhere.

This allows you to preserve the data architecture and networking that you have for an application while at the same time ensuring that you protect that application with very strong cryptography.

Again, if I want to go out and go break into this database with an exploit that I know for that specific type of database, as an adversary I'm confronted with something that looks like an NIR, but it's not an NIR. It is an encoded piece of information. And mathematically speaking, the process for me to decode that is just as hard as it is to protect and decode AES XTS information.

That's all the data that's protected within Vault.

And that is hard. This is another heat-death-of-the-universe kind of challenge where, given even a quantum computer, you're not able to break this information in an expedient fashion. You're much more likely to have to go talk to every single person in France and ask them whether or not this is their social security number. And even then, assuming that you had a lot of frequent-flier miles to be able to conduct this operation, it's something that is going to take you way too long.

Again, we want to make sure that, as an adversary, you are constantly confronted with this challenge where there is no side channel. There is no mechanism for you to be able to break past a vulnerability, your exploit within Vault or another system. We want you to have to deal with that hard mathematical problem. And in the case of Vault, this is something that we can leverage the transform secrets engine for.

Other Applications of FPE

Let's talk a bit about some of the other applications of FPE that we've seen in the wild. First, credit card numbers, the kind of age-old challenge associated with very high-speed information that needs to be externally referenceable and grows really quickly. The challenges associated with credit card numbers are highlighted by the fact that this is data that needs to be protected and referenceable outside of Vault.

It's of a size and scale that are far too large for a system like Vault to manage, but it also needs to be protected in a way that that information is proof against those complex adversaries that are going to go in and steal this information. They're motivated by committing financial crimes. You want to make sure that they can't otherwise be able to compromise that information using a complex cyber-attack.

This is an area for FPE to be very powerful. If I'm using a relational database that protects credit card numbers, each credit card number has a different way of identifying the number, whether its an Amex or Mastercard or Visa, etc. It's going to be something that requires me to be able to store it in a specific format. You know, Amex numbers don't look like Visa credit card numbers. They have a different string size, different coupling, etc.

That challenge associated with not being able to store things that are from different credit card number manufacturers in a specific way highlights the fact that I need to be able to protect the formatting of it. An Amex is just not going to work the same way if it's stored within a specific formatting that is not like an American Express credential.

That challenge associated with protecting a credit card number and predicting its formatting, and to do so in a way that doesn't otherwise require me to compromise my ability to protect and store that information in an external database, is very important. So format-preserving encryption: This is a great application for it. I can protect that information such that when I need to access that information, it is protected in that database.

It's in the same database that I expect it to be, but I could just leverage Vault to be able to protect it in such a way that if an adversary was able to dump that database, they don't gain access to that information in its plaintext state.

With electronic medical records, the challenge is even greater in the context of COVID-19. If I need to share sensitive information, this is critical privacy information that is the heart and soul of GDPR.

This is the most privileged, private information that's out there. And if I'm storing this EMR data, this PII (personally identifiable information), it runs into some of the same challenges that credit card numbers do.

I need to be able to make sure that the application is able to access that data in a very expected format, that it's going to live within the same database as it's going to live.

But the formatting of it may be very specific. It could be one of these social identifier numbers, like we saw with the French NIR number. It could be a Social Security number in the United States, or it could be a molecule, if I'm doing molecular analysis and molecular surveillance and trying to share contact-tracing information.

So formatting is very important, but with the availability of that data and the scale and size of it, our challenge is such that I need to store it in an external database, again.

This is a perfect application for Vault Enterprise. This is something where I would really need to use Vault to be able to encrypt and protect that data wherever it lives outside of Vault, but at the same time ensure that it is accessible by an application. And it doesn't otherwise add undue complexity to that infrastructure.

FPE to Keep Your Managed Provider from Seeing Your Data

Finally, the last good example of this is what we see with managed systems.

For example, let's say that I want to be able to protect a piece of information that is going to live in a managed systems provider outside of Vault. For example, I want to store this data in the cloud. And my cloud vendor is really good. I like my cloud vendor. They are nice people. But the challenge with them is that, for some reason or another, I am concerned that they might otherwise be able to unduly access my information.

There are a lot of regulatory requirements that are coming up where cloud vendors need to expose any access to information to legal authorities. If I'm an international entity and I don't like that legal authority that the cloud vendor is operating in, I want to make sure that even if the cloud vendor was compromised, that they couldn't access my information.

Some of this information is very format-specific. This information could be, again, the credit card numbers, the EMR numbers, everything we talked about before, or any other kinds of credentialed information that is otherwise very format-preserving.

If I want to make sure that data is protected in such a way that the cloud vendor themselves or any other kind of privileged managed services provider is not able to access that information, I can leverage forms or encryption to do that.

I can leverage the capabilities in Vault to make sure that I don't compromise where I'm going to store all this data or compromise the format of it. But I'm not going to make it harder for an application developer to write an application to consume that data.

All they need to do is come back to Vault and say, "Hey, Vault, can you swap this for me?" And we are good to go.

There are dozens of different types of use cases where format-preserving encryption is important. And it highlights what we're trying to do with Vault. We're trying to build systems that minimize the complexity associated with protecting data.

You should not be in a situation where you have to leverage the challenge and complexity associated with protecting a piece of information versus the security of information.

Security should never be a hindrance to your ability to conduct operations, whether we're talking about format-preserving encryption or the mechanisms within Vault, to be able to protect any kind of piece of information, to secure security orchestration workflow, to enable you to do full-disk encryption, transparent database encryption, BYOK situations with things like VMware, etc.

The Advanced Data Protection suite is there to allow you to preserve your method and style of conducting a security operation, conducting a traditional IT workflow, or accessing information or an external database while simultaneously ensuring that data is protected.

But it's a very rigorous, highly audited, and defensible mechanism of protection.

This is what we're trying to do at Vault. We are trying to force adversaries to confront cryptography while simultaneously making it easier for you to deploy cryptographic systems that make it really hard for them, or more specifically to make them wait past the heat death of the universe to be able to decode that information.

Thank you for your time.

Q&A

*Rob Barnes: *

Thank you very much, Andy. I'm absolutely mind-blown by that talk. There are so many gems that you dropped in there. Thank you very much for that.

I'd like to welcome Andy back to a live Q&A discussion panel, along with Jeff Mitchell. These 2 are for me legends. They are the foundations for how Vault was built.

They're idols to me personally. I can remember before I joined HashiCorp, I was always reaching out to them on Twitter and GitHub, for my issues and pull requests.

If you have any questions for them, this is your chance.

With that, I'm going to hand it over to Domi.

*Dominique Top: *

Thank you. And welcome to Andy and Jeff.

Let's just kick off right away. You did speak about secrets, but I think it would be important to briefly know what secrets management is.

*Jeff Mitchell: *

Anyone can define what a secret is. It's anything that you want to protect that you don't want adversaries or anyone unauthorized to see. Secrets management is just the paradigm of how you deal with the mechanism of protecting secrets.

It's one thing to say, "We can encrypt it."

But what do you do with that encryption key? How do you protect that encryption key? How do you gain access to it? There are turtles all the way down. Every time that you think you've solved one problem with security, you've shifted it to the next layer down, and you have to keep solving the problems over and over again.

Ideally in a secrets management solution, you're finding something that tries to take care of as much of that for you as possible, so that you can focus on doing the work of getting a secret, getting a secret back out, doing an encryption, getting something decrypted back out.

But let something else do the work of figuring out how to make sure that, every step along the way, it's defensible and cryptographically protected.

*Dominique Top: *

Thank you. Next question. Could you tell me about how this concept of secrets management has evolved inside of the world of Vault over time?

*Andy Manoske: *

When we look at how secrets management has evolved through Vault, it's really less a function of Vault specifically and more a function of how the world has evolved. Historically, we've thought of secrets management as tied to how data is supposed to be managed.

Think about managing an email address. I would originally think of it in a different way than I would for managing something like a credit card number or a Social Security number or a name, for example. What we've seen within Vault is, first, the evolution of us being able to make sure that the cryptographic processes to protect that data are the same.

You don't need to think about cryptography differently for protecting an email address versus a name. But also moving more toward a model where users don't have access to that information, where you minimize the possibility for employing those side-channel attacks.

And we maximize the ability for you to continue to conduct operations. You can use these security suites to conduct whatever you're trying to do.

In the case of something like an email address or a credential, you'll be able to leverage generating dynamic credentials or generating email addresses or other things that an adversary could theoretically compromise, but otherwise not gain access to your infrastructure with.

It's a combination of deployed dynamic credentials to be able to manage your infrastructure in a safer way, combined with the fact that I want to build one cryptographic infrastructure to protect everything and then focus on how I encode more workflows into it, as opposed to changing my infrastructure every time I change an application architecture.

*Dominique Top: *

Thank you. In regards to security, obviously not everybody is a security or a cryptography expert. Jeff, could you tell me a bit more about what's changing in the security landscape?

*Jeff Mitchell: *

Sure. We talk a lot about the crawl-walk-run journey at HashiCorp. If you think about getting to more modern workflows and more modern DevOps and secrets management, you have to start very slowly, and you have to start getting those first workflows onto your tools, and then you have to migrate more and more onto it.

One thing that I've seen is that, in many ways, the things that we have built into Vault are really ahead of where a lot of companies are. We've really thought about, What is it that you need to be able to do? You want to protect secrets.

A lot of companies are starting out with just a single, basic idea, like storing a key-value secret in Vault. And yet there are so many more things that Vault can do.

We think of things as dynamic secrets or static secrets. So, something that you put into Vault, you get it out of Vault, it's a static secret. Something that Vault generates for you, like a database password that it revokes over time, is more dynamic. And one of the trends we're seeing is that more and more people are starting to leverage those dynamic secrets, which is great because they get you more toward that zero trust model.

If you're dealing with KV secrets, then essentially what you're doing is you're saying, "I have a static thing. It might be a password. It might be a cryptographic key. And everyone who has access to it has that same value." And what you really want to get to is a more zero trust model where you say, "Every value that everyone has is different, or they never see the value in the first place."

I think that we're seeing people catch up to where we envisioned things would be.

At the same time, with the Advanced Data Protection module, I think that's us looking at it and saying, "We've been looking at the future for a long time and we believe in zero trust, we believe in identity-based access, all those kinds of things. But we need to help people get there."

And I think that the things we're doing with ADP that help you do cryptography while keeping it in an existing database schema, or a lot of the use like legacy applications with KMIP, those are things where we're saying, "We need to help you catch up to where we want you to be. We'd like to help you get there. But you're not there yet. So we're building these things that help you interoperate with what you already have that's legacy."

*Dominique Top: *

Rob, do we have any audience questions?

*Rob Barnes: *

We have plenty. Unfortunately, we have more questions than we do time. So I'm just going to ask 1, maybe 2, and then we'll see if we have more time.

The first one is: It seems like there's a steep learning curve here. How much does someone really need to understand about cryptography theory to get started? Can you recommend some practical ways for getting started?

*Andy Manoske: *

When it comes to using Vault, hopefully the answer to that is: none. We don't want to force you to have to sit through years of math and cryptographic study to be able to deploy Vault. When you deploy FPE, you don't need to know what an AES XTS transform is or anything else like that.

Now, if you want to get into the field, there are a number of really good guides and books and things you can study. If you're trying to leverage cryptography to build cryptographic systems, there's a lot of math that you need to understand, but you don't have to have a deep background in number theory or combinatorics.

*Rob Barnes: *

The next question is: Is there any guidance on data of very large sizes? I take it they're talking about either storing that data securely or maybe transforming that data using cryptography methods that Vault offers.

*Jeff Mitchell: *

It's funny: One thing that we always keep in mind is that Vault isn't a database, even though sometimes it looks like a database. When you're dealing with very large datasets, it becomes infeasible to do something like dump huge amounts of data into Vault. It just isn't built for that. However, it's very fast to do encryption operations.

What we usually recommend if you have very large sets of data is 1 of 2 things. First, create a data key locally. That's basically a keyword that you use to encrypt that locally, and then you have Vault encrypt that key for you. In cryptography terms, that would be a DEK, a data encryption key. Then you would use Vault's transit plugin to access a KEK, a key encrypting key.

Second, just send it through Vault's transit itself, depending on how big those payloads are.

Generally, when you have very large sets of data and you trust the systems that are holding the data or that are working with the data, it's often better to try to push the cryptography there and then protect the key, because it's much faster.

You get less data transmitting over the wire. That way, for instance, if you aren't sure that you trust your connection to the secrets manager, then you can ensure that the KEK never leaves the system.

You just are protecting an encryption key. Generally speaking, if you trust the systems that you're running on, you can push some of the encryption out. But if you don't, then make more use of Vault directly.

*Dominique Top: *

Jeff, in your opinion, what is the problem that everyone's trying to solve?

*Jeff Mitchell: *

That's a very broad question. I think that, from a general high-level standpoint, security is a too-easy answer for that. In some ways I think the answer is really, What are people trying to do with security? What it really is is that the modern world is complex. And the more things are interconnected and the more things are electronic, the easier it is for people to do bad things, and the harder it is to keep things that you want to keep private away from outside eyes.

For instance, it used to be that if I have a document I want to keep secure, I could put that document in a fireproof safe, and I have a pretty good idea that no one else is ever gonna see it.

Now, I may not want to trust a fireproof safe. I might want to put it into a cloud drive somewhere where it's backed up and where I feel like it's not 1 copy in 1 location. But now we have adversaries that can attack and try to discover that data all along the way.

That data is on my computer. It's in transit to the cloud service. It's at the cloud service. It's in the encryption protecting that channel. It just opens up a lot of venues.

So, as our world has gotten more connected, many things have gotten better and simpler but at the same time more complex. It's much more convenient, and I love our modern world. I love how we can do things electronically with computers rapidly.

But along with that, it opens up all these different avenues for adversaries to try to do malfeasance. So what's everyone's trying to do? They're trying to protect against that. Sometimes it's because they have internal secrets that would be embarrassing. Sometimes it's because of regulatory concerns.

But regardless of the reason, that's what it comes down to: You want to protect your secrets. You want to protect other secrets that people have entrusted to you.

*Rob Barnes: *

How do you handle data with FPE that have built-in checksums? For example, credit card numbers and DL numbers must satisfy a group property.

*Andy Manoske: *

Good question. Short answer: If I want to do what's called "convergent encryption" or any other type, where I need to have a specific encoded set of information, that encoded set of information is built in such a way that I can have a deterministically created tweak value. I can manage my own tweak value outside of Vault in such a way that that checksum value exogenous to Vault is enforced within the secret that Vault is encoding within FPE.

*Rob Barnes: *

When storing the encryption key, would there be a support for HSM for some of the secrets engines, for example, KMIP, transform, and PKI?

*Jeff Mitchell: *

If you're running against what we call "auto-seal mechanisms," that's something like HSM, already we have something that's on by default. You can turn it off, but it's on by default in the enterprise binary called "seal wrapping." What we've done there is we've identified the things that are a critical security grammar.

These critical security grammars are things like, What are the root encryption keys? What are the root certificates? Things like that.

What we do is we encrypt it not only with Vault, but with the seal mechanism as well. If that's an HSM, then we do this double encryption.

If what you're wondering how to get to higher levels of trust, that's something that we already do for you automatically by default.

If the question is about wanting to make use of Vault's PKI engine while having the signing operation done by a certificate that's in a private key that's in an HSM, that's something that we don't have now.

I don't want to comment on the roadmap, because that's in the domain of product. But I would say that if there are things like that that you want, then reach out to your TM or your SE and make your voice heard. That's something that the product can consider for upcoming things.