HashiCast Episode 7 - Seth Vargo, Google

In this episode we talk to Seth Vargo, Developer Advocate at Google and HashiCorp alumni.

Join us as we take a dive into Seth's career, past, present, and future, why Seth is still talking about HashiCorp Vault and his love for technology.We may even find out if Seth has gotten over the PTSD of developing Consul Template.

Welcome to HashiCast the self-proclaimed number one podcast about the world of DevOps practices, demos, and practitioners.

Anubhav Mishra: Welcome to another episode of HashiCast. This one is a bit special for us here at HashiCorp. This week, it’s a privilege to welcome Seth Vargo, who’s a developer advocate at Google, also a friend of the show. Seth has previously worked at HashiCorp, Chef Software, CustomInk, and a few Pittsburgh-based startups. He’s also the author of the book Learning Chef and is passionate about reducing inequality in technology. Welcome, Seth. This is a brief introduction, but please tell us a little bit more about yourself.

Seth Vargo: Thanks for having me. I’m really excited to be here today. Like Mishra said, I’m a developer advocate at Google, specifically Google Cloud, focusing on the infrastructure and operations space. Like Mishra said, prior to that, I was the founding member of the technical advocacy team over at HashiCorp. I was Nic and Mishra before they existed. At HashiCorp, I worked a lot in the infrastructure and operations space, particularly around tools like Terraform and Vault and Consul and Nomad, but also tools like Chef and Puppet and Ansible, things from the configuration management space, but then a lot with containers as well—Kubernetes, container runtimes, etc.—and the complexities that come with managing those complex systems at scale.

Mishra: Today, I’m sure we’re going to go through your work at HashiCorp as one of the really early members in the company, and also we still follow a lot of the work that you’ve done, which is pretty awesome for us, because we have a blueprint that we can follow, which is pretty amazing. Today, we are going to take a journey through your experiences over the years. We’ll start with where it all began. And I want to ask you: Tell us what pulled you into the world of writing and building software and now your position as an advocate in the technology industry. Tell us all about that.

Seth: I think there are a few factors. When I was an undergrad at Carnegie Mellon, just a little undergraduate student, the program I was in was heavily invested in open-source software. It was actively encouraged that not only would you write code, but that you would open source that code. It was actively encouraged that you would consume other people’s open source code and create a vibrant ecosystem. And for context, this is before open source really took off. GitHub existed, but it was still in its infancy. This is back in like 2009, 2010. We didn’t have integrated CIs, and pull requests were just this thing that existed; they weren’t nearly as popular as they are today.

And I quickly found that—yes, I was writing code, and I was getting A’s in my classes (for the record)—but what I really enjoyed was interacting with people in the tools that I created and this open-source community through GitHub issues and pull requests. I open sourced this tool called ISBNdb. It was a Ruby gem that queried the—whatever ISBN stands for, the book numbers, the 13- and 10-digit book numbers—so that if you had a barcode for a college textbook, you could find that textbook, all the author information, all the metadata that had been published about it, and then also find it on different sites. Because it’s 2009; Amazon also wasn’t that popular at the time, people were still buying textbooks and things from other locations, including—I don’t know if you are aware of this—there used to be physical bookstores that people would walk into and hand over money or a credit card. Yeah, it was a different time, different time entirely.

But what I found is, over the years, I’ve flipped back and forth between being a traditional software engineer and an advocate. And what inevitably happens is, even if I become a software engineer for a short period of time (I write really cool things; I have a computer science degree, I can do that), but I get more benefit, and I get more enthusiasm, out of taking that thing on the road, blogging about it, talking about it, writing presentations, and basically showing it off to people.

And I think that’s really why I like the developer advocacy role. But then there’s the flipside of that, which I think a lot of people don’t see. Because as advocates, we’re very forward-facing, everyone sees what we do on stages and in blog posts. But there’s a whole other component to developer advocacy, or just advocacy in general, which is, I have this cool thing. Maybe I wrote it, maybe someone else wrote it. But yes, I’m talking about it. I’m trying to get people to use it. But I’m also getting their feedback, I’m trying to understand their pain points.

I’m trying to understand what the next version of this tool looks like. What features and functionality are missing? Where are things broken? And then I go back to my organization, and I take that feedback. And sometimes I have to massage it a little bit; sometimes people’s feedback is a little more abrasive than I would give to a product team directly. But you massage that feedback, and you give it to them for prioritization. And in a way you’re like a product manager or product owner for the community, where you’re representing their needs at the table among paying customers and internal stakeholders, etc.

Mishra: That you talked about, the flipside of being a developer advocate, I think that role is not an easy one. Because you are playing a devil’s advocate in your company, and you are trying to make sure of the direction of the project and that things that we are building and we’re working so hard to build are resonating with the audience, other practitioners. I’ve personally found it really difficult to funnel the feedback, filter it, and then get it back and talk about it in a way that’s consumable by product owners. I’m glad that you brought that up. And, as you said, not many people see that side of developer advocacy. I’m glad you care about it so much.

I think then, after university, you joined Chef, and Chef is known for its tools like Chef and InSpec and, more recently, Habitat. What was your experience like at Chef, and what were the areas you were focused on at Chef?

Seth: Before I joined Chef, I worked at a place called CustomInk. And there’s like a 100% chance that you have a T-shirt from CustomInk.com somewhere in your closet. But when I worked there, I was an intern, and then a professional contractor for a little bit. And they were pretty heavy users of Chef at the time. And I found some gaps in the ecosystem. I wrote this open-source tool called Fauxhai that basically collected machine data and made it open source—think like a giant JSON payload about systems, like CPU usage or memory and users or packages, but open source so that you could query it. And that really was like, my foyer into the Chef community. I think it’s ultimately one of the reasons I got hired at Chef.

When I joined Chef, I joined early. I graduated college a semester early, and I wasn’t supposed to start until June. So it’s like January 3, and I pick up the phone and call my hiring manager. And I’m like, “So I know I have this offer, but I graduated and need a job. Can I start sooner?” And they pulled some strings. And I was supposed to start on the community engineering team. But I actually started earlier on the sales enablement team. I was basically what you would call like a sales engineer. But I didn’t visit customers. I was doing a lot of back-end portfolio development, so that when the salespeople would go on site, they would have collections of resources.

I started the Learn Chef campaign as a result of that, which grew very popular. It was an online tool where you could learn Chef in an interactive way. We had hangouts, where random people from the internet could join, in all different time zones, and I hosted those. And then around June or July, when my real job was supposed to start, I transitioned over to the community engineering team. I worked a lot on the open-source tools in Chef, both Chef itself, the Chef core, but also Berkkshelf, I was one of the core original authors of Berkshelf, which is a dependency manager for Chef cookbooks. I’m still one of the maintainers for Chef’s back-end Fauxhai, which is a unit testing framework for Chef.

Nic: I have so much love for that tool. Unit testing and Chef were, for me, coming from a developer’s background—to start using infrastructure as code with Chef, and then realizing you could do cool stuff like unit testing, just blew my mind. I personally thank you very much for that tool.

Seth: Not a problem, Nic. That’s what I’m here for.

And then, following along that testing line, I was one of the core contributors to Test Kitchen and a lot of the drivers for Test Kitchen. I guess you could say that my job at Chef was very much ecosystem-related. Very rarely did I contribute to core Chef. But I wrote a ton of cookbooks, I wrote a ton of plugins or ecosystem tools that really helped enable other people to use Chef well. And then, for about a year, I transitioned onto the release engineering team. I effectively was Jenkins as a service.

Chef has this really interesting problem, where it’s primarily written in Ruby, the Chef software is, and you need to compile Ruby against a bunch of different target operating systems, Solaris, and AIX and Windows and Linux and Darwin, and they don’t have the luxury that Go has, which is you can do that on one build machine. You need a huge matrix of different Jenkins followers to build these and aggregate them and tag them and get them into Artifactory and eventually expose them via Apt and Yum repositories. I was working a lot with that on a really great team of people to make sure that people could get Chef in whatever way that they wanted to get Chef.

Mishra: I still feel the Chef community is one of the biggest communities in terms of infrastructure as code, config management. It has a lot of different personas that it targets. What was the fundamental truth that you came to. Like, “This is the one thing that just makes them tick.” What would you say that would be?

Seth: I don’t know if there’s really one thing that makes everyone tick. I think there’s a vast majority of people who are interested in solving their own problems. I think that’s really the crux of the infrastructure-as-code movement, this idea that, prior to Chef and Puppet and other tools, there were some companies out there that were trying to sell you an ocean. And I think what Chef and Puppet and Ansible and Salt do well is they don’t sell you oceans; they sell you Legos, and then they sell you a couple different schemas. One is the Starship Enterprise, one is the little Smurf dude. They give you recipes—no pun intended on the Chef thing—they give you a series of steps to take to compose those Legos into something that works for you.

I think that’s the emphasis that I learned working in the config management and infrastructure-as-code space, that everyone’s infrastructure is a special snowflake. A lot of people like to think, “We want to look like Google. We want to look like Facebook. We want to look like Amazon.” But the fact of the matter is, if you’re not the world’s largest search engine and ad agency, if you’re not the world’s largest online retailer, if you’re not the world’s largest social network, you just don’t have those same problems. And that’s okay. There are definitely things that we can learn there, and there’s a lot of sharing of knowledge that can be done. But I think Legos provide building blocks that let people build the things that work best for them at the stage in the lifecycle of their company. A startup’s requirements are very different than a midsize or very different than an enterprise.

Mishra: Transitioning a little bit. Then I think you join HashiCorp, which is like the most amazing thing, and I think you were one of the early employees. I think you were employee No. 4, which is amazing. Tell us about your first week at HashiCorp. believe you worked on something called Consul Template as their onboarding project.

Seth: Let’s back up before we bring up the PTSD.

When I left Chef, I took about a month off just because there were some negative experiences in the community. There were a lot of positive experiences, but also negative experiences. The fact of the matter is, there are just some not-nice people in the world. And when I left, Gigamon and a couple other places picked up an article, and a bunch of companies reached out. They were like, “Oh, come work for us, come work for us.” And I had met Mitchell at ChefCon in like 2014, 2013, and we had talked back and forth were DMing on Twitter. We texted a few times.

But he called me, which was weird—like, we don’t call people anymore. He called me, and he was like, “Hey, how are you doing?” I was like, “I’m good.” And he was the only company, I should say, that reached out and asked if I was okay, and I told him, “I’m taking a month off,” and he’s like, “Yeah, that’s fine. Whenever you’re ready, we’re doing this thing. Jack and Armon and I are starting this company.” HashiCorp didn’t exist at the time. There was Vagrant and a VMware plugin. “We’re going to do this plugin thing. And we have some plans, but just promise me that before you go work somewhere, you’ll talk to me. Give me a call.” So I did that. I took some time off. I called Mitchell. I talked with Jack and Armon. They decided to hire me. I still don’t know what they were doing.

And I joined. I flew to San Francisco. This was pre-Series A funding. So there’s no money. So we stay in an Airbnb, because San Francisco hotels are like $800 a night. And I go in for my first day. And the HashiCorp office—sorry, closet—is a conference room that we were subletting from this other company. It was very weird. They were never there. We had this conference room, like a round table, boardroom-type thing. And day one I come in. And Armon and Mitchell give me the grand scheme of things. This is like 2014, and they’re already talking about Nomad and Vault and secrets management and the strategy for how Packer is going to put Terraform on the map. And deploying immutable infrastructure.

And I’m like, “What did I sign up for?” I was just brain dead after day one. Day two comes around and Armon’s like, “Consul’s not really getting a lot of traction in the market. And we think it’s because there’s a gap in this thing. Could you take a look at it?” And at the time, he had written this thing called Consul HAProxy that’s still on GitHub, that is like dumping HAProxy config from Consul in a very specific format. Armon comes up with this genius idea that what we really need is an abstraction of Consul. We need a templating language where people can query for different information from Consul and put it in a file in any way that they want. And it needs to be kept up to date, basically instantly. Keep in mind that I had written like six lines of Go before I joined HashiCorp. No idea what I’m doing. And Armon’s like, “I think for like your first project, just go ahead and write that.”

I think Consul Template is like 76,000 lines of code, not including vendor dependencies. Later, we have Consul Template. At the time the company was so small, we had one-on-ones with Armon and Mitchell, individually. I had my first one-on-one with Armon. And he was like, “Yeah, sorry, I don’t know how you didn’t quit.”

Mishra: He still talks about it, by the way, about how Seth would probably quit the first week after we we gave him Consul Template.

Seth: But it’s a fun project. It’s still actively used on production systems a couple of million times.

Mishra: I used it today.

Seth: I think it is interesting, though. It showcases how I think HashiCorp was a little bit different than other companies. Like Mitchell and Armon both wholeheartedly and somewhat jokingly admit the mistake they made. And they were very young founders at the time and had no idea how to run a company. So giving someone a project that would ultimately become one of the core pieces of the HashiCorp ecosystem and expecting them to complete it in a week—like, that was a reasonable thing for them to consider. And I think they’ve come a long way in terms of understanding, hiring, and onboarding, and different people’s different capabilities, and giving people true onboarding projects.

But on the other side, I thoroughly enjoyed it. I learned more Go and more about distributed systems in one week than I think anyone on the planet has ever done. And I was only mildly stressed out once. But it was fun. I think it was a very interesting learning opportunity. I was very thankful that I was done, but also thankful for having been given the opportunity to do that.

Mishra: I think the result we see is that you pretty much worked on every tool. I remember prior to my joining HashiCorp, you were on every GitHub issue that I touched on—different projects across the ecosystem, answering questions on Twitter, answering questions for the folks, and even sharing things like the roadmap and making sure everyone’s focused and everyone’s on the same page when it comes to the roadmaps and stuff like that. I really appreciate your work there.

And then I also know that you worked on Vault, and Vault is something that I think is still pretty close to you, you still contribute to it. You’re still one of the core contributors to Vault. Tell us the story there, how did Vault come about and what we know as HashiCorp Vault now in public?

Seth: We had this tool at HashiCorp at the time—it was going to be the next big thing—called Atlas. Atlas was effectively Terraform and Packer as a service. It has since been sunset for a number of reasons. But the challenge was that if you want to run both Packer and Terraform as a service, they need to connect to cloud providers, which means that users have to give us credentials in order to talk to their cloud providers.

And that was kind of weird. We could encrypt them, but we needed a way to prove to people that the information they were giving us was both secure and inaccessible by the average person. Armon came up with the design for this thing that we were going to have inside of Atlas and that would later become Vault. It was just going to be this tiny little closed-source thing. We wrote a proof of concept for it. And then I suggested that it just become its own open-source tool.

And we started speccing it out, we started doing a lot of research, we learned about Shamir’s secret-sharing algorithm. Mitchell and Armon both did a lot of research and whitepapers and stuff on the background and came up with this idea for Vault. We wrote it mostly to use internally. The plan was to open source it, but—whatever—eventually, right?

And I forget—we were talking to some customer at the time, we were like, “Yeah, we have this system, it does these things, this is why your credentials are secure. We’re basically selling Atlas. I think Kevin was doing it, talking to this customer. And they’re like, “Well, you have this system. What is this system?” And they were very interested in this tool that we had built that that did dynamic acquisition of credentials, for example.

We took a step back and decided maybe Vault should be its own thing. And it became its own thing. We redid it from scratch, open sourced a lot of it, or open sourced all of it, gave it a brand, gave it an identity, did a big announcement around it at a HashiConf. And I think, since then, it’s obviously taken the market by storm. It’s one of the leading secrets management and credential identity brokering systems out there, and it’s grown from even what we originally imagined it to be, back in 2015.

Nic: So with Vault, you seem to still be doing quite a bit of work on it. You did a presentation at Next with Armon on Vault, I saw you doing a podcast a while back, and a few different things. Are a lot of those self-led activities, just because you’re interested in the product and believe in it still, or is it anything else?

Seth: Yeah, I do everything. I’m not a big fan of sleep, but I definitely work in with Vault a lot, in particular how to run Vault in this new distributed, containerized world, where Kubernetes seems to be eating most of the world. I’m not just working with Vault; I’m still doing a lot with Chef and Terraform and a bunch of other tools and the DevOps ecosystem,

I would say that it’s partially self-led. And partially because Google is very interested in enabling customers to choose what software they run on their platform. Obviously, Vault is the leader in the market, and it’s open source. Small customers and large customers are very interested in being able to audit the source code. They’re very interested in running it on different cloud providers like Google.

We obviously have an interest in making our customers successful on our platform, which is why you’re seeing us starting to do investing and things like the Google Cloud Storage back end, and we wrote the spanner back end, making improvements to the different plugins, including writing the GCB secrets plugin, which gives you time-based IM credentials, and some other stuff we have in the pipeline. Basically, just making sure that if users want to use Vault on GCP, they have the best possible experience. And that’s ultimately our goal with any product, whether it’s one we build internally at Google, or some third-party open-source tools. If people are using it, we want to make them successful on our platform.

Nic: That’s really neat. And I want to talk about Vault and Kubernetes. But I’m just wondering, maybe it is worth back-tracking just a little, just in case there are a few listeners who are not familiar with Vault. Vault is obviously an open-source secrets management platform. But maybe it’s worth approaching it from the perspective of the developer or the operator. As a developer or operator—you touched on secrets and the need to be able to manage them all—what does Vault give me? What can I do with Vault?

Seth: I think there are four fundamental pieces to talk about with Vault. As a developer or an operator, I need access to secrets—as a human, right? I’m a human being. I need to connect to a database so that I can run some read-only command or get a schema. As a human, I need to SSH into machines at some point in time, because I need to debug something in the staging environment.

But as an application, I need to also get database credentials, as some code, I need to be able to talk to a database or a Redis instance, and I need to get information. There are two distinct audiences in the secrets management world: There are humans—operators, security teams, developers—and machines or services. Those might be containers, they might be bare metal, they might be a Raspberry Pi; it doesn’t matter. And we need a strategy for both of those identities to be able to access these credentials.

That’s the first bit, this identity brokering system, where humans can authenticate with things like username and password or GitHub, whereas machines can authenticate with things like TLS certificates or instance metadata, things that are cryptographically complex in nature. That’s the first thing.

The second thing is Vault has this idea of a key value store: You put data in, it gets encrypted in transit with CLS, and at rest with ES 256 CBC encryption. Honestly, most software engineers could build something like that in an afternoon; it’s just an encrypted key value store. Think of it as encrypted Redis or encrypted mem cache. But on top of that, it has a really powerful ACL system. And that’s something that you probably wouldn’t be able to build in an afternoon, being able to give people read-only, read-write, access, sub pathing, etc.—all of that’s built into Vault’s ACL system.

But where Vault’s real power comes from is this concept of dynamic credentials. Yes, I could create a database password and put it in the encrypted key value store. Great. But what would be better is if instead, I could give Vault some credentials to actually connect to my database and have Vault generate those users. Instead of opening a JIRA ticket and having my DBAs create a username and a password to connect to my Oracle database, instead I configure Vault to talk to that Oracle database, I give it a username and password with privilege to create other users. I work with my DBAs to codify or capture that configuration as code. And then I programmatically generate these credentials.

Any user can generate these credentials; they don’t have to wait for the DBA. You take something that’s manual and you automate it—that whole process has been audited and logged. But what’s different about those credentials is, unlike the credentials that a DBA may create through a JIRA ticket, the credentials that Vault creates have a lifetime. They might live for 30 minutes, they might live for 24 hours—all of that is configurable. But at the end of that lifetime, those credentials expire.

And this is very important because it prevents secrets sprawl; it prevents hundreds of credentials living out there forever. It significantly reduces the surface area for an attack. Because even if an attacker or hackers are able to gain access to those credentials, they have, at most, 30 minutes or 24 hours to do something nefarious with those credentials. Those credentials can also be revoked early. If you have some network intrusion detection system and you decide that a hacker is on your network, you can revoke them early. So even if they have them and it still hasn’t hit the 30-minute mark, you can still revoke those credentials early. And again, that whole process is audited and logged.

What’s really great is that each time an application, service, or human requests one of those credentials, it’s unique. If I have 10 services or 10 instances of one service and they all request a credential to my database, they all get a different credential. Whenever I’m looking at my audit logs or my database logs, I have what’s called “provenance,” which is this one-to-one mapping between an application, service, or human back to the credentials that it’s using. I can know definitively that if I see anomalies from this one database user, I can tie that directly back to the application and know for certain, or with high probability, that that application has been compromised, instead of having one database password everywhere. In which case, you’re like, I don’t even know where to start.

On the flipside of that, if that one application is compromised, you take down just that one application, and all of my other services continue to operate and function as needed, because they’re using different credentials. I’m not revoking one credential that is the keys to the kingdom. And this is a really great architecture, if you will, for designing distributed systems. Because we not only have to think about security and this onion-like approach where no longer do we have this overall firewall/perimeter security, where everything inside is trusted.

But this idea that we are not as smart as hackers or we’re not going to find all the vulnerabilities—you look at the Spectre vulnerability, there was no application code we could have written that would have prevented that, that’s a core vulnerability in the way CPUs work. We need to prioritize or at least emphasize revokeability and time-based access over the security itself. We get to a certain bar of security, and that’s great. But then, after that, we have to start thinking about our break-glass procedure, our disaster recovery, our revokeability. And that’s where Vault really helps build a strong story around those patterns.

Nic: Is fault just for enterprises? Because what you’re describing sounds like it’s pretty easy to use.

Seth: No, it’s definitely not for enterprises. I think enterprises benefit from some of Vault’s enterprise features, like integrations with HSMs and things like that. But the open-source version of Vault is really great for even a small or midsize company. We ran Vault internally at HashiCorp when we were less than 20 people. There are obviously some major banks and financial institutions that are running Vault at the same time, but it works anywhere. I would say, maybe for your home office it’s a little bit of overkill. For personal use, you might want to leverage something like 1Password or Last Pass, some type of GUI-based application. But for any company or organization where you have to share credentials, I think Vault is a really great choice, regardless of the size or the industry or the scale.

Nic: Back into Kubernetes ,and we talked with Mitchell last time about the integration between Consul Connect and Kubernetes. But do Vault and Kubernetes play nice together? I know that there’s the Kubernetes authentication, but from a running-an-application perspective, is it a good process to be able to access secrets and things like that from Kubernetes?

Seth: I think there are two parts to the story. The first part is, Can I run Vault on Kubernetes in the same way I would run an application or service? The second question is, Now that I have Vault running—maybe it’s on Kubernetes, maybe it’s on Heroku, I don’t know—can I access it from Kubernetes?

Let’s talk about the first one. Vault runs really well on Kubernetes. It specifically works even better on GKE, which I’ll talk about in a second. But Vault has a container. It’s published on the Docker Hub. So you can just pull the official container; you can run that directly in GKE. Or you can mirror it to the Container Registry if you’d like some faster start times.

I’ve published a set of Terraform configurations, and Kelsey Hightower has published a set of what I like to call “Terraform configurations the hard way” with the gcloud command that go through exactly how you set up a Kubernetes cluster on GKE with Vault deployed. And this cluster is deployed with the best practices and the production hardening guide that HashiCorp recommends. We use memory locking, we use high-availability, leveraging a stateful set, which will give us both predictable naming and “exactly once” semantics.

It has built-in auto-unsealing. Basically, most of the operational overhead is taken care of by Kubernetes itself or some sidecar applications. The reason that you might want to choose GKE over your own Kubernetes cluster, for example, is the way we supply entropy to the containers and the VMs that are running on GKE.

We supply true random via TRNG to the containers on the cluster itself. This gives you significantly more randomness, which gives you significantly better cryptographic distribution on your cluster.

Nic: How does that compare to an HSM for entropy?

Seth: It’s not quite as high of entropy as something like an HSM. That being said, if you have the Enterprise version of Vault, you could use the Google Cloud HSM product and connect it to that, which is a physical HSM, physical data center, that will give you higher entropy. I don’t know the exact numbers, but it’s significantly more entropy than a standard VM with Docker. It mirrors much closer to a bare-metal source of randomness. But it’s not quite as random as an HSM.

Nic: And random is obviously incredibly important for anything that’s cryptographically oriented.

Seth: Yes, if you’re generating the same set of random numbers continuously, you’re going to have the same keys, you’re going to have very low-probability distributions, very low curves. It reminds me of a story: When I was in high school, we found that if you took a TI-84 calculator and did a factory reset on them, and then did “equals rand(),” they would generate the exact same series of random numbers. We really freaked out my physics professor whenever we told her that we could predict the random numbers that her calculator was going to produce. And we did a real quick factory reset and handed it to her. And she was very flabbergasted.

So, yes, random is important. Random seeds are important, but having a solid source of random is important. What you might find, if you’re running, say, Kubernetes on-prem, or if you’re pulling for /dev/random instead of /dev/arandom, for example, you can block.

So if Vault is trying to do something cryptographic, and it can’t generate enough random bytes or source enough random bytes, /dev/random will block, which means all your operations are now blocked, waiting for more entropy in the system. /dev/arandom doesn’t have that problem. And obviously if you hook up like a real virtual random device, you don’t have that problem. This is important as you start scaling Vault and running Vault in a container or in an orchestrator. There are a lot of considerations to take care of, in terms of randomness.

Nic: I suppose the question that a lot of people are going to ask is, “Well, why not just use Kubernetes secrets?”

Seth: I think that’s the other side of the question, which is, Now I have a Vault. I have some other team that has given me an IP address where Vault exists. How do I get the credentials from there? Should I use Kubernetes secrets? Should I have some sidecar application? I think there are a few things to consider here. There’s a lot of stuff on the Kubernetes roadmap that is going to make Kubernetes secrets better. The challenge with Kubernetes secrets right now, today, is that they’re stored base64-encoded in etcd.

Anyone with access to the etcd cluster effectively has plaintext access to the secrets. And this is because base64 is not encryption; it’s just encoded. There are a lot of items on the roadmap to encrypt those values, provide sidecar encryption, etc. There’s been some work for KMS enveloping. I believe there’s a plugin that was published that lets you use a third-party KMS provider, including Vault’s own transit back end, for encrypting data before it’s written to etcd and automatically encrypting it when it comes out.

I think there’s a lot of merit there. I do think there’s some exploration that we need to do in terms of the security of that model, but also the operability of that model. Right now, you have another system that you have to keep up and running in order to get those credentials. And the overhead of doing that encryption and decryption. Another pattern, and the pattern that Armon and I showcased at our NXT talk (GCP NXT) was this idea of running a sidecar container with a tool like Consul Template or Enterprise Consul that basically does the whole Kubernetes authorization process for you, uses the service account job token, authenticates to Vault and pulls down a Vault token, and hands that to Consul Template. And Consul Template then presents as a file on disk in a virtual file system, or an in-memory file system, the data to the application. This pattern is really great, because it means that you’re not tying your application to Vault. You’re instead tying your application to a config file, which it probably already has that type of coupling, let’s be honest.

The idea here is that local development becomes a breeze. You have some stock template to use for local development, you can run a Vault server in local development if you want to. That’s one of the reasons I love Vault: It’s just vaultserver-dev on any operating system. Or you can have fake data in development and you don’t have that runtime dependency in development on Vault. This is why that abstraction is really nice.

I do think that there are a lot of integrations that we could think about. Like, maybe there’s a world in which Kubernetes secrets are an abstraction and we can write a Kubernetes secrets plugin that’s for Vault. And someone might write one for CyberArk and someone might write one for KMS, etc. I don’t think we’re a world where that has really been proposed yet. I know it’s been talked about a few times. There are no RFCs or anything put forward yet, though. But I would personally like to see Kubernetes secrets be something a little bit more pluggable than just this direct to etcd backend.

Nic: Just back on yourself: What’s next with Seth? What’s coming up? What are you thinking about? What’s interesting? You’ve got a bit of a sparkle in the back of your eye there around new technology or something like that.

Seth: I mean, obviously, the blockchain, I mean that’s what.

Nic: I’m dumbstruck that Google doesn’t have a managed blockchain. It really makes me sad. When is that coming?

Seth: I can’t discuss upcoming products that may or may never exist. I try not to say the “b” word a lot. But it’s funny because, most people don’t know this, but Vault’s enterprise implementation for replication is based on the blockchain. But it was based on the blockchain far before blockchain was important. The way Vault does performance replication is via Merkel trees, which are hashes of hashes, which is what the blockchain is. So it’s actually really funny; I was being funny, but then I realized that there’s actually a tie back into Vault here.

I think in terms of what’s next, the landscape is rapidly changing, particularly with containers and orchestrators, and I like to play hypothetical scenarios in my mind. Like, 10 years from now, everyone in the world is running on some orchestration framework, whether that’s Kubernetes or Nomad or Mesosphere. No one cares about VMs, no one cares about bare metal, or most people, like the vast majority of people don’t care about that. What’s next? I think there are a lot of interesting things that we’re working on at Google that have really caught my eye, particularly around voice commands or even Google Assistant, things like Duplex and how companies can build deep integrations with those AI-based tools, the stuff we announced at Next around BigQuery SQL as a language for machine learning. You don’t need a PhD in machine learning. You can now write SQL and you’re doing machine learning, which is crazy, right? That opens a whole new world for people. I think that’s exciting.

But what I want to look at is, “What’s the next big problem?” If you look in the 1980s, it was, “How do I get physical machines into a data center?” We solved that, right? Thanks, FedEx. And then, the ’90s came along, and it’s, “How do I virtualize these machines to maximize isolation and resource consumption?” And that’s where Hyper-V and VMware and the different hypervisors, xhyve and bhyve and all that stuff came from. Great, we solved that. The 2000s come along, and it’s, “Okay, well, VMs are great. And I’m using resources. But I need an isolation layer for my application now. I need to run 50 versions of Ruby.” So we have userland isolation, right? And that’s where containers came in.

It’s like, what’s next, right? What’s the next isolation layer we need? What’s the subcomponent of a container—if there is one? Or is it higher level? Are we moving to the world that’s more Heroku-like, which is like, I just push in code, and it runs somewhere. And I don’t care if it’s a container or a pod or an app or a service. I just want to get it a credential, talk to the database. And I don’t want to think about it.

Nic: Mishra and I were just talking the other day. When are we going to get Go on Google Cloud Functions?

Seth: We announced at Google Cloud Next the ability to run Google Cloud Functions as any container. That’s currently in alpha, so you can register for early access to that if you’re interested. Obviously, Go runs in a container; so does Java, so does Jazz, so does Python, so does Crystal, anything you want to run at that point, will work on GCF. And I think this is really the future. We talked a lot about building first-class language support for Go and other languages. And it’s really just not feasible for us to support the vast matrix. We’re always going to be leaving some language behind either because we don’t have the expertise in that language or because the language evolves much faster than we push new updates to Google Cloud Functions.

By giving you an arbitrary base container from which you can run your function, we effectively allow you to run any language. You could pack your own dependencies in your own container, and your own private Google Cloud Container Registry, and you’ll have faster startup time, faster boot time, faster cold boot time, because we’re pulling from GCR. And you can run on whatever language you want, including your own proprietary language you wrote in-house that you’re never going to share with anyone.

Nic: I really like that as an approach. I’m 100% behind that. I think the dependencies thing is the key thing. Maybe if you want to do something like image magic, like trying to get that to run in a cloud function is—I’m going to check that out and have a play-around with that. Do you think this is going to be a future direction for Cloud Functions? Because it makes sense. I mean, it’s not difficult to create a language container for your function, and packaging with something like Docker is pretty easy as well. It’s almost as easy as zipping something up. Is this gonna be a future direction? Or is that still a little bit unclear?

Seth: It’s currently an alpha. We’re testing it out. And we’ve obviously built it, and we’re getting people’s feedback on it. If all of our customers and future customers are like, “Yes, we love this. This the future,” then it’s the future. If our customers are like, “This solves 1% of my use case,” then that’s probably not the future, right? It’s just a future. I don’t think we know that yet.

Nic: I think it sounds pretty dope. I’m looking forward to that.

I think it’s been amazing to talk you, Seth. It’s been an absolute pleasure to catch up with you and see you again. But really I want to thank you for spending the time. Just a quick question: Where in the hell in the world are you right now?

Seth: I’m at home right now, which is very rare. But I go to New York on Wednesday.

Nic: Can we get a fact check on that?

Mishra: You don’t believe that.

Seth: I’ll send you my location on iMessage.

Mishra: Seth, thank you so much for taking the time and talking to us. This has been incredibly interesting. Amazing conversations, all the way from your past to what we’re talking about today in terms of serverless computing and things like that. One final—I think the most important question. It has nothing to do with the podcast or what we talked about in the podcast. I hope you’re ready for this because this is really important stuff. If you were an ingredient in a hamburger …

Nic: Or a vegan burger, or a veggie burger, just in case you don’t eat meat.

Mishra: What would you be and why?

Seth: That is a tough one. I mean, two answers come to mind. I think first and foremost, if I were an ingredient in the hamburger, I would have to be the hamburger. But if that’s not an option, I think Worcestershire sauce is probably my go-to. I think it gives, especially if you’re grilling the burgers—there’s not enough information here. To solve the problem, we need to take this back to the product managers and get more information. But I would say if we’re grilling these burgers, especially on an open-flame grill like a petrol grill, we would definitely need Worcestershire sauce. It gives it that smoky flavor, but also like a little bit of tang, almost like a sweet and sour. But it really helps the burger stay moist on the inside and the outside. Yeah, that’s probably what I would go for.

Nic: I was gonna say, comparing that to your role at Google: Are you the thing that gives all of Google’s products that little bit of a tang and just that little bit extra flavor, and … ?

Seth: I’m like that animated GIF where they sprinkle glitter everywhere. That’s me.

Mishra: What about you, Nic?

Nic: I was thinking bacon, because, well, why not? But I actually think cheese, because I think—I don’t know why, but I just really like cheese.

Mishra: Cool. All right, thank you so much, Seth, for making this time. We all appreciate this. And this is a rare opportunity. But I’m glad you were able to make some time for us.

Seth: Yeah, thanks so much for having me. This has been great.

Nic: Cheers, Seth.

Seth: Cheers. Bye-bye.

Mishra: You’ve been listening to HashiCast, with your hosts, Mishra and Nic. Today’s guest has been Seth Vargo from Google. Thanks for listening. And we hope you join us next time.