Terraforming the Kubernetes land: How HashiCorp Terraform works with K8S

Modern infrastructure can sometimes look like a huge wedding cake with many different layers.

For the longest time, HashiCorp Terraform could provision the lowest layer -- compute. But Terraform is also able to provision schedulers such as Kubernetes -- exposing the ability to schedule and scale pods, manage persistent volumes, and enforce limits on scheduling.

In this talk Radek:

• gives specific examples of using Terraform to manage Kubernetes resources
• shows the benefits you get compared to other solutions
• demos how to get from zero to an application running on Kubernetes

Transcript

I work for HashiCorp as part of the Terraform team, maintaining mainly the provider line, mostly AWS, but I also built the Kubernetes provider.

Quick show of hands, how many of you use Kubernetes? Okay. It's not as many as I expected. Anyone in production? It's about the same amount. [00:00:30] Is there anyone who uses the Kubernetes terraform provider already? Wow. Thank you. That's quite early.

Managing the Kubernetes cluster to me, really means managing a few different levels, so let me decompose the whole thing into three layers.

First of all, you need some kind of base infrastructure, or what I call base infrastructure. It [00:01:00] depends on where you run, if you have [inaudible 00:01:05] VMware, Amazon EC2, Google Cloud GCE, and once you have that, once you build that, you get to expose some kind of interface to the next layer. That interface typically consists of your favorite flavor of Linux, and then you get [00:01:30] to the next stage, to the next layer, which is configuration management. That part is typically handled by tools like Puppet, Chef or Ansible. In the Kubernetes role that involves things like XED, setting up the API server, installing Kubernetes, setting up CA certs, etc. Finally, once you do that you get to the last layer and that's the Kubernetes API [00:02:00] where you can config maps, schedule pods, replication controllers and generally Kubernetes resources.

Since the 0.9.something, or 0.10, there is the Kubernetes provider, which covers the top layer mainly. That's what we're gonna be talking about in this talk. You can certainly use Terraform to [00:02:30] manage the lower layer, but doesn't mean always the case.

Quick disclaimer: This talk is not supposed to convince you that Terraform is the only way to manage your communities resources because every company has different needs and workflows, so keep that in mind. Read that as there is no silver bullet. I'll try to describe context in which this provider [00:03:00] will be useful to you.

Just a brief outline of what we're gonna cover in this talk. Firstly, I'm gonna have a look into HCL. Why HCL may be useful to you. The nature of workload that would be good to run through this provider. Why synchronicity may work in your benefit. Why full life cycle coverage is important. [00:03:30] After covering the overlap of responsibilities between Kubernetes and Terraform, we're hopefully gonna get to demo.

HCL, or so-called HashiCorp Config Language, unless you manage your resources manually, which usually doesn't scale, if you're not using Terraform, you would be building [00:04:00] your own scripts. That means you would be using some kind of high-level language. For example, [inaudible 00:04:08]. Those languages are amazing. They offer you a lot of different ways to shoot yourselves in the foot, like classes, functions and loops and all kinds of things. On the other side you have languages for data, which [00:04:30] typically includes JSON and YAML. These languages lack these features. They were designed to transfer data obviously, which is usually good for describing infrastructure, but the trouble is that they also lack features which you may find useful, like referencing or comments for example. Despite that designing a DSL is a huge challenge, Terraform [00:05:00] and Puppet has chosen this path basically.

HCL is a separate project, which you can find on Github and it is used in various Github and HashiCorp projects, like Consul, Vault, Nomad and Terraform. It is also JSON compatible, which you may find useful for generated code. You may ask yourselves why generated [00:05:30] code? Why would you generate a code? Here's an example. The GDS, which is the Government Digital Service in the UK, has this project where they take advantage of this feature by having bind zone files for their DNS and they translate that into JSON and hand it over to Terraform, which can then create their DNS record in GCP, [00:06:00] DNS and Route 53.

Coming up, workload nature. What kind of Kubernetes workload is best managed by Terraform? At the moment we support these kind of resources. There's 13 resources in total and two data services, which is mostly the V1 coverage. Out of the V1, [00:06:30] those which kind of make sense to support in Terraform, so you won't find things like notes in there. There is a bunch of resources, which are kind of ups focused. This list basically, config maps, limit range, resource quota, secrets, name spaces, service accounts, and those are the ones that operators are likely going [00:07:00] to use. Operators are usually those who are already familiar with Terraform and they are already responsible for bringing up the cluster anyway, so this is a great fit for the provider.

Then we have what I call workload resources. We have a pod, which those of you who are not familiar with Kubernetes, pod is a collection of containers scheduled together and kept it alive [00:07:30] by Kubernetes. Then we have replication control, which basically replicates those pods for resiliency purposes and we have horizontal pod scaler, which scales pods in a given replication controller, usually based on consumed CPU. Finally, service allows you to expose parts out to the user. While you could ... Generally a [00:08:00] good use case for using these resources is for shared services, like monitoring or logging, anything that is shared within your company, within your teams.

Why? Because Kubernetes has this concept of admission controllers, which is a controller that sits in communities and get interest of any of your requests for creation of any resource. It can modify [00:08:30] the resource that you're submitting and give you back a different thing basically. As an operator you are more likely to be familiar with those admission controllers that are in place, that may affect the conflict, or may cause the conflict. That's why I'm suggesting that you should use these as an operator because as a developer [00:09:00] you might not be aware of these admission controls and they would cause troubles.

Synchronicity. Communities API is designed to be a synchronous, which allows scalability. That's great right? Communities always expect you to be proactive about asking whether an operation has finished or failed, at least initially? [00:09:30] Not all failures are re-triable though. Let's look behind the scenes for a moment. We have a pod here at the top ... Let's see if I can use this. At the top we have the name of the pod and we have a spec, which says what image that we want to launch and [inaudible 00:09:58] and the name of the container. [00:10:00] Now, the trouble is that what really happens behind the scenes, even though this is a declarative language, what happens behind the scenes is that Kubernetes needs to ask [inaudible 00:10:15] to pull down the image and start the container. A single container failing because of lack of capacity in the cluster probably doesn't matter unless it's the only one.

This is how you can create [00:10:30] a pod using the kubectl. You have some kind of [inaudible 00:10:37] config and basically, as I said, you have to be proactive about asking whether an operation has finished or failed. You would use this kubectl, my pod and ask for that status and if it has failed you have to go to the event log to see what caused the failure. In [00:11:00] this case it's just a typo, right? In case of Terraform, you get to see the error message that we pull out of the event log. If you do make a typo, you get to see the failure after a while because obviously we expect things to fail sometimes, but that's why there's the time out.

[00:11:30] This is how you create a service by a kubectl. You get to submit the service, then you ask for the service whether it was able to actually get the external IP and provision the load or whatever. If it has failed, because it may fail for various reasons, one of them being that you've reached your quota on GCP [00:12:00] for example. It's just typically something that's not re-triable. You're not gonna get through until you actually bump the limit. Once you do bump the limit, you still need to wait until Kubernetes creates the load balancer and gives you the external IP, which you can then reference elsewhere if need be, for example, in a DNS record.

Service [00:12:30] in Terraform can be defined like this. Here is again, the name of the service. Here is a selector, which basically says where to direct the traffic from the service, to which pod based on the label. Then we have the port forwarding here. It says from which port to which port to direct that traffic, and then the type load balancer means that it's going to provision our load balancer. [00:13:00] Here you can reference the load balancer once it is created in an Amazon Route 53 DNS code. What happens if a service creation fails in Terraform?

First, we hit Terraform apply and we wait for a moment to see if the load balancer was finished, our text was fully [00:13:30] created, and we get this error. Again, it's pulled off the ... It queries the event log to see what's there and what's the last warning or what's the last error to give you that context. In case of successful creation, in case you have a configure that just makes sense, and works and you didn't reach any of your quotas or anything, it can wait until the load balancer [00:14:00] is successfully created and therefore it has the IP address, which it may use in the DNS record later on. You can also see, it knows that the service needs to be created first before the DNS record. It's taking advantage of the graph basically in terraform. Depending on [00:14:30] your way of thinking in the company and whether you want to be proactive and keep checking for the status, and depending on your workflows and team responsibilities, synchronicity may or may not be what you're looking for. Just keep that in mind.

Now, up to full life cycle coverage. Why covering the whole life cycle is important. Well, because every app [00:15:00] will eventually be decommissioned. [inaudible 00:15:03] until it's decommissioned, right? Many folks believe that the initial creation is the only step to production, but reality usually differs. You actually need to create the app first, and then after a while you need to update it as well. Eventually, you need to delete it. With Terraform [00:15:30] you can basically use the same command and it will figure out based on the conflict, what to do. You can build a whole pipeline where you have diversion controlled code and it goes through the same apply command and enables you to do continuous delivery. That's why it is always advisable to review the plan before making changes.

Looking at the whole [00:16:00] life cycle here of a resource, we get to create the resource, we update it, delete it and sometimes we need to recreate the resource. Whether the resource is updatable or not often depends on a field. Again, to give you a brief explanation, service is again, something that allows you to expose [00:16:30] a pod due to user through an IP or a load balancer. Here in this conflict, not every field is actually mutable. You can't update everything straight away. In order to know that one has to read the docs very carefully or hope for the best. That's why it's useful to use Terraform where you can see from the plan clearly [00:17:00] that the selector is an updatable field. The plan will tell me that I can update it and the operation should not be disruptive. If I decide to change the cluster IP, that's a field that's not updatable, so that will be reflected in the plan again.

Coming up to responsibilities overlap. Where do we draw [00:17:30] the lines between Kubernetes and Terraform? There are four main areas which I would like to point out where you may see the overlap. It's good to know these areas because even though with the Kubernetes provider, because you can do something doesn't always mean you should. The fact that we allow you to do certain things. [00:18:00] Annotations, Terraform's expectation is always that what you submit to the API is what you get back, so that we can generate the diff and subtract. At the very least it's clear what aspect or what field of the resource may change over time so we can ignore it. That's not the case for communities annotations. What I mean by those is, the annotation that has this host name in there. [00:18:30] In communities these can be used to enable alpha or beat the features, but they can also be attached by the user to resources by the communities, resources to enrich the resource in a way that you can see what region or zone is it coming in. There is no creative line between user defined and [00:19:00] communities managed annotations, which is why we ignore them at the moment.

That said, there is obviously demand for managing those annotations, so it's a decision that needs revisiting. Coming up, workload. That's another overlapping context. [00:19:30] You can use Terraform to schedule a pod and then communities is responsible for keeping those containers within the pod up. That has nothing to do with Terraform. In this case, Terraform can however provide you the initial feedback. For example, a container has failed to start because you just made a typo in the tag or in the image. Next [00:20:00] example is replication controller. We are just moving up.

In case you manage your additional controller through Terraform, then Kubernetes isn't responsible for managing the pods, which are within that replication controller. Pod obviously manages containers. In this case, Terraform doesn't see inside the pod and it wouldn't even make sense to deal [00:20:30] with pod failures on this level because failures are just expected at that level. If you have an even more dynamic application, you're going to use pod on a scaler. Here's a bit of an overlap because Terraform needs to create the auto scaler which defines a minimum and maximum number of replicas, and it also needs to create the application controller, which defines the original number of replicas, [00:21:00] which will then be managed by the auto scaler. In this case, Kubernetes will obviously manage the pods and the containers, but you need to deal with the overlap. This is one way of dealing with the overlap, where you can specify the life cycle here. In the life cycle block you get to ignore changes, or you can say, [00:21:30] please Terraform, just ignore any changes that will happen outside of my control, IE. by the auto scaler in this field. You won't get to see the diff for this field. It will prevent the conflict.

Now, storage. Volumes can obviously always be managed by Terraform. By volumes I mean EBS or Google Cloud Resistance storage, but it depends on your workflow, [00:22:00] and trust between your developers and operators. It's quite common that people who run Kubernetes are a bit more relaxed in these relationships and a bit more flexible. They also demand very easy scalability. In this case, you would more likely let the storage class be managed by Terraform, and then Kubernetes would manage, provision these [00:22:30] volumes for you. You can do the same with persistent volume and same with persistent volume claim, because the API is obviously complex, so there is a lot of ways to do the same thing, or even pod, which can be a stand alone pod scheduled as part of the replication controller. Coming up, [00:23:00] service.

In Terraform you can obviously manage your load balances and turn all the knobs, all the fields directly, but that won't be very convenient if you have a more dynamic workload in Kubernetes. Their surfaces just come up and go away after, or in case your load balancers just get some dynamic IPs assigned or dynamic host names, in which case [00:23:30] it's just better to let Terraform manage the service and let Kubernetes provision the load balancers.

We are quite ahead of time and it's time for demo. Yay. Can everyone see that in that back? [00:24:00] Yeah? Cool.

We have ... Actually, let me switch these lights for a quick moment. We're gonna have two personas in this demo. We have Developer Bob and Operator Alice. In the first demo we're gonna deploy a who's on call app. It's going to involve minikube, which is a smaller version of Kubernetes that [00:24:30] you can run on your laptop and virtual box, and that will also involve replication controller and the service. Let's see, what do we have?

We have got this configure and we have the replication controller, the spec. [00:25:00] We have special image name for the container and then we also have resource limits specified, which means how much CPU and memory we can go up to and how much do we request. Then we have the service, which will expose this app to our users. You can see that we take advantage of the referencing here. [00:25:30] We finally output the name of the service, again by referencing the service because that's just handy. Let's see. Let me just verify that minikube is actually running. Okay, that's running. Now, we see if kubectl can reach it? That's fine.

First thing I'm going to do is [00:26:00] run Terraform in it, which would normally download the Kubernetes login, but since I have it compiled locally, it didn't really do anything. Now I can run Terraform plan which tells me you're going to create resources and I get to see all the defaults and fields that are going to be computed. [00:26:30] Since I'm happy with that, I run Terraform apply. Now that it's done and I can use this nice minikube feature, minikube service, and I can leverage the output from Terraform and yay. You know, obviously, this is just the first iteration, you know? Maybe the next iteration [00:27:00] we can just connect it to [inaudible 00:27:02] or something. You know. We will get there eventually.

Next up is our Operator Alice. Alice, because she's an operator, she just decided to go with the GKE. She's going to deploy [00:27:30] a GKE cluster and she's going to create a name space. Name space in Kubernetes allows you to scope your, for example, limits, resource quotas and generally scope down the resources. She's also going to create a limit change. Let's see. What do we got? We [00:28:00] have the container cluster, we are leveraging the Google Cloud provider here, give it a badass name and we used a random provider to generate the user and the password and give it some worth scopes.

I actually created this cluster before because creating the cluster [00:28:30] takes about three minutes, at least, and despite the fact that I like just watching stuff in the console, I thought you might not be like me. Then since we have the cluster up and running ... Actually, let me ... Let me do one thing. [00:29:00] I need this G Cloud command, to get the credentials for the cluster. We should have access to the cluster now. Just to verify. Yay, we do. That means Alice can [00:29:30] create the name space here. We just specify the name of the name space and the limit range, which basically limits the amount of CPU that a single pod, and the amount of memory that the single pod can consume within the name space.

Again, I run Terraform in it and plan. Plan tells me that I'm going to create the resources, [00:30:00] which is exactly what I intend to do. This was actually ... Oh yeah. That was fast as expected. There's not much to see on the WAP obviously, so let's go through the third part of the demo and [00:30:30] that is production. Bob and Alice came together and they said, why don't we just put this into production? They saw their conflicts and this conflict, for this one, has this provider block which defines how to connect through Kubernetes. Here we specify the conflict contacts which says connect to minikube. [00:31:00] In the case of Alice, she connects to the GKE cluster. The nice thing is that you can use Terraform workspace to have the same config and talk to both clusters. Here's an example of doing that.

Even though you might not be using workspaces, you actually are because by default you get a default [00:31:30] workspace. In this case, we say if the workspace is default, just use minikube, talk to minikube. Otherwise, talk to GKE. The other difference that we need to reflect in production on this case is the type of the service that we are deploying because minikube can use the note part, which means the service just exposes, [00:32:00] sets up the [inaudible 00:32:04] within the context of the note, but that's not quite helpful in case of GKE. In GKE you want the load balance to be proficient. That's a much more scalable way to do this. Terraform in it, plan. [00:32:30] Since we are at the service, we have the note part because we are deploying to minikube. That's done, so we can use minikube service and do ... Eventually, if I change my context [00:33:00] like that, it still works. I'm still on call. Then we get to switch the workspace.

Let's say we want to create a new workspace called prod and we also switched to the workspace immediately. [00:33:30] If I run Terraform in it, if I run Terraform plan, you can see that there are two new resources to be created. That's because the workspace allows you to separate out, isolate out the state. This part is going to take approximately one minute because in this [00:34:00] case we are letting Kubernetes provision the load balancer. I'll just let that go through and we can come back to it. In the meantime, I'm going to talk a bit about alpha and beta resources, which you might not have seen on the screenshot from the website. The communities, as I said, currently supports all of the [00:34:30] one resources as communities 1.6 and we have received many requests for alpha or beta resources, like deployment or ingress.

The problem with alpha or beta is the potential for breaking API changes and how do we deal with those without impacting the Terraform and Terraform user too much? I'm aware that Google and the communities community [00:35:00] and maybe other communities perceive alpha or beta slightly different to others and the API's remain in that stage for quite a long time. I totally respect that and I appreciate that they are so up front. The Google team working on the Google provider came up with a good solution in that area because the Google provider is also affected by this and I'll [00:35:30] probably apply that solution in this communities provider too, eventually.

Before we wrap up, let's see if this has finished. Yay. It has finished. You can just click on the IP. Yay. To wrap [00:36:00] up, keep a few things in mind. There's a few things to remember from this talk. Use DSL to your advantage and use the workload resources mainly for ups focused workload so it is very much ups focus provider, and be careful about running the application workload through it, especially if you use the admission controllers. Think about your culture [00:36:30] and workflows first before picking a solution and be aware of who owns the resource in communities. Try to avoid overlap if you can. Thank you.