Why I manage Kubernetes clusters on AWS with EKSCTL
I've been asked many times how I set up and manage Kubernetes clusters so easily. The secret weapon is EKSCTL, an Amazon co-engineered piece of software that manages the entire lifecycle of your AWS EKS cluster. In this post I'll explain what it is and why you should be using it.
The history of Kubernetes, AWS and EKS
Originally, Amazon Web Services (AWS) didn’t natively support Kubernetes. Amazon for a long while have been trying to push their own docker deployment technology, Elastic Container Service (ECS) instead. Any experienced AWS implementers would have war stories of its repeated shortcomings and the amount of oft-repeated effort migrating clients off of ECS.
Although AWS continues to try to improve ECS, it pales in comparison to what Kubernetes can do. Many engineers saw this, and the open-source community kicked into gear and authored the tool “kOps” which makes creating a highly-available Kubernetes cluster on Amazon trivial, instead of a complex task.
Amazon realized they weren’t going to win the war of attrition, and implemented their fully managed Kubernetes support service – Elastic Kubernetes Service (EKS). Implementers like myself adopted EKS using the Infrastructure as Code (IaC) tools we know and love to automate AWS.
We quickly learned that Kubernetes is not something you can just automate from the “outside”. There was a gap in what IaC tools alone could (and couldn’t) do in order to reliably upgrade, update, rotate nodes, and manage a Kubernetes cluster. The missing component here was that they needed to “also” manage Kubernetes – to speak to the Kubernetes API and to check, verify, update, delete, and manage things inside of its API, not just give it more resources (eg: add nodes) via the AWS APIs from the outside. Enter EKSCTL.
EKSCTL, the bridge between Kubernetes and AWS EKS
Amazon and partner WeaveWorks created a new open source tool to fill the gap between AWS EKS and Kubernetes. They wrote EKSCTL, which:
- Generates, submits, and manages CloudFormation templates via the AWS API
- Works with the AWS API to customize settings which are only able to be set in certain states that are difficult or impossible to do via CloudFormation
- Speaks to the Kubernetes API to manage the state of nodes and pods inside of Kubernetes
No other tool, including Terraform or CloudFormation, really does all three of these. With EKSCTL, I can manage the entire lifecycle of EKS with zero downtime and minimal effort over time.
While creating an EKS cluster without EKSCTL is not so difficult, Kubernetes needs to be maintained and supported to deliver its best value. Using EKSCTL eliminates much of the manual effort required to support effective use of Kubernetes, with zero-downtime.
Managing Kubernetes clusters with 100% uptime
The manual approach was the only way to maintain zero-downtime with Kubernetes, until EKSCTL.
Amazon realized that to effectively use Kubernetes is not just about setting it up, and that IaC tools don’t manage the lifecycle of Kubernetes.
Anyone who’s managed Kubernetes for a time has probably done what I have, and taken down a critical service as a result of a bad script. Or as a result of IaC tools destroying something unexpectedly at a really bad time. Or perhaps because of not “gracefully” removing a node, so the services have to reschedule aggressively to a new node which may not yet exist. EKSCTL eliminates all of these pitfalls.
Without manual fiddling in Kubectl or with custom scripts, EKSCTL will roll out new node groups and then gracefully remove old nodes, so your service doesn’t go down in production.
I have recently demoed using EKSCTL to teams who were just floored. What would have been a 2-3 day project – to roll out new nodes, pivot services onto them and remove old nodes, all with potential downtime and guaranteed stress… took minutes. Yes, it’s that good.
Can’t I just use Terraform/CloudFormation?
CloudFormation and Terraform do not exactly “speak” Kubernetes; they don’t use Kubectl, talk to the Kubernetes API nor lookup or manage things inside of it. They can’t ask Kubernetes to gracefully drain the nodes to ensure your services are highly available and always online.
Now, yes, if you are an experienced practitioner of either of these tools, there is no such thing as “impossible”. Though it is a lot of unnecessary work to take on, when Amazon wrote a tool to do this for you.
The painful deficit in using only CloudFormation or Terraform is only apparent over time, when you must intermittently run IaC commands and then run Kubectl commands, then call some AWS APIs; in order to update. With EKSCTL, you never need to do any of that ever again. EKSCTL allows you to spin up new/updated nodes, then gracefully spin down old ones automatically. You just sit back and let this tool do your job for you.
How to start using EKSCTL right now
There are virtually no barriers to transitioning any existing EKS cluster you have, however the heck you configured it, to use EKSCTL to manage it from this point on.
To get started just install EKSCTL and check out our sample EKSCTL commands and configs on GitHub.
DevOps-NirvanaRead more about the various methods of creating a Kubernetes cluster on AWS, why EKSCTL is my recommended option, and walk through how to get started creating an EKS cluster and a VPC in the following post:
Farley Farley
Are you ready to start really leveraging the power of Amazon’s Shared Responsibility Model to the fullest? With the benefits of using EKSCTL, there’s no good reason to wait.