Creating a multi-node cluster with k3d

In this section, we'll create a multi-node cluster using k3d from Rancher. We will not repeat the deployment of the echo server because it is identical to the KinD cluster, including accessing it though a proxy. Spoiler alert – it is even faster and more user-friendly than KinD!

Quick introduction to k3s and k3d

Rancher created k3s, which is a lightweight Kubernetes distribution. Rancher says that k3s is five less than k8s, if that makes any sense. The basic idea is to remove features and capabilities that most people don't need, such as:

• Non-default features
• Legacy features
• Alpha features
• In-tree storage drivers
• In-tree cloud providers

However, the big ticket item is that k3s removed Docker and uses containerd instead. You can still bring Docker back if you depend on it.

Another major change is that k3s stores its state in a SQLite DB instead of etcd.

For networking and DNS, k3s uses Flannel and CoreDNS.

k3s also added a simplified installer that takes care of SSL and certificate provisioning.

The end result is astonishing – a single binary (less than 40 MB) that needs only 512 MB of memory.

Unlike Minikube and KinD, k3s is actually designed for production. The primary use case is for edge computing, IoT, and CI systems. It is optimized for ARM devices.

OK. That's k3s, but what's k3d? k3d takes all the goodness that is k3s, packages it in Docker (similar to KinD), and adds a friendly CLI to manage it.

Installing k3d

Installing k3d is as simple as:

$ curl -s https://raw.githubusercontent.com/rancher/k3d/master/install.sh | bash

The usual disclaimer is in effect – make sure to read the installation script before downloading and piping it to bash.

Creating the cluster with k3d

Are you ready to be amazed? Creating a single-node cluster with k3d takes less than 2 seconds!

$ time k3d create --workers 1
2020/05/28 17:07:36 Created cluster network with ID f09fde83314b059d1a442ec1d01fcd62e522e5f1d838121528c5a1ae582e3cbf
2020/05/28 17:07:36 Creating cluster [k3s-default]
2020/05/28 17:07:36 Creating server using docker.io/rancher/k3s:v1.17.3-k3s1...
2020/05/28 17:07:36 Booting 1 workers for cluster k3s-default
2020/05/28 17:07:37 Created worker with ID 8a6bd47f7a5abfbac5c396c45f13db04c7e18749ff4d2e054e737fe7f7843010
2020/05/28 17:07:37 SUCCESS: created cluster [k3s-default]
2020/05/28 17:07:37 You can now use the cluster with:
export KUBECONFIG="$(k3d get-kubeconfig --name='k3s-default')"
kubectl cluster-info
real    0m1.896s
user    0m0.009s
sys 0m0.011s

What about a multi-node cluster? We saw that KinD was much slower, especially when creating a HA cluster with multiple control-plane nodes and an external load balancer.

Let's delete the single-node cluster first:

$ k3d delete
2020/05/28 17:08:42 Removing cluster [k3s-default]
2020/05/28 17:08:42 ...Removing 1 workers
2020/05/28 17:08:43 ...Removing server
2020/05/28 17:08:45 SUCCESS: removed cluster [k3s-default]

Now, let's create a cluster with three worker nodes. That takes a little over 5 seconds:

$ time k3d create --workers 3
2020/05/28 17:09:16 Created cluster network with ID 5cd1e01434edb1facdab28e563b78b605af416e2ad062dc121400c3f8a5d166c
2020/05/28 17:09:16 Creating cluster [k3s-default]
2020/05/28 17:09:16 Creating server using docker.io/rancher/k3s:v1.17.3-k3s1...
2020/05/28 17:09:17 Booting 3 workers for cluster k3s-default
2020/05/28 17:09:19 Created worker with ID 4b442116f8df7debecc9d70cee8ae8fb8f16783c0a8f111268be531f71dd54fa
2020/05/28 17:09:20 Created worker with ID 369879f1a38d60935908705f56b34a95caf6a44970beeb509c0cfb2047cd503a
2020/05/28 17:09:20 Created worker with ID d531937996fd25490276e32150b69aa2356c90cfcd1b480ab77ec3d2be08a2f6
2020/05/28 17:09:20 SUCCESS: created cluster [k3s-default]
2020/05/28 17:09:20 You can now use the cluster with:
export KUBECONFIG="$(k3d get-kubeconfig --name='k3s-default')"
kubectl cluster-info
real    0m5.164s
user    0m0.011s
sys 0m0.019s

Let's verify the cluster works as expected:

$ export KUBECONFIG="$(k3d get-kubeconfig --name='k3s-default')"
$ kubectl cluster-info
Kubernetes master is running at https://localhost:6443
CoreDNS is running at https://localhost:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

Here are the nodes. Note that there is just one master called k3d-k3s-default-server:

$ k get nodes
NAME                       STATUS   ROLES    AGE   VERSION
k3d-k3s-default-server     Ready    <none>   14h   v1.17.3-k3s1
k3d-k3s-default-worker-0   Ready    <none>   14h   v1.17.3-k3s1
k3d-k3s-default-worker-1   Ready    <none>   14h   v1.17.3-k3s1
k3d-k3s-default-worker-2   Ready    <none>   14h   v1.17.3-k3s1

You can stop and start clusters, create multiple clusters, and list existing clusters using the k3d CLI. Here are all the commands. Feel free to explore them further:

$ k3d
NAME:
   k3d - Run k3s in Docker!
USAGE:
   k3d [global options] command [command options] [arguments...]
VERSION:
   v1.7.0
AUTHORS:
   Thorsten Klein [email protected]
   Rishabh Gupta [email protected]
   Darren Shepherd
COMMANDS:
     check-tools, ct  Check if docker is running
     shell            Start a subshell for a cluster
     create, c        Create a single- or multi-node k3s cluster in docker containers
     delete, d, del   Delete cluster
     stop             Stop cluster
     start            Start a stopped cluster
     list, ls, l      List all clusters
     get-kubeconfig   Get kubeconfig location for cluster
     help, h          Shows a list of commands or help for one command
GLOBAL OPTIONS:
   --verbose      Enable verbose output
   --help, -h     show help
   --version, -v  print the version

You can repeat the steps for deploying, exposing, and accessing the echo service on your own. It works just like KinD.

OK. We created clusters using Minikube, KinD, and k3d. Let's compare them so that you can decide which one works for you.