Creating a Multi-Arch Kubernetes Cluster

During the last days and weeks, we focused on Kubernetes. It sounds like the most logical next step for evolving our CI/CD pipeline. And we knew before that Kubernetes is a different beast. It takes a while to get used to the concepts of Kubernetes, but that’s not even the hardest part. We figured out that support for setting up a Kubernetes cluster on your own machines, maybe even HA (High-Availability = multi masters) or using different architectures for the master (amd64, linux server) and nodes (arm, rpi3) is still very limited. There are some tutorials out there, none of them really worked for us or with “our” version of Kubernetes. After a lot of tinkering and experimenting, with bloody fingers from flashing SD cards (fresh OS installs) we finally figured out a simple way to install a Multi-Arch Kubernetes Cluster. It’s not HA (yet, once we figure that out we’ll definitely write another blog post), but:

  • a single Kubernetes master runs on a capable linux server, typically having amd64 architecture, running a flavor of Linux (arch in our case)
  • several nodes are running on available Raspberry PIs

 

Why can’t I just follow the tutorial at Kubernetes.io?

Well, it did not work for us. There are two main reasons:

  • There are two multi-arch networks that you can use, mainly Flannel and Weave. We heard good things about Weave, but it failed miserably for us. We believe this is a momentary situation, but as of Fri May 4 2018, using the Weave network on arm devices seems broken. Then there is still Flannel. Using the default YAML for the API Objects will install amd64 binaries on your PIs, which then of course fail. We extended the config to choose the right binaries depending on the node architecture.
  • After joining the master from a RPI-base node, the wrong kube-proxy is installed by the nodes. It seems the kube-proxy docker image used by the master is used and there is no way to override this. We used an ugly hack, “retagging” to fix this for right now.

If you’re OK with the above workarounds, it is though quite easy and fast to setup a cluster. So let’s dive right in.

Setting up the Kubernetes Master

First, install the necessary binaries for creating and managing a Kubernetes cluster. We’re using Arch Linux on our server, you obviously need to run apt-get install on Debian-based systems. So on Arch, just run:

yaourt -S {kubectl,kubelet,kubeadm,kubernetes-cni}-bin --needed --noconfirm

If you have previously played with setting up a cluster, make sure you really start from scratch. The next steps are optional, but if you want to be 100% sure that your system is fresh, then execute these commands to remove all remainders of previous installations:

rm -rf .kube
sudo -i
rm /var/lib/cni/networks/cbr0/*
kubeadm reset

We can now init our master with the kubeadm command. As explained above, the only network that worked for us was Flannel. It means that kubeadm init has to be run with the –pod-network-cidr flag. It essentially specifies the subnet that Flannel will later use to get IPs for Pods etc from.

Run this to initiate the master and prepare it for the Flannel network:

sudo kubeadm init --pod-network-cidr=10.244.0.0/16

This will take a while. Once the init finishes, copy the join URL that you will later need to join the nodes to the master.

Next, run these commands (kubeadm also tells you in the standard output):

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

Finally, we need to install the Flannel network. The trick here is to use a special YAML file which we forked from the official Flannel YAML.

We had to

  • copy the DaemonSet with metadata.name: kube-flannel-ds.
    change the metadata.name of the 2 DaemonSets to kube-flannel-ds-amd64 and kube-flannel-ds-arm
  • in the DaemonSet kube-flannel-ds-arm, replace the 2 occurencies of amd64 with arm

Here’s a gist with the compelte file. It’s based on the official Flannel config file, with the above mentioned changes:

You need to copy the flannel-multiarch.yaml file onto your server and apply it to the Kubernetes cluster:

kubectl apply -f flannel_multiarch.yaml

This concludes the master setup.

Joining RPI-based nodes to your Kubernetes cluster

To join the RPI-based nodes, all it takes is the join URL that kubeadm init outputted during the init phase. The problem is, that the kube-proxy docker image pulled and run is not built for arm. We looked into the code of kubeadm and believe that there is a bug with determining the kube-proxy image. It’s not a multi-arch docker image nor does it try to pull the arm-flavored direct docker image. So we need to fix that.

For each PI, we need to perform these steps:

  • flash Raspbian Stretch Lite onto a SD card, change password and setup SSH
  • assign a unique hostname and static ip addres
  • install docker
  • disable swap
  • fix the cgroup settings and
  • install kubeadm to issue the join cluster command.

Luckily, with a bit of preparation in a text-editor, these steps are executed quite quickly.

First, let’s setup the Raspberry PIs. For the PIs we used Raspbian Stretch Lite, as it is well supported in the RPI community. Install Raspbian Stretch Lite, e.g. 2018-04-18 from the download page. You can burn the image onto a good SD card using etcher.io.

Boot up the PIs one after each other, then change password and ssh settings as you desire.

We need to set unique hostnames (we recommend ‘node-X’) and assign static ip addresses. Here, we specify a few environment variables and later use these variables. You can prepare the snippets in a text editor and then copy the same chunks of commands over to your PIs:

hostname=node-1
ip=192.168.178.300
dns=192.168.178.1

Here, we change the hostname:

sudo hostnamectl --transient set-hostname $hostname
sudo hostnamectl --static set-hostname $hostname
sudo hostnamectl --pretty set-hostname $hostname
sudo sed -i s/raspberrypi/$hostname/g /etc/hosts

And we assign the static IP (if you have an IP address via eth0, just use ‘ip addr’ and use this IP address):

sudo cat <<EOT >> /etc/dhcpcd.conf
interface eth0
static ip_address=$ip/24
static routers=$dns
static domain_name_servers=$dns
EOT

Now, reboot the system:

sudo reboot

Once you logged back into the PI, we install docker. Kubenetes will output a warning for the latest docker version that we use here, but it seems to work fine.

curl -sSL get.docker.com | sh && \
 sudo usermod pi -aG docker

The Kubernetes documentation mentions that swap needs to be disabled, so we thought this is a great idea.

sudo dphys-swapfile swapoff && \
 sudo dphys-swapfile uninstall && \
 sudo update-rc.d dphys-swapfile remove

Kubernetes requires cgroup support. This can be enabled in the /boot/cmdline.txt. These three lines fix the default cmdline.txt:

sudo cp /boot/cmdline.txt /boot/cmdline_backup.txt
orig="$(head -n1 /boot/cmdline.txt) cgroup_enable=cpuset cgroup_enable=memory"
echo $orig | sudo tee /boot/cmdline.txt

And FINALLY… we add the kubernets repo and install kubeadm:

curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add - && \
 echo "deb http://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list && \
 sudo apt-get update -q && \
 sudo apt-get install -qy kubeadm

Almost done, BEFORE joining the cluster, we need to apply the ugly image retagging hack. We essentially download the arm32 kube-proxy image and retag it to look like the amd64 image that kubeadm will try to install. As we pulled and retagged the right image, the image will not be downloaded over the network and we’re good.

To figure out the version, we need to ssh into the master and figure out what version we’re on. If you’re sure, you can skip this step and change the commands accordingly. But to make sure, log in to master (new terminal) and run:

docker image ls

Take note of the kube-proxy docker image and its version. In our case it is v1.10.2.

Back on the PIs, pull the right arm image and then retag it to make it look like the amd64 image:

docker pull k8s.gcr.io/kube-proxy-arm:v1.10.2
docker tag k8s.gcr.io/kube-proxy-arm:v1.10.2 k8s.gcr.io/kube-proxy-amd64:v1.10.2

Now, you’re node is ready to join the master. Use the join url that you copied previously or generate and create a new join token. This will look similar to this (run on each node):

sudo kubeadm join 192.168.178.65:6443 --token speworks.nquhxh32ad3dfw9d --discovery-token-ca-cert-hash sha256:29515464342ee869647e655af97309accfd54746d2f6d4c9360f197204b4d9a2

 

Congratulations, you should be able to run

kubectl get nodes

on your master and see Ready nodes after a while.

pi@master:~ $ kubectl get nodes
NAME      STATUS    ROLES     AGE       VERSION
master    Ready     master    14h       v1.10.2
node-1    Ready     <none>    13h       v1.10.2
node-2    Ready     <none>    13h       v1.10.2
node-3    Ready     <none>    13h       v1.10.2

 

 

 

Leave a Reply

%d bloggers like this: