Kubernetes/Cluster

Create Cluster

References:

How To Setup Kubernetes Cluster Using Kubeadm - Easy Guide
https://devopscube.com/setup-kubernetes-cluster-kubeadm/

Creating a cluster with kubeadm | Kubernetes
https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/

Install kubeadm

Reference:

Installing kubeadm | Kubernetes
https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/

Install dependencies:

sudo apt-get update
sudo apt-get install -y apt-transport-https ca-certificates curl gpg
curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.29/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.29/deb/ /' | sudo tee /etc/apt/sources.list.d/kubernetes.list

Install kubelet kubeadm kubectl

sudo apt-get update
sudo apt-get install -y kubelet kubeadm kubectl
sudo apt-mark hold kubelet kubeadm kubectl

Install Container Runtime

Install Containerd

apt install containerd

mkdir /etc/containerd
containerd config default | tee /etc/containerd/config.toml > /dev/null

sed -i 's/SystemdCgroup = false/SystemdCgroup = true/' /etc/containerd/config.toml

systemctl daemon-reload
systemctl start containerd

fix ref: [1]

Note to view the images and containers:

crictl images
crictl ps -a

sudo ctr ns ls

sudo ctr -n k8s.io images ls
sudo ctr -n k8s.io containers ls  # or 'c ls'

OR Setup Docker Runtime with Shim

Note: Technically Docker seems to have had a falling out with Kubernetes, but as it is still popular, you can still use it with cri-dockerd shim...

Install Docker:

apt install docker.io

See Kubernetes/Cluster#Install_Container_Runtime

Disable swap on all the Nodes

WARNING: Kubernetes doesn't like Swap. Disable it. 'kubeadm init' only says warning, but kubelet will fail to start if swap is installed!

Only warning here, but really it should be fatal!

root@kube1:~# kubeadm init
[init] Using Kubernetes version: v1.29.0
[preflight] Running pre-flight checks
        [WARNING Swap]: swap is supported for cgroup v2 only; the NodeSwap feature gate of the kubelet is beta but disabled by default

List swap:

swapon -s

Disable swap:

sudo swapoff -a
(crontab -l 2>/dev/null; echo "@reboot /sbin/swapoff -a") | crontab - || true

Enable iptables Bridged Traffic on all the Nodes

Note: installing docker.io seems to do most of these for you (so if you go that route, you can skip these manual steps)...

cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
overlay
br_netfilter
EOF

sudo modprobe overlay ; sudo modprobe br_netfilter

# sysctl params required by setup, params persist across reboots
cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables  = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward                 = 1
EOF

# Apply sysctl params without reboot
sudo sysctl --system

Initalize Cluster

Run pre-flight checks:

sudo kubeadm init phase preflight

Pre pull images:

sudo kubeadm config images pull

Fake initalize cluster:

sudo kubeadm init --dry-run --pod-network-cidr=10.244.0.0/16

Note: the "--pod-network-cidr=10.244.0.0/16" is needed for Flannel. Silly Flannel.

Initalize cluster:

sudo kubeadm init --pod-network-cidr=10.244.0.0/16

Initalize with Docker

Run pre-flight checks:

sudo kubeadm init phase preflight --cri-socket=unix:///var/run/cri-dockerd.sock

Pre pull images:

sudo kubeadm config images pull --cri-socket=unix:///var/run/cri-dockerd.sock

Fake initalize cluster with Docker:

sudo kubeadm init --dry-run --pod-network-cidr=10.244.0.0/16 --cri-socket=unix:///var/run/cri-dockerd.sock

Note: the "--pod-network-cidr=10.244.0.0/16" is needed for Flannel. Silly Flannel.

Or Initalize cluster with Docker:

sudo kubeadm init --pod-network-cidr=10.244.0.0/16 --cri-socket=unix:///var/run/cri-dockerd.sock

Initalized Success

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

  export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 192.x.x.x:6443 --token exxxxxxx.xxxxxxxxxxxx \
        --discovery-token-ca-cert-hash sha256:4xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

WARNING WARNING WARNING - before joining another node, make sure to add your pod network!

Get Join Command

kubeadm token create --print-join-command

Example:

kubeadm join 192.x.x.x:6443 --token xxx.xxxxxxx --discovery-token-ca-cert-hash sha256:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

Example with Docker:

kubeadm join 192.x.x.x:6443 --token xxx.xxxxxxx --discovery-token-ca-cert-hash sha256:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx --cri-socket=unix:///var/run/cri-dockerd.sock

Initalize Pod Network - Flannel

export KUBECONFIG=/etc/kubernetes/admin.conf

k get pods -o wide --all-namespaces

kubectl apply -f https://github.com/flannel-io/flannel/releases/latest/download/kube-flannel.yml

kubectl get pods --all-namespaces
# or just
kubectl get pods -n kube-flannel

Note, if you need to retry, you can delete, then retry:

kubectl delete -f https://github.com/flannel-io/flannel/releases/latest/download/kube-flannel.yml

reference:

GitHub - flannel-io/flannel: flannel is a network fabric for containers, designed for Kubernetes
https://github.com/flannel-io/flannel#deploying-flannel-manually

Join Nodes

1. Disable Swap

2. Configure

kubeadm join 192.x.x.x:6443 --token exxxxxxx.xxxxxxxxxxxx \
       --discovery-token-ca-cert-hash sha256:4xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

Delete Node

How to delete a node from Kubernetes

kubectl drain <node-name>

Might have to ignore some deamonsets:

kubectl drain <node-name> --ignore-daemonsets --delete-local-data

Delete node:

kubectl delete node <node-name>

Reset node locally: (run locally)

kubeadm reset

Install Docker

Overview of installing Docker Compose | Docker Docs
https://docs.docker.com/compose/install/

Setup Repo

sudo apt-get update
sudo apt-get install -y ca-certificates curl gnupg

sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
sudo chmod a+r /etc/apt/keyrings/docker.gpg

echo \
 "deb [arch="$(dpkg --print-architecture)" signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
 "$(. /etc/os-release && echo "$VERSION_CODENAME")" stable" | \
 sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

ref

Install Docker Enginer

sudo apt-get update
sudo apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

ref [2]

Install docker and plugin

sudo apt-get update
sudo apt-get install docker-compose-plugin

ref [3]

Install Container Runtime

Install cri-dockerd binary:

wget https://github.com/Mirantis/cri-dockerd/releases/download/v0.3.8/cri-dockerd-0.3.8.amd64.tgz
tar xvf cri-dockerd-0.3.8.amd64.tgz
sudo mv cri-dockerd/cri-dockerd /usr/local/bin/
rmdir cri-dockerd/
ln -s /usr/local/bin/cri-dockerd /usr/bin/cri-dockerd

wget https://github.com/Mirantis/cri-dockerd/releases/download/v0.2.0/cri-dockerd-v0.2.0-linux-amd64.tar.gz
tar xvf cri-dockerd-v0.2.0-linux-amd64.tar.gz
sudo mv ./cri-dockerd /usr/local/bin/
ln -s /usr/local/bin/cri-dockerd /usr/bin/cri-dockerd 

cri-dockerd --help

Install service:

wget https://raw.githubusercontent.com/Mirantis/cri-dockerd/master/packaging/systemd/cri-docker.service
wget https://raw.githubusercontent.com/Mirantis/cri-dockerd/master/packaging/systemd/cri-docker.socket
sudo mv cri-docker.socket cri-docker.service /etc/systemd/system/
sudo sed -i -e 's,/usr/bin/cri-dockerd,/usr/local/bin/cri-dockerd,' /etc/systemd/system/cri-docker.service

systemctl daemon-reload
systemctl enable cri-docker.service
systemctl enable --now cri-docker.socket

systemctl start cri-docker.service
systemctl start cri-docker.socket

systemctl status cri-docker.service
systemctl status cri-docker.socket

journalctl -u cri-docker.service
journalctl -u cri-docker.socket

crictl info
crictl --runtime-endpoint unix:///var/run/cri-dockerd.sock info

/etc/crictl.yaml

runtime-endpoint: unix:///var/run/containerd/containerd.sock
image-endpoint: unix:///var/run/containerd/containerd.sock
timeout: 10
debug: true

runtime-endpoint: unix:///var/run/cri-dockerd.sock
image-endpoint: unix:///var/run/cri-dockerd.sock
timeout: 10
debug: true

1. /var/lib/kubelet/kubeadm-flags.env

ref [4]

---

Old:

git clone https://github.com/Mirantis/cri-dockerd.git

cd cri-dockerd
mkdir bin
VERSION=$((git describe --abbrev=0 --tags | sed -e 's/v//') || echo $(cat VERSION)-$(git log -1 --pretty='%h')) PRERELEASE=$(grep -q dev <<< "${VERSION}" && echo "pre" || echo "") REVISION=$(git log -1 --pretty='%h')
go build -ldflags="-X github.com/Mirantis/cri-dockerd/version.Version='$VERSION}' -X github.com/Mirantis/cri-dockerd/version.PreRelease='$PRERELEASE' -X github.com/Mirantis/cri-dockerd/version.BuildTime='$BUILD_DATE' -X github.com/Mirantis/cri-dockerd/version.GitCommit='$REVISION'" -o cri-dockerd

...

ref [5]

Install Kubeadm & Kubelet & Kubectl on all Nodes

sudo apt-get update && \
sudo apt-get install -y apt-transport-https ca-certificates curl && \
sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg

#echo "deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list
echo "deb https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list

sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys B53DC80D13EDEF05

sudo apt-get update -y && \
sudo apt-get install -y kubelet kubeadm kubectl && \
sudo apt-mark hold kubelet kubeadm kubectl

sudo apt-get update && \
sudo apt-get install -y kubelet=1.26.5-00 kubeadm=1.26.5-00 kubectl=1.26.5-00 docker.io && \
sudo apt-mark hold kubelet kubeadm kubectl docker.io

sudo mkdir /etc/containerd && \
sudo containerd config default > /etc/containerd/config.toml && \
sudo sed -i 's/            SystemdCgroup = false/            SystemdCgroup = true/' /etc/containerd/config.toml && \
sudo systemctl restart containerd && \
sudo systemctl restart kubelet

Initialize Kubeadm On Master Node To Setup Control Plane

sudo kubeadm init

Sockets found:

unix:///var/run/containerd/containerd.sock
unix:///var/run/cri-dockerd.sock

sudo kubeadm init --cri-socket=unix:///var/run/cri-dockerd.sock

Tear Down Cluster

If you need to try again:

kubeadm reset

Sort of, this doesn't do a great job...

Useful Debug

crictl images
crictl ps -a
sudo ctr ns ls
sudo ctr -n k8s.io images ls
sudo ctr -n k8s.io containers ls  # or 'c ls'

service kubelet status
journalctl -u kubelet

Notes

How To Setup Kubernetes Cluster Using Kubeadm - Easy Guide
https://devopscube.com/setup-kubernetes-cluster-kubeadm/

How to Set Up Three Node Kubernetes Cluster on Ubuntu
https://k21academy.com/docker-kubernetes/three-node-kubernetes-cluster/

5 Fastest Ways to Install a k8s Cluster - Your DevOps Mentor
https://yourdevopsmentor.com/blog/5-ways-to-install-a-kubernetes-cluster/

Kubernetes Cluster setup on Ubuntu 22.04 using kubeadm with Calico, By Sir Babar Zahoor
https://www.linkedin.com/pulse/kubernetes-cluster-setup-ubuntu-2204-using-kubeadm-calico-md-sajjad

How to Set Up Three Node Kubernetes Cluster on Ubuntu 20.04
https://www.atlantic.net/dedicated-server-hosting/how-to-set-up-three-node-kubernetes-cluster-on-ubuntu-20-04/

Using Kubeadm to Create a Cluster | Airplane
https://www.airplane.dev/blog/kubeadm

Install Mirantis cri-dockerd as Docker Engine shim for Kubernetes | ComputingForGeeks
https://computingforgeeks.com/install-mirantis-cri-dockerd-as-docker-engine-shim-for-kubernetes/

Install Kubernetes Cluster on Ubuntu 22.04 with kubeadm | ComputingForGeeks
https://computingforgeeks.com/install-kubernetes-cluster-ubuntu-jammy/

Install Kubernetes 1.26 on Ubuntu 20.04 or 22.04 LTS | by Akriotis Kyriakos | Medium
https://akyriako.medium.com/install-kubernetes-on-ubuntu-20-04-f1791e8cf799

How to Install Kubernetes Cluster on Ubuntu 22.04
https://www.linuxtechi.com/install-kubernetes-on-ubuntu-22-04/