Kubernetes/Cluster: Difference between revisions

From Omnia
Jump to navigation Jump to search
No edit summary
 
(47 intermediate revisions by the same user not shown)
Line 1: Line 1:
<!-- == Subpage Table of Contents == -->
'''Subpage Table of Contents'''
{{Special:PrefixIndex/{{PAGENAME}}/}}
<br/>
== Create Cluster ==
== Create Cluster ==


Line 26: Line 33:
  sudo apt-mark hold kubelet kubeadm kubectl
  sudo apt-mark hold kubelet kubeadm kubectl


== Setup Docker Runtime with Shim ==
== Install Container Runtime ==
 
=== Install Containerd ===
 
apt install containerd
 
mkdir /etc/containerd
containerd config default | tee /etc/containerd/config.toml > /dev/null
 
sed -i 's/SystemdCgroup = false/SystemdCgroup = true/' /etc/containerd/config.toml
 
systemctl daemon-reload
systemctl start containerd
 
fix ref: [https://github.com/kubernetes/kubernetes/issues/112622]
 
Note to view the images and containers:
crictl images
crictl ps -a
 
sudo ctr ns ls
 
sudo ctr -n k8s.io images ls
sudo ctr -n k8s.io containers ls  # or 'c ls'
 
=== OR Setup Docker Runtime with Shim ===


Note: Technically Docker seems to have had a falling out with Kubernetes, but as it is still popular, you can still use it with cri-dockerd shim...
Note: Technically Docker seems to have had a falling out with Kubernetes, but as it is still popular, you can still use it with cri-dockerd shim...
Line 33: Line 65:
  apt install docker.io
  apt install docker.io


Install shim: [https://github.com/Mirantis/cri-dockerd]
See [[Kubernetes/Cluster#Install_Container_Runtime]]
wget https://github.com/Mirantis/cri-dockerd/releases/download/v0.3.8/cri-dockerd-0.3.8.amd64.tgz
tar xvf cri-dockerd-0.3.8.amd64.tgz
sudo mv ./cri-dockerd /usr/local/bin/
ln -s /usr/local/bin/cri-dockerd /usr/bin/cri-dockerd


== Initalize Cluster ==
== Disable swap on all the Nodes ==


Run pre-flight checks:
WARNING: Kubernetes doesn't like Swap. Disable it.  'kubeadm init' only says warning, but kubelet will fail to start if swap is installed!
  kubeadm init phase preflight


Pre pull images:
Only warning here, but really it should be fatal!
kubeadm config images pull
<pre>
root@kube1:~# kubeadm init
[init] Using Kubernetes version: v1.29.0
[preflight] Running pre-flight checks
        [WARNING Swap]: swap is supported for cgroup v2 only; the NodeSwap feature gate of the kubelet is beta but disabled by default
</pre>


Fake initalize cluster:
List swap:
  kubeadm init --dry-run
  swapon -s


Initalize cluster:
Disable swap:
kubeadm init
<pre>
sudo swapoff -a
(crontab -l 2>/dev/null; echo "@reboot /sbin/swapoff -a") | crontab - || true
</pre>


== Enable iptables Bridged Traffic on all the Nodes ==
== Enable iptables Bridged Traffic on all the Nodes ==
Note: installing docker.io seems to do most of these for you (so if you go that route, you can skip these manual steps)...


<pre>
<pre>
Line 61: Line 98:
EOF
EOF


sudo modprobe overlay
sudo modprobe overlay ; sudo modprobe br_netfilter
sudo modprobe br_netfilter


# sysctl params required by setup, params persist across reboots
# sysctl params required by setup, params persist across reboots
Line 75: Line 111:
</pre>
</pre>


== Disable swap on all the Nodes ==
== Initalize Cluster ==
 
Run pre-flight checks:
sudo kubeadm init phase preflight
 
Pre pull images:
sudo kubeadm config images pull
 
Fake initalize cluster:
sudo kubeadm init --dry-run --pod-network-cidr=10.244.0.0/16
 
''Note: the "--pod-network-cidr=10.244.0.0/16" is needed for Flannel.  Silly Flannel.''
 
Initalize cluster:
sudo kubeadm init --pod-network-cidr=10.244.0.0/16
 
=== Initalize with Docker ===
 
Run pre-flight checks:
sudo kubeadm init phase preflight --cri-socket=unix:///var/run/cri-dockerd.sock
 
Pre pull images:
sudo kubeadm config images pull --cri-socket=unix:///var/run/cri-dockerd.sock
 
Fake initalize cluster with Docker:
sudo kubeadm init --dry-run --pod-network-cidr=10.244.0.0/16 --cri-socket=unix:///var/run/cri-dockerd.sock
 
''Note: the "--pod-network-cidr=10.244.0.0/16" is needed for Flannel.  Silly Flannel.''
 
Or Initalize cluster with Docker:
sudo kubeadm init --pod-network-cidr=10.244.0.0/16 --cri-socket=unix:///var/run/cri-dockerd.sock
 
=== Initalize with Redundant Control Plane ===
 
sudo kubeadm init --pod-network-cidr=10.244.0.0/16 --control-plane-endpoint=192.x.x.x:6443 --upload-certs  --cri-socket=unix:///var/run/cri-dockerd.sock
 
With keepalived on same host:
sudo kubeadm init --pod-network-cidr=10.244.0.0/16 --control-plane-endpoint=192.x.x.x:26443 --upload-certs  --cri-socket=unix:///var/run/cri-dockerd.sock
 
== Initalized Success ==


Kubernetes doesn't like Swap.  Disable it.
<pre>
<pre>
root@kube1:~# kubeadm init
Your Kubernetes control-plane has initialized successfully!
[init] Using Kubernetes version: v1.29.0
 
[preflight] Running pre-flight checks
To start using your cluster, you need to run the following as a regular user:
         [WARNING Swap]: swap is supported for cgroup v2 only; the NodeSwap feature gate of the kubelet is beta but disabled by default
 
  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config
 
Alternatively, if you are the root user, you can run:
 
  export KUBECONFIG=/etc/kubernetes/admin.conf
 
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/
 
Then you can join any number of worker nodes by running the following on each as root:
 
kubeadm join 192.x.x.x:6443 --token exxxxxxx.xxxxxxxxxxxx \
         --discovery-token-ca-cert-hash sha256:4xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
</pre>
</pre>


Disable swap:
 
<pre>
WARNING WARNING WARNING - before joining another node, make sure to add your pod network!
sudo swapoff -a
 
(crontab -l 2>/dev/null; echo "@reboot /sbin/swapoff -a") | crontab - || true
== Get Join Command ==
</pre>
 
kubeadm init phase upload-certs --upload-certs
 
kubeadm token create --print-join-command
 
Example:
kubeadm join 192.x.x.x:6443 --token xxx.xxxxxxx --discovery-token-ca-cert-hash sha256:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
 
Example with Docker:
kubeadm join 192.x.x.x:6443 --token xxx.xxxxxxx --discovery-token-ca-cert-hash sha256:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx --cri-socket=unix:///var/run/cri-dockerd.sock
 
== Initalize Pod Network - Flannel ==
 
export KUBECONFIG=/etc/kubernetes/admin.conf
 
k get pods -o wide --all-namespaces
 
kubectl apply -f https://github.com/flannel-io/flannel/releases/latest/download/kube-flannel.yml
 
kubectl get pods --all-namespaces
# or just
kubectl get pods -n kube-flannel
 
Note, if you need to retry, you can delete, then retry:
kubectl delete -f https://github.com/flannel-io/flannel/releases/latest/download/kube-flannel.yml
 
Note, flannel expects the CIDR to be 10.244.0.0/16.  If you have it set to something else, you will need to modify the manifest.  To check what your cluster is configured for, or if you even specified the the cidr during cluster creation (required for Flannel, if it returns nothing that is a problem): <ref>https://devops.stackexchange.com/questions/5898/how-to-get-kubernetes-pod-network-cidr</ref>
kubectl cluster-info dump | grep cluster-cidr
 
reference:
GitHub - flannel-io/flannel: flannel is a network fabric for containers, designed for Kubernetes
https://github.com/flannel-io/flannel#deploying-flannel-manually
 
== Join Worker Nodes ==
 
1. Disable Swap
 
2. Configure
 
kubeadm join 192.x.x.x:6443 --token exxxxxxx.xxxxxxxxxxxx \
        --discovery-token-ca-cert-hash sha256:4xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
 
== Test a few pods ==
 
kubectl run shell1 --image ubuntu sleep infinity
kubectl run shell2 --image ubuntu sleep infinity
kubectl run shell3 --image ubuntu sleep infinity
kubectl run shell4 --image ubuntu sleep infinity
kubectl run shell5 --image ubuntu sleep infinity
 
== Delete Node ==
 
How to delete a node from Kubernetes <ref>How to gracefully remove a node from Kubernetes? - Stack Overflow - https://stackoverflow.com/questions/35757620/how-to-gracefully-remove-a-node-from-kubernetes</ref>
 
kubectl drain <node-name>
 
Might have to ignore some deamonsets:
kubectl drain <node-name> --ignore-daemonsets --delete-local-data
 
Delete node:
kubectl delete node <node-name>
 
Reset node locally: (run locally)
kubeadm reset


== Install Docker ==
== Install Docker ==


ref [https://docs.docker.com/compose/install/]
Overview of installing Docker Compose | Docker Docs
https://docs.docker.com/compose/install/


=== Setup Repo ===
=== Setup Repo ===
Line 109: Line 263:
   sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
   sudo tee /etc/apt/sources.list.d/docker.list > /dev/null


ref [https://docs.docker.com/engine/install/ubuntu/#set-up-the-repository]
ref <ref>Install Docker Engine on Ubuntu - https://docs.docker.com/engine/install/ubuntu/#set-up-the-repository</ref>


=== Install Docker Enginer ===
=== Install Docker Enginer ===
Line 127: Line 281:
== Install Container Runtime ==
== Install Container Runtime ==


Install cri-dockerd
Install cri-dockerd binary:
 
  wget https://github.com/Mirantis/cri-dockerd/releases/download/v0.3.8/cri-dockerd-0.3.8.amd64.tgz
  wget https://github.com/Mirantis/cri-dockerd/releases/download/v0.3.8/cri-dockerd-0.3.8.amd64.tgz
  tar xvf cri-dockerd-0.3.8.amd64.tgz
  tar xvf cri-dockerd-0.3.8.amd64.tgz
  sudo mv ./cri-dockerd /usr/local/bin/
  sudo mv cri-dockerd/cri-dockerd /usr/local/bin/
  ln -s /usr/local/bin/cri-dockerd /usr/bin/cri-dockerd  
rmdir cri-dockerd/
  ln -s /usr/local/bin/cri-dockerd /usr/bin/cri-dockerd


<s>
  wget https://github.com/Mirantis/cri-dockerd/releases/download/v0.2.0/cri-dockerd-v0.2.0-linux-amd64.tar.gz
  wget https://github.com/Mirantis/cri-dockerd/releases/download/v0.2.0/cri-dockerd-v0.2.0-linux-amd64.tar.gz
  tar xvf cri-dockerd-v0.2.0-linux-amd64.tar.gz
  tar xvf cri-dockerd-v0.2.0-linux-amd64.tar.gz
  sudo mv ./cri-dockerd /usr/local/bin/
  sudo mv ./cri-dockerd /usr/local/bin/
  ln -s /usr/local/bin/cri-dockerd /usr/bin/cri-dockerd  
  ln -s /usr/local/bin/cri-dockerd /usr/bin/cri-dockerd  
</s>


  cri-dockerd --help
  cri-dockerd --help


Install service:
  wget https://raw.githubusercontent.com/Mirantis/cri-dockerd/master/packaging/systemd/cri-docker.service
  wget https://raw.githubusercontent.com/Mirantis/cri-dockerd/master/packaging/systemd/cri-docker.service
  wget https://raw.githubusercontent.com/Mirantis/cri-dockerd/master/packaging/systemd/cri-docker.socket
  wget https://raw.githubusercontent.com/Mirantis/cri-dockerd/master/packaging/systemd/cri-docker.socket
Line 234: Line 391:


  sudo kubeadm init --cri-socket=unix:///var/run/cri-dockerd.sock
  sudo kubeadm init --cri-socket=unix:///var/run/cri-dockerd.sock
== Tear Down ==
If you need to reset a node (even controller) locally:
kubeadm reset
If with docker:
kubeadm reset --cri-socket=unix:///var/run/cri-dockerd.sock
== Useful Debug ==
crictl images
crictl ps -a
sudo ctr ns ls
sudo ctr -n k8s.io images ls
sudo ctr -n k8s.io containers ls  # or 'c ls'
service kubelet status
journalctl -u kubelet


== Notes ==
== Notes ==
Line 268: Line 444:


== keywords ==
== keywords ==
== References ==
<references />

Latest revision as of 22:04, 22 December 2023


Subpage Table of Contents


Create Cluster

References: 

How To Setup Kubernetes Cluster Using Kubeadm - Easy Guide
https://devopscube.com/setup-kubernetes-cluster-kubeadm/

Creating a cluster with kubeadm | Kubernetes
https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/

Install kubeadm

Reference: 

Installing kubeadm | Kubernetes
https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/

Install dependencies: 

sudo apt-get update
sudo apt-get install -y apt-transport-https ca-certificates curl gpg
curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.29/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.29/deb/ /' | sudo tee /etc/apt/sources.list.d/kubernetes.list

Install kubelet kubeadm kubectl 

sudo apt-get update
sudo apt-get install -y kubelet kubeadm kubectl
sudo apt-mark hold kubelet kubeadm kubectl

Install Container Runtime

Install Containerd

apt install containerd

mkdir /etc/containerd
containerd config default | tee /etc/containerd/config.toml > /dev/null

sed -i 's/SystemdCgroup = false/SystemdCgroup = true/' /etc/containerd/config.toml

systemctl daemon-reload
systemctl start containerd

fix ref: [1]

Note to view the images and containers: 

crictl images
crictl ps -a

sudo ctr ns ls

sudo ctr -n k8s.io images ls
sudo ctr -n k8s.io containers ls  # or 'c ls'

OR Setup Docker Runtime with Shim

Note: Technically Docker seems to have had a falling out with Kubernetes, but as it is still popular, you can still use it with cri-dockerd shim...

Install Docker: 

apt install docker.io

See Kubernetes/Cluster#Install_Container_Runtime

Disable swap on all the Nodes

WARNING: Kubernetes doesn't like Swap. Disable it. 'kubeadm init' only says warning, but kubelet will fail to start if swap is installed!

Only warning here, but really it should be fatal! 

root@kube1:~# kubeadm init
[init] Using Kubernetes version: v1.29.0
[preflight] Running pre-flight checks
        [WARNING Swap]: swap is supported for cgroup v2 only; the NodeSwap feature gate of the kubelet is beta but disabled by default

List swap: 

swapon -s

Disable swap: 

sudo swapoff -a
(crontab -l 2>/dev/null; echo "@reboot /sbin/swapoff -a") | crontab - || true

Enable iptables Bridged Traffic on all the Nodes

Note: installing docker.io seems to do most of these for you (so if you go that route, you can skip these manual steps)... 

cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
overlay
br_netfilter
EOF

sudo modprobe overlay ; sudo modprobe br_netfilter

# sysctl params required by setup, params persist across reboots
cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables  = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward                 = 1
EOF

# Apply sysctl params without reboot
sudo sysctl --system

Initalize Cluster

Run pre-flight checks: 

sudo kubeadm init phase preflight

Pre pull images: 

sudo kubeadm config images pull

Fake initalize cluster: 

sudo kubeadm init --dry-run --pod-network-cidr=10.244.0.0/16

Note: the "--pod-network-cidr=10.244.0.0/16" is needed for Flannel. Silly Flannel.

Initalize cluster: 

sudo kubeadm init --pod-network-cidr=10.244.0.0/16

Initalize with Docker

Run pre-flight checks: 

sudo kubeadm init phase preflight --cri-socket=unix:///var/run/cri-dockerd.sock

Pre pull images: 

sudo kubeadm config images pull --cri-socket=unix:///var/run/cri-dockerd.sock

Fake initalize cluster with Docker: 

sudo kubeadm init --dry-run --pod-network-cidr=10.244.0.0/16 --cri-socket=unix:///var/run/cri-dockerd.sock

Note: the "--pod-network-cidr=10.244.0.0/16" is needed for Flannel. Silly Flannel.

Or Initalize cluster with Docker: 

sudo kubeadm init --pod-network-cidr=10.244.0.0/16 --cri-socket=unix:///var/run/cri-dockerd.sock

Initalize with Redundant Control Plane

sudo kubeadm init --pod-network-cidr=10.244.0.0/16 --control-plane-endpoint=192.x.x.x:6443 --upload-certs  --cri-socket=unix:///var/run/cri-dockerd.sock

With keepalived on same host: 

sudo kubeadm init --pod-network-cidr=10.244.0.0/16 --control-plane-endpoint=192.x.x.x:26443 --upload-certs  --cri-socket=unix:///var/run/cri-dockerd.sock

Initalized Success

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

  export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 192.x.x.x:6443 --token exxxxxxx.xxxxxxxxxxxx \
        --discovery-token-ca-cert-hash sha256:4xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx


WARNING WARNING WARNING - before joining another node, make sure to add your pod network!

Get Join Command

kubeadm init phase upload-certs --upload-certs

kubeadm token create --print-join-command

Example: 

kubeadm join 192.x.x.x:6443 --token xxx.xxxxxxx --discovery-token-ca-cert-hash sha256:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

Example with Docker: 

kubeadm join 192.x.x.x:6443 --token xxx.xxxxxxx --discovery-token-ca-cert-hash sha256:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx --cri-socket=unix:///var/run/cri-dockerd.sock

Initalize Pod Network - Flannel

export KUBECONFIG=/etc/kubernetes/admin.conf

k get pods -o wide --all-namespaces

kubectl apply -f https://github.com/flannel-io/flannel/releases/latest/download/kube-flannel.yml

kubectl get pods --all-namespaces
# or just
kubectl get pods -n kube-flannel

Note, if you need to retry, you can delete, then retry: 

kubectl delete -f https://github.com/flannel-io/flannel/releases/latest/download/kube-flannel.yml

Note, flannel expects the CIDR to be 10.244.0.0/16. If you have it set to something else, you will need to modify the manifest. To check what your cluster is configured for, or if you even specified the the cidr during cluster creation (required for Flannel, if it returns nothing that is a problem): [1] 

kubectl cluster-info dump | grep cluster-cidr

reference: 

GitHub - flannel-io/flannel: flannel is a network fabric for containers, designed for Kubernetes
https://github.com/flannel-io/flannel#deploying-flannel-manually

Join Worker Nodes

1. Disable Swap

2. Configure 

kubeadm join 192.x.x.x:6443 --token exxxxxxx.xxxxxxxxxxxx \
       --discovery-token-ca-cert-hash sha256:4xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

Test a few pods

kubectl run shell1 --image ubuntu sleep infinity
kubectl run shell2 --image ubuntu sleep infinity
kubectl run shell3 --image ubuntu sleep infinity
kubectl run shell4 --image ubuntu sleep infinity
kubectl run shell5 --image ubuntu sleep infinity

Delete Node

How to delete a node from Kubernetes [2] 

kubectl drain <node-name>

Might have to ignore some deamonsets: 

kubectl drain <node-name> --ignore-daemonsets --delete-local-data

Delete node: 

kubectl delete node <node-name>

Reset node locally: (run locally) 

kubeadm reset

Install Docker

Overview of installing Docker Compose | Docker Docs
https://docs.docker.com/compose/install/

Setup Repo

sudo apt-get update
sudo apt-get install -y ca-certificates curl gnupg

sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
sudo chmod a+r /etc/apt/keyrings/docker.gpg

echo \
 "deb [arch="$(dpkg --print-architecture)" signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
 "$(. /etc/os-release && echo "$VERSION_CODENAME")" stable" | \
 sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

ref [3]

Install Docker Enginer

sudo apt-get update
sudo apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

ref [2]

Install docker and plugin

sudo apt-get update
sudo apt-get install docker-compose-plugin

ref [3]

Install Container Runtime

Install cri-dockerd binary: 

wget https://github.com/Mirantis/cri-dockerd/releases/download/v0.3.8/cri-dockerd-0.3.8.amd64.tgz
tar xvf cri-dockerd-0.3.8.amd64.tgz
sudo mv cri-dockerd/cri-dockerd /usr/local/bin/
rmdir cri-dockerd/
ln -s /usr/local/bin/cri-dockerd /usr/bin/cri-dockerd


wget https://github.com/Mirantis/cri-dockerd/releases/download/v0.2.0/cri-dockerd-v0.2.0-linux-amd64.tar.gz
tar xvf cri-dockerd-v0.2.0-linux-amd64.tar.gz
sudo mv ./cri-dockerd /usr/local/bin/
ln -s /usr/local/bin/cri-dockerd /usr/bin/cri-dockerd 


cri-dockerd --help

Install service: 

wget https://raw.githubusercontent.com/Mirantis/cri-dockerd/master/packaging/systemd/cri-docker.service
wget https://raw.githubusercontent.com/Mirantis/cri-dockerd/master/packaging/systemd/cri-docker.socket
sudo mv cri-docker.socket cri-docker.service /etc/systemd/system/
sudo sed -i -e 's,/usr/bin/cri-dockerd,/usr/local/bin/cri-dockerd,' /etc/systemd/system/cri-docker.service

systemctl daemon-reload
systemctl enable cri-docker.service
systemctl enable --now cri-docker.socket

systemctl start cri-docker.service
systemctl start cri-docker.socket

systemctl status cri-docker.service
systemctl status cri-docker.socket

journalctl -u cri-docker.service
journalctl -u cri-docker.socket

crictl info
crictl --runtime-endpoint unix:///var/run/cri-dockerd.sock info

/etc/crictl.yaml

runtime-endpoint: unix:///var/run/containerd/containerd.sock
image-endpoint: unix:///var/run/containerd/containerd.sock
timeout: 10
debug: true

runtime-endpoint: unix:///var/run/cri-dockerd.sock
image-endpoint: unix:///var/run/cri-dockerd.sock
timeout: 10
debug: true
  1. /var/lib/kubelet/kubeadm-flags.env

ref [4]

Old: 

git clone https://github.com/Mirantis/cri-dockerd.git

cd cri-dockerd
mkdir bin
VERSION=$((git describe --abbrev=0 --tags | sed -e 's/v//') || echo $(cat VERSION)-$(git log -1 --pretty='%h')) PRERELEASE=$(grep -q dev <<< "${VERSION}" && echo "pre" || echo "") REVISION=$(git log -1 --pretty='%h')
go build -ldflags="-X github.com/Mirantis/cri-dockerd/version.Version='$VERSION}' -X github.com/Mirantis/cri-dockerd/version.PreRelease='$PRERELEASE' -X github.com/Mirantis/cri-dockerd/version.BuildTime='$BUILD_DATE' -X github.com/Mirantis/cri-dockerd/version.GitCommit='$REVISION'" -o cri-dockerd

...

ref [5]

Install Kubeadm & Kubelet & Kubectl on all Nodes

sudo apt-get update && \
sudo apt-get install -y apt-transport-https ca-certificates curl && \
sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg

#echo "deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list
echo "deb https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list

sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys B53DC80D13EDEF05

sudo apt-get update -y && \
sudo apt-get install -y kubelet kubeadm kubectl && \
sudo apt-mark hold kubelet kubeadm kubectl



sudo apt-get update && \
sudo apt-get install -y kubelet=1.26.5-00 kubeadm=1.26.5-00 kubectl=1.26.5-00 docker.io && \
sudo apt-mark hold kubelet kubeadm kubectl docker.io

sudo mkdir /etc/containerd && \
sudo containerd config default > /etc/containerd/config.toml && \
sudo sed -i 's/            SystemdCgroup = false/            SystemdCgroup = true/' /etc/containerd/config.toml && \
sudo systemctl restart containerd && \
sudo systemctl restart kubelet

Initialize Kubeadm On Master Node To Setup Control Plane

sudo kubeadm init

Sockets found: 

unix:///var/run/containerd/containerd.sock
unix:///var/run/cri-dockerd.sock

sudo kubeadm init --cri-socket=unix:///var/run/cri-dockerd.sock

Tear Down

If you need to reset a node (even controller) locally: 

kubeadm reset

If with docker: 

kubeadm reset --cri-socket=unix:///var/run/cri-dockerd.sock

Useful Debug

crictl images
crictl ps -a
sudo ctr ns ls
sudo ctr -n k8s.io images ls
sudo ctr -n k8s.io containers ls  # or 'c ls'

service kubelet status
journalctl -u kubelet

Notes

How To Setup Kubernetes Cluster Using Kubeadm - Easy Guide
https://devopscube.com/setup-kubernetes-cluster-kubeadm/

How to Set Up Three Node Kubernetes Cluster on Ubuntu
https://k21academy.com/docker-kubernetes/three-node-kubernetes-cluster/

5 Fastest Ways to Install a k8s Cluster - Your DevOps Mentor
https://yourdevopsmentor.com/blog/5-ways-to-install-a-kubernetes-cluster/

Kubernetes Cluster setup on Ubuntu 22.04 using kubeadm with Calico, By Sir Babar Zahoor
https://www.linkedin.com/pulse/kubernetes-cluster-setup-ubuntu-2204-using-kubeadm-calico-md-sajjad

How to Set Up Three Node Kubernetes Cluster on Ubuntu 20.04
https://www.atlantic.net/dedicated-server-hosting/how-to-set-up-three-node-kubernetes-cluster-on-ubuntu-20-04/

Using Kubeadm to Create a Cluster | Airplane
https://www.airplane.dev/blog/kubeadm

Install Mirantis cri-dockerd as Docker Engine shim for Kubernetes | ComputingForGeeks
https://computingforgeeks.com/install-mirantis-cri-dockerd-as-docker-engine-shim-for-kubernetes/

Install Kubernetes Cluster on Ubuntu 22.04 with kubeadm | ComputingForGeeks
https://computingforgeeks.com/install-kubernetes-cluster-ubuntu-jammy/

Install Kubernetes 1.26 on Ubuntu 20.04 or 22.04 LTS | by Akriotis Kyriakos | Medium
https://akyriako.medium.com/install-kubernetes-on-ubuntu-20-04-f1791e8cf799

How to Install Kubernetes Cluster on Ubuntu 22.04
https://www.linuxtechi.com/install-kubernetes-on-ubuntu-22-04/

keywords

References

  1. ↑ https://devops.stackexchange.com/questions/5898/how-to-get-kubernetes-pod-network-cidr
  2. ↑ How to gracefully remove a node from Kubernetes? - Stack Overflow - https://stackoverflow.com/questions/35757620/how-to-gracefully-remove-a-node-from-kubernetes
  3. ↑ Install Docker Engine on Ubuntu - https://docs.docker.com/engine/install/ubuntu/#set-up-the-repository
Retrieved from "https://aznot.com/index.php?title=Kubernetes/Cluster&oldid=6636"