255 lines
8.0 KiB
Markdown
255 lines
8.0 KiB
Markdown
# Install instructions for a 3-Node Kubernetes Cluster
|
||
|
||
## Prerequisites
|
||
Minimal Installed Debian 12/11 \
|
||
2 CPU / vCPU, 2 GB RAM \
|
||
20 GB free disk space \
|
||
Sudo User with Admin rights \
|
||
Stable Internet Connectivity
|
||
|
||
Ensure that each node can communicate with the others via a reliable network connection.
|
||
|
||
## Lab Setup
|
||
For the demonstration, I am using three Debian 12 systems.
|
||
|
||
Master Node (vmcon-master.his.de) – 192.168.202.10 \
|
||
Worker Node 1 (vmcon-node1.his.de) – 192.168.202.11 \
|
||
Worker Node 2 (vmcon-node2.his.de) – 192.168.202.12
|
||
|
||
|
||
# Installation
|
||
|
||
## Set Host Name and Update Hosts File
|
||
Login to each node (master & woker nodes) and set their hostname using hostnamectl command.
|
||
|
||
```bash
|
||
$ hostnamectl set-hostname "vmcon-master.his.de" // Run on master node
|
||
$ hostnamectl set-hostname "vmcon-node1.his.de" // Run on 1st worker node
|
||
$ hostnamectl set-hostname "vmcon-node2.his.de" // Run on 2nd worker node
|
||
```
|
||
Also add the following entries in /etc/hosts file on all the nodes,
|
||
|
||
192.168.202.10 vmcon-master.his.de vmcon-master.his.de \
|
||
192.168.202.11 vmcon-node1.his.de vmcon-node1 \
|
||
192.168.202.12 vmcon-node2.his.de vmcon-node2
|
||
|
||
## Disable Swap on All Nodes
|
||
For kubelet to work smoothly, it is recommended to disable swap. Run following commands on master and worker nodes to turn off swap.
|
||
|
||
```bash
|
||
$ swapoff -a
|
||
$ sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab
|
||
```
|
||
|
||
## Add Firewall Rules for Kubernetes Cluster
|
||
In case, OS firewall is enabled on your Debian systems then allow following ports on master and worker nodes respectively.
|
||
|
||
On Master node, run
|
||
```bash
|
||
$ ufw allow 6443/tcp
|
||
$ ufw allow 2379/tcp
|
||
$ ufw allow 2380/tcp
|
||
$ ufw allow 10250/tcp
|
||
$ ufw allow 10251/tcp
|
||
$ ufw allow 10252/tcp
|
||
$ ufw allow 10255/tcp
|
||
$ ufw reload
|
||
```
|
||
|
||
On Worker Nodes,
|
||
```bash
|
||
$ ufw allow 10250/tcp
|
||
$ ufw allow 30000:32767/tcp
|
||
$ ufw reload
|
||
```
|
||
|
||
Note: If firewall is disabled on your Debian 12/11 systems, then you can skip this step.
|
||
|
||
## Install Containerd Run time on All Nodes
|
||
Containerd is the industry standard container run time and supported by Kubernetes. So, install containerd on all master and worker nodes.
|
||
|
||
Before installing containerd, set the following kernel parameters on all the nodes.
|
||
```bash
|
||
$ cat <<EOF | sudo tee /etc/modules-load.d/containerd.conf \
|
||
overlay \
|
||
br_netfilter \
|
||
EOF \
|
||
$ modprobe overlay \
|
||
$ modprobe br_netfilter \
|
||
$ cat <<EOF | sudo tee /etc/sysctl.d/99-kubernetes-k8s.conf \
|
||
net.bridge.bridge-nf-call-iptables = 1 \
|
||
net.ipv4.ip_forward = 1 \
|
||
net.bridge.bridge-nf-call-ip6tables = 1 \
|
||
EOF
|
||
```
|
||
|
||
To make above changes into the effect, run
|
||
```bash
|
||
$ sysctl --system
|
||
```
|
||
|
||
Now, install conatinerd by running following apt command on all the nodes.
|
||
```bash
|
||
$ apt update
|
||
$ apt -y install containerd
|
||
```
|
||
|
||
Next, configure containerd so that it works with Kubernetes, run beneath command on all the nodes
|
||
```bash
|
||
$ containerd config default | sudo tee /etc/containerd/config.toml >/dev/null 2>&1
|
||
```
|
||
|
||
Edit the file ‘/etc/containerd/config.toml’ and look for the section ‘[plugins.”io.containerd.grpc.v1.cri”.containerd.runtimes.runc.options]’ and change ‘SystemdCgroup = false’ to ‘SystemdCgroup = true‘
|
||
```bash
|
||
$ nano /etc/containerd/config.toml
|
||
```
|
||
|
||
Save and exit the file.
|
||
|
||
Restart and enable containerd service on all the nodes,
|
||
|
||
```bash
|
||
$ systemctl restart containerd
|
||
$ systemctl enable containerd
|
||
```
|
||
|
||
## Add Kubernetes Apt Repository
|
||
In Debian 12/11, Kubernetes related packages are not available in the default package repositories. We have to add additional Kubernetes apt repository on all the nodes, run
|
||
```bash
|
||
$ echo "deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.28/deb/ /" | sudo tee /etc/apt/sources.list.d/kubernetes.list
|
||
$ curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.28/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
|
||
```
|
||
|
||
## Install Kubernetes Tools
|
||
Next, install the Kubernetes tools, including kubeadm, kubelet, and kubectl on all the nodes.
|
||
```bash
|
||
$ apt update
|
||
$ apt install kubelet kubeadm kubectl -y
|
||
$ apt-mark hold kubelet kubeadm kubectl
|
||
```
|
||
|
||
## Install Kubernetes Cluster with Kubeadm
|
||
kubelet doesn’t appreciate the command-line options anymore (these are deprecated). Instead, I suggest to create a configuration file, say ‘kubelet.yaml’ with following content.
|
||
```bash
|
||
$ nano kubelet.yaml
|
||
```
|
||
|
||
```yaml
|
||
apiVersion: kubeadm.k8s.io/v1beta3 \
|
||
kind: InitConfiguration \
|
||
--- \
|
||
apiVersion: kubeadm.k8s.io/v1beta3 \
|
||
kind: ClusterConfiguration \
|
||
kubernetesVersion: "1.28.0" # Replace with your desired version \
|
||
controlPlaneEndpoint: "k8s-master" \
|
||
--- \
|
||
apiVersion: kubelet.config.k8s.io/v1beta1 \
|
||
kind: KubeletConfiguration
|
||
```
|
||
|
||
Now, we are all set to initialize Kubernetes cluster, run following command only from master node,
|
||
```bash
|
||
$ kubeadm init --config kubelet.yaml
|
||
```
|
||
|
||
Above output confirms that control plane has been initialized successfully. In the output, we have commands for regular user for interacting with the cluster and also the command to join any worker node to this cluster.
|
||
|
||
To start interacting with cluster, run following commands on master node,
|
||
```bash
|
||
$ mkdir -p $HOME/.kube
|
||
$ sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
|
||
$ sudo chown $(id -u):$(id -g) $HOME/.kube/config
|
||
```
|
||
|
||
Run following kubectl command to get nodes and cluster information
|
||
```bash
|
||
$ kubectl get nodes \
|
||
$ kubectl cluster-info
|
||
```
|
||
|
||
On your worker nodes, join them to the cluster by running the command that was displayed when you initialized the master node. It will look something like this ‘Kubeadm join’
|
||
|
||
>Note: Copy the exact command from the output of ‘kubeadm init’ command. In my case, following is the command
|
||
|
||
```bash
|
||
$ sudo kubeadm join k8s-master:6443 --token 21nm87.x1lgd4jf0lqiiiau \
|
||
--discovery-token-ca-cert-hash sha256:28b503f1f2a2592678724c482776f04b445c5f99d76915552f14e68a24b78009
|
||
```
|
||
|
||
Check the nodes status by running following command from master node,
|
||
```bash
|
||
$ kubectl get nodes
|
||
```
|
||
|
||
To make nodes status ready, we must install POD network addons like Calico or flannel.
|
||
|
||
## Setup Pod Network Using Calico
|
||
On the master node, run beneath command to install calico,
|
||
```bash
|
||
$ kubectl apply -f https://raw.githubusercontent.com/projectcalico/calico/v3.26.1/manifests/calico.yaml
|
||
```
|
||
|
||
Allow Calico ports in OS firewall, run beneath ufw commands on all the nodes,
|
||
```bash
|
||
$ sudo ufw allow 179/tcp \
|
||
$ sudo ufw allow 4789/udp \
|
||
$ sudo ufw allow 51820/udp \
|
||
$ sudo ufw allow 51821/udp \
|
||
$ sudo ufw reload \
|
||
```
|
||
|
||
Verify the status of Calico pods, run
|
||
```bash
|
||
$ kubectl get pods -n kube-system
|
||
```
|
||
Confirm, that master and worker nodes are in ready staate. Now, we can say our cluster ready handle workload.
|
||
|
||
## Install MetalLB
|
||
```bash
|
||
$ kubectl create ns metallb-system
|
||
$ helm upgrade --install -n metallb-system metallb \
|
||
oci://registry-1.docker.io/bitnamicharts/metallb
|
||
```
|
||
|
||
Next create L2-range-allocation.yaml and define your IP range pool.
|
||
```yaml
|
||
# L2-range-allocation.yaml
|
||
apiVersion: metallb.io/v1beta1
|
||
kind: IPAddressPool
|
||
metadata:
|
||
name: main-pool
|
||
namespace: metallb-system
|
||
spec:
|
||
addresses:
|
||
- 192.168.202.160/27
|
||
---
|
||
apiVersion: metallb.io/v1beta1
|
||
kind: L2Advertisement
|
||
metadata:
|
||
name: empty
|
||
namespace: metallb-system
|
||
```
|
||
|
||
Apply with
|
||
```bash
|
||
$ kubectl apply -f L2-range-allocation.yaml
|
||
```
|
||
|
||
Check if all speakers are running
|
||
```bash
|
||
$ kubectl get pods -n metallb-system
|
||
```
|
||
>Note: https://itnext.io/kubernetes-loadbalancer-service-for-on-premises-6b7f75187be8
|
||
|
||
## Test Kubernetes Cluster Installation
|
||
In order validate and test Kubernetes cluster installation, let’s try to deploy nginx based application via deployment. Run beneath commands,
|
||
```bash
|
||
$ kubectl create deployment nginx-app --image=nginx --replicas 2
|
||
$ kubectl expose deployment nginx-app --name=nginx-web-svc --type NodePort --port 80 --target-port 80
|
||
$ kubectl describe svc nginx-web-svc
|
||
```
|
||
Try to access the nginx based application using following curl command along with the nodeport 32283.
|
||
|
||
>Note : In the curl command we can use either of worker node’s hostname.
|
||
|
||
>$ curl http://vmcon-node1.his.de:32283 |