Create a Kubernetes Cluster from Scratch with Alma Linux
This document will take you through the process of initializing a Kubernetes from scratch on baremetal. I’ll include steps to configure a hypervisor and maybe split into 3 separate documents in the future.
Create the Alma Linux VM that will function as the control plane node
virt-install --name cluster3 --memory 8096 --vcpus=1 --location ./AlmaLinux-9.5-x86_64-minimal.iso \
--os-type=linux --os-variant=almalinux9 --disk pool=default,bus=virtio,size=50 --network bridge=kvm_bridge,model=virtio \
--autostart --graphics none --console=pty,target_type=serial --extra-args "console=ttyS0,115200n8 locale=en_US auto=true"
Note: I have several kickstart files, but not sure if they’d work with Alma Linux and also wanted to manually do this as it’d been awhile.
Start the installation in text mode:
Starting installer, one moment...
anaconda 34.25.5.9-1.el9.alma.1 for AlmaLinux 9.5 started.
* installation log files are stored in /tmp during the installation
* shell is available on TTY2
* if the graphical installation interface fails to start, try again with the
inst.text bootoption to start text installation
* when reporting a bug add logs from /tmp as separate text/plain attachments
================================================================================
================================================================================
Text mode provides a limited set of installation options. It does not offer
custom partitioning for full control over the disk layout. Would you like to use
VNC mode instead?
1) Start VNC
2) Use text mode
Please make a selection from the above ['c' to continue, 'q' to quit, 'r' to
refresh]: 2
Initial screen looks like:
================================================================================
================================================================================
Installation
1) [x] Language settings 2) [x] Time settings
(English (United States)) (America/New_York timezone)
3) [!] Installation source 4) [!] Software selection
(Processing...) (Processing...)
5) [!] Installation Destination 6) [x] Kdump
(Processing...) (Kdump is enabled)
7) [x] Network configuration 8) [!] Root password
(Connected: enp1s0) (Root account is disabled)
9) [!] User creation
(No user will be created)
Please make a selection from the above ['b' to begin installation, 'q' to quit,
'r' to refresh]:
Select ‘r’ to refresh and Installation source should have an ‘x’ (good to go):
================================================================================
================================================================================
Installation
1) [x] Language settings 2) [x] Time settings
(English (United States)) (America/New_York timezone)
3) [x] Installation source 4) [!] Software selection
(Processing...) (Processing...)
5) [!] Installation Destination 6) [x] Kdump
(Processing...) (Kdump is enabled)
7) [x] Network configuration 8) [!] Root password
(Connected: enp1s0) (Root account is disabled)
9) [!] User creation
(No user will be created)
Please make a selection from the above ['b' to begin installation, 'q' to quit,
'r' to refresh]: r
Select ‘5’ to configure an installation destination. I have a single virtual disk which is automatically selected:
Installation Destination
1) [x] DISK: 50 GiB (vda)
1 disk selected; 50 GiB capacity; 50 GiB free
Select ‘c’ to continue. I have nothing special to configure, I just let it use all of the disk which is selected by default:
================================================================================
================================================================================
Partitioning Options
1) [ ] Replace Existing Linux system(s)
2) [x] Use All Space
3) [ ] Use Free Space
4) [ ] Manually assign mount points
Installation requires partitioning of your hard drive. Select what space to use
for the install target or manually assign mount points.
Please make a selection from the above ['c' to continue, 'q' to quit, 'r' to
refresh]: c
Select ‘c’ to continue. I just want a standard partition so I pressed ‘1’ to select Standard Partition:
================================================================================
================================================================================
Partition Scheme Options
1) [ ] Standard Partition
2) [x] LVM
3) [ ] LVM Thin Provisioning
Select a partition scheme configuration.
Please make a selection from the above ['c' to continue, 'q' to quit, 'r' to
refresh]: 1
Select ‘c’ to continue when Standard Partition is selected:
================================================================================
================================================================================
Partition Scheme Options
1) [x] Standard Partition
2) [ ] LVM
3) [ ] LVM Thin Provisioning
Select a partition scheme configuration.
Please make a selection from the above ['c' to continue, 'q' to quit, 'r' to
refresh]: c
Saving storage configuration...
Checking storage configuration...
Select ‘9’ to create a user and then ‘1’ to create the user (seems superfluous, but meh):
================================================================================
================================================================================
User creation
1) [ ] Create user
Please make a selection from the above ['c' to continue, 'q' to quit, 'r' to
refresh]: 1
User creation screen:
================================================================================
================================================================================
User creation
1) [x] Create user
2) Full name
3) User name
4) [x] Use password
5) Password
6) [ ] Administrator
7) Groups
Please make a selection from the above ['c' to continue, 'q' to quit, 'r' to
refresh]:
Select ‘3’ to enter the username:
================================================================================
================================================================================
User creation
1) [x] Create user
2) Full name
3) User name
4) [x] Use password
5) Password
6) [ ] Administrator
7) Groups
Please make a selection from the above ['c' to continue, 'q' to quit, 'r' to
refresh]: 3
================================================================================
================================================================================
Enter a new value for 'User name' and press ENTER: username
Select ‘5’ to enter a password. Enter the password and then once again to confirm:
================================================================================
================================================================================
User creation
1) [x] Create user
2) Full name
3) User name
username
4) [x] Use password
5) Password
6) [ ] Administrator
7) Groups
Please make a selection from the above ['c' to continue, 'q' to quit, 'r' to
refresh]: 5
================================================================================
================================================================================
Password:
Password (confirm):
Enter ‘yes’ if it tells you that the password is weak and asks if you want to keep it or ’no’ to change it
================================================================================
================================================================================
Question
The password you have provided is weak: The password fails the dictionary check
- it is based on a dictionary word
Would you like to use it anyway?
Please respond 'yes' or 'no': yes
Select ‘6’ to make them an Administrator
================================================================================
================================================================================
User creation
1) [x] Create user
2) Full name
3) User name
username
4) [x] Use password
5) Password
Password set.
6) [x] Administrator
7) Groups
wheel
Please make a selection from the above ['c' to continue, 'q' to quit, 'r' to
refresh]:
Select ‘c’ to continue and then ‘b’ to begin installation.
================================================================================
================================================================================
Progress
.
Setting up the installation environment
Configuring storage
Creating disklabel on /dev/vda
Creating swap on /dev/vda2
Creating xfs on /dev/vda3
Creating xfs on /dev/vda1
...
Running pre-installation scripts
.
Running pre-installation tasks
....
Installing.
Starting package installation process
Downloading packages
Preparing transaction from installation source
...
Press Enter when the installation has completed
.
Installing boot loader
..
Performing post-installation setup tasks
.
Configuring installed system
..............
Writing network configuration
.
Creating users
.....
Configuring addons
.
Generating initramfs
....
Storing configuration files and kickstarts
.
Running post-installation scripts
.
Installation complete
Use of this product is subject to the license agreement found at:
/usr/share/almalinux-release/EULA
Installation complete. Press ENTER to quit:
Configure the OS
Test sudo works
[username@localhost ~]$ sudo su -
We trust you have received the usual lecture from the local System
Administrator. It usually boils down to these three things:
#1) Respect the privacy of others.
#2) Think before you type.
#3) With great power comes great responsibility.
[sudo] password for username:
[root@localhost ~]#
Test SSH works
My interface was named enp1s0, but your mileage may vary
ip a s enp1s0 | grep inet | grep -v inet6
inet 192.168.0.118/24 brd 192.168.0.255 scope global dynamic noprefixroute enp1s0
From another computer, ssh into the new virtual machine
username@desktop:~$ ssh 192.168.0.118
The authenticity of host '192.168.0.118 (192.168.0.118)' can't be established.
ED25519 key fingerprint is SHA256:GhxhtW4UG/de3/rWqHRIbba2W0U8ca90gSMRgZw1ef4.
This key is not known by any other names.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '192.168.0.118' (ED25519) to the list of known hosts.
username@192.168.0.118's password:
Last login: Thu Feb 13 18:09:05 2025
[username@localhost ~]$
Set the hostname
sudo hostnamectl set-hostname controller --static --transient
Configure networking
Install bash-completion
sudo dnf -y install bash-completion
Configure a static IP
nmcli connection modify [ interface_name ] ipv4.addresses [ ipv4_address ]/24 ipv4.gateway [ ipv4_gateway ] ipv4.dns [ ipv4_dns ] && \
nmcli connection modify [ interface_name ] ipv4.method manual && \
nmcli connection reload && \
nmcli connection up [ interface_name ]
For example
sudo nmcli connection modify enp1s0 ipv4.addresses 192.168.0.63/24 ipv4.gateway 192.168.0.1 ipv4.dns 192.168.0.1 && \
sudo nmcli connection modify enp1s0 ipv4.method manual && \
sudo nmcli connection reload && \
sudo nmcli connection up enp1s0
Re-establish another ssh connection to the new ip
username@desktop:~$ ssh 192.168.0.63
The authenticity of host '192.168.0.63 (192.168.0.63)' can't be established.
ED25519 key fingerprint is SHA256:GhxhtW4UG/de3/rWqHRIbba2W0U8ca90gSMRgZw1ef4.
This host key is known by the following other names/addresses:
~/.ssh/known_hosts:234: 192.168.0.118
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '192.168.0.63' (ED25519) to the list of known hosts.
username@192.168.0.63's password:
Last login: Thu Feb 13 18:11:54 2025 from 192.168.0.31
[username@controller ~]$
Add public ssh key to virtual machine known_hosts
username@desktop:~$ ssh-copy-id -i ~/.ssh/id_rsa.pub 192.168.0.63
Apply outstanding updates
sudo dnf -y update
and reboot
[username@controller ~]$ sudo reboot
Install Kubernetes
Note: I have ansible tasks to address this, but thought I’d go through the process manually as a refresher.
Select a release
Latest Kubernetes releases page:
I’m going to go with release v1.30
Install Kubernetes tools
cat <<EOF | sudo tee /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://pkgs.k8s.io/core:/stable:/v1.30/rpm/
enabled=1
gpgcheck=1
gpgkey=https://pkgs.k8s.io/core:/stable:/v1.30/rpm/repodata/repomd.xml.key
EOF
sudo dnf update && sudo dnf -y install kubeadm kubectl kubelet && sudo systemctl enable kubelet
Lock this version:
sudo dnf -y install 'dnf-command(versionlock)' && \
sudo dnf versionlock kubelet kubeadm kubectl
Disable swap
Temporarily
sudo swapoff -a
Permanently
- Edit /etc/fstab and comment out the swap entry
modprobe
Permanent
cat << EOF | sudo tee /etc/modprobe.d/kubernetes.conf
modprobe overlay
modprobe br_netfilter
EOF
Temporary
modprobe overlay
modprobe br_netfilter
sysctl
Permanent
cat << EOF | sudo tee /etc/sysctl.d/kubernetes.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
EOF
Temporary
sudo systctl --system
Install a container runtime
Containerd
Can be installed from binary, but easier - when manually doing this - to use Linux repos & dnf
sudo dnf remove docker \
docker-client \
docker-client-latest \
docker-common \
docker-latest \
docker-latest-logrotate \
docker-logrotate \
docker-engine
sudo dnf -y install dnf-plugins-core && \
sudo dnf config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo && \
sudo dnf update && sudo dnf -y install containerd.io
containerd config default | sudo tee /etc/containerd/config.toml
Edit /etc/containerd/config.toml and make sure SystemdCgroup = true is set
sudo systemctl restart containerd && \
sudo systemctl enable containerd
Install nerdctl & crictl
nerdctl
version="2.0.3"
curl -L https://github.com/containerd/nerdctl/releases/download/v2.0.3/nerdctl-$version-linux-amd64.tar.gz --output nerdctl-$version-linux-amd64.tar.gz
sudo tar zxvf nerdctl-$version-linux-amd64.tar.gz -C /usr/local/bin
rm -f nerdctl-$version-linux-amd64.tar.gz
crictl
version="v1.30.0"
curl -L https://github.com/kubernetes-sigs/cri-tools/releases/download/$version/crictl-${version}-linux-amd64.tar.gz --output crictl-${version}-linux-amd64.tar.gz
sudo tar zxvf crictl-$version-linux-amd64.tar.gz -C /usr/local/bin
rm -f crictl-$version-linux-amd64.tar.gz
Open firewall ports
firewalld is active, please ensure ports [6443 10250] are open or your cluster may not function correctly
sudo firewall-cmd --zone=public --add-port=6443/tcp --permanent && \
sudo firewall-cmd --zone=public --add-port=10250/tcp --permanent && \
sudo firewall-cmd --complete-reload
Edit kubelet systemd unit file
sudo echo 'KUBELET_RUNTIME_ARGS="--container-runtime=remote --container-runtime-endpoint=unix:///run/containerd/containerd.sock"' >> /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf && \
systemctl daemon-reload && systemctl restart kubelet
Initialize a new cluster
sudo kubeadm init --pod-network-cidr 192.168.0.0/16
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
Install Calico networking
kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.29.2/manifests/tigera-operator.yaml
kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.29.2/manifests/custom-resources.yaml
Add worker nodes
Print join command
sudo kubeadm token create --print-join-command
From each worker node in the cluster run the output of the above command.
kubectl get nodes -o wide
Troubleshooting
If things get really bad and kubectl commands are not working, crictl or nerdctl are what you’ll want to use to check how the pods are doing.
sudo crictl ps -a
I actually haven’t used nerdctl as of yet, but I’ll update this later on when I have.