Mitch Murphy

Mitch Murphy

Improving humanity by teaching machines how to learn.

How to Guide: RKE2 Install with Cilium, Hubble and Cluster Mesh

Title: RKE2 Install with Cilium, Hubble and Cluster Mesh
Author: Mitch Murphy
Date: 2023-11-25

Table of contents

Introduction

This guide will walk you through the steps to install RKE2, Cilium, Hubble, Cluster Mesh, MetalLB, Nginx-Ingress, Cert Manager, the PLG stack, Fluent Bit, Harbor and Longhorn on a Windows Server/Data Center 2022 using Hyper-V VMs running Rocky Linux 8.8.

What is Cilium?

Cilium is an open-source software project that provides networking and security for containerized applications in platforms like Kubernetes. It is designed to enhance the networking capabilities of container orchestration systems by offering features such as load balancing, service discovery, and security enforcement.

In the context of Kubernetes, Cilium serves as a networking and security solution that leverages the Linux kernel’s eBPF (extended Berkeley Packet Filter) technology. eBPF is a powerful and flexible in-kernel execution environment that allows the dynamic insertion of custom code into the Linux kernel without modifying its source.

Here are some key aspects of Cilium in relation to Kubernetes:

  • Networking: Cilium provides efficient and high-performance networking for containerized workloads. It supports features like load balancing, network visibility, and fine-grained network policies.
  • Security: Cilium enhances security by using eBPF to enforce network security policies at the kernel level. This allows for the implementation of fine-grained security policies based on factors such as application identity and context.
  • Load Balancing: Cilium includes load balancing capabilities that help distribute traffic across services, ensuring high availability and optimal performance.
  • Service Discovery: Cilium aids in service discovery by providing mechanisms for applications to discover and connect to each other dynamically within the Kubernetes cluster.
  • API-Aware Network Security: Cilium allows for the creation of security policies based on the API and application layer information. This enables the definition of security rules that consider the specific requirements of applications.
  • Integration with Kubernetes: Cilium integrates seamlessly with Kubernetes and is often used as a replacement for the default Kubernetes networking solution (e.g., kube-proxy) to provide additional features and improvements.

By leveraging eBPF, Cilium is able to achieve these functionalities with low overhead and high efficiency. It is worth noting that the Kubernetes ecosystem is dynamic, and the capabilities of projects like Cilium may evolve over time. Always refer to the official documentation and community resources for the most up-to-date information.

What is Hubble?

Hubble is a network visibility and monitoring tool that is closely associated with Cilium. It is part of the Cilium project and is designed to provide real-time visibility into the network traffic within a Cilium-enabled Kubernetes cluster. Hubble leverages the eBPF (extended Berkeley Packet Filter) technology to capture and analyze network events at the kernel level, allowing for detailed insights into the communication between services and workloads.

Key features and aspects of Hubble in relation to Cilium and Kubernetes include:

  • Real-time Visibility: Hubble provides real-time visibility into the network communication between microservices and containers within a Kubernetes cluster. This visibility includes information about network flows, latencies, and error rates.
  • Topology Mapping: Hubble generates topology maps that illustrate the relationships and connections between different services and workloads in the cluster. These maps help administrators and developers understand the network architecture and dependencies.
  • Flow Tracing: Hubble allows for the tracing of network flows, enabling the visualization of the entire path taken by a packet as it traverses the network. This feature is valuable for troubleshooting and understanding the network behavior of applications.
  • Security Insights: By capturing and analyzing network events, Hubble can contribute to security insights by providing information about communication patterns and potential anomalies. This information can be useful for identifying and responding to security incidents.
  • Integration with Cilium: Hubble is tightly integrated with Cilium and relies on Cilium’s eBPF-based networking and security capabilities. It complements Cilium’s features by offering a tool specifically focused on network visibility and monitoring.
  • Web UI and CLI: Hubble provides both a web-based user interface (UI) and a command-line interface (CLI) for interacting with and querying network visibility data. The UI offers a graphical representation of the network topology and flow information.

Overall, Hubble enhances the observability and troubleshooting capabilities of a Cilium-enabled Kubernetes environment. It is part of the broader ecosystem of tools and features provided by Cilium to address networking, security, and observability challenges in containerized environments. Keep in mind that the specifics of Hubble’s features and capabilities may evolve, so it’s advisable to refer to the official documentation and community resources for the latest information.

Prerequisites

In order to follow this guide, there are a few prerequisites that need to be met. Thes include: networking, kernel configuration, security settings, storage configuration, DNS, and a few other things.

Networking Pre-Requisites

Since we will be using cilium (eBPF) as the CNI, we will not need to configure iptables at all. However, packets must be forwaded and a couple kernel modules need to be enabled:

sudo modprobe br_netfilter
sudo modprobe overlay
sudo su -
cat <<EOT | sudo tee /etc/modules-load.d/kubernetes.conf
br_netfilter
overlay
EOT
cat <<EOT | sudo tee /etc/sysctl.d/kubernetes.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
EOT
sysctl --system
exit

Kernel Configuration

Before installing Cilium, please ensure that your system meets the minimum requirements below. Most modern Linux distributions already do.

  • Hosts with either AMD64 or AArch64 architecture
  • Linux kernel >= 4.19.57 or equivalent (e.g., 4.18 on RHEL8)

Even though Rocky Linux is a RHEL clone, and the default kernel is 4.18, we will be using the latest kernel available from the Rocky Linux kernel-ml repo. This is because the latest kernel has the latest security patches and bug fixes. To install the latest kernel, run the following commands:

sudo dnf -y upgrade --refresh
sudo rpm --import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org
sudo dnf install https://www.elrepo.org/elrepo-release-8.el8.elrepo.noarch.rpm -y
sudo dnf --enablerepo=elrepo-kernel install -y kernel-ml kernel-ml-core kernel-ml-headers kernel-ml-modules kernel-ml-modules-extra

Now you must restart.

sudo reboot now

After reboot, check the kernel version:

uname -r

selinux Configuration

If you want to use selinux you must install container-selinux:

sudo dnf install -y container-selinux

Otherwise (for starters I suggest not using selinux), SELinux must be disabled on the nodes. To do so, run the following commands:

sudo setenforce 0
sudo sed -i 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config

Firewall Configuration

If you are using firewalld, you will need to add the following rules:

ALLOWED_PORTS=( 6443 9100 8080 4245 9345 6443 6444 10250 10259 10257 2379 2380 9796 19090 9090 6942 9091 4244 4240 80 443 9963 9964 8081 8082 7000 9001 6379 9121 8084 6060 6061 6062 9879 9890 9891 9892 9893 9962 9966 ) 
for i in "${ALLOWED_PORTS[@]}"
do
  sudo firewall-cmd --add-port=$i/tcp --permanent
done

sudo firewall-cmd --add-port=30000-32767/tcp --permanent
sudo firewall-cmd --remove-icmp-block=echo-request --permanent
sudo firewall-cmd --remove-icmp-block=echo-reply --permanent
# Since we are using Cilium with GENEVE as overlay, we need the following port too:
# UDP
UDP_PORTS=( 8472 4789 6081 51871 53 55355 58467 41637 39291 38519 46190 )
for i in "${UDP_PORTS[@]}"
do
  sudo firewall-cmd --add-port=$i/udp --permanent
done
sudo firewall-cmd --reload
### Ingress Controller specific ports

### To get DNS resolution working, simply enable Masquerading.
sudo firewall-cmd --zone=public  --add-masquerade --permanent

sudo firewall-cmd --zone=trusted --permanent --add-source=192.168.0.0/16

### Finally apply all the firewall changes
sudo firewall-cmd --reload

Storage Configuration

We will be using Longhorn and local storage for this guide. To use Longhorn, you need iscii:

sudo su -
yum install -y nano curl wget git tmux jq vim-common iscsi-initiator-utils
echo "iscsi_tcp" >/etc/modules-load.d/iscsi-tcp.conf
systemctl enable iscsid --now
systemctl start iscsid

cat <<EOF>> /etc/NetworkManager/conf.d/rke2-canal.conf
[keyfile]
unmanaged-devices=interface-name:cali*;interface-name:flannel*
EOF
systemctl reload NetworkManager
exit

This completes the prerequisites. Now we can install RKE2.

RKE2 Installation

Control Plane Node

To beghin with, lets create a config file for RKE2. This will be used to configure the cluster. Take note that we are defining the cluster and service CIDR, which is required if we want to use Cilium Cluster Mesh (so that we have no collisions with the default CIDR of the cluster). We are also defining the tls-sans which is the IP address of the master node. This is required for the kubeconfig to work properly.

sudo su -
mkdir -p /etc/rancher/rke2/
cat <<EOF > /etc/rancher/rke2/config.yaml
write-kubeconfig-mode: "0644"
# profile: "cis-1.5"
selinux: false
# add ips/hostname of hosts and loadbalancer
tls-sans:
  - "c1cp1.kubula.internal"
  - "192.168.7.11"
# Make a etcd snapshot every day at 4am
etcd-snapshot-schedule-cron: "0 4 * * *"
# Keep 14 etcd snapshots
etcd-snapshot-retention: 14
etcd-expose-metrics: true
disable:
  - rke2-canal
  - rke2-kube-proxy
network:
  plugin: none
disable-kube-proxy: true
disable-cloud-controller: true
cluster-cidr: 10.42.0.0/16
service-cidr: 10.96.0.0/16
EOF

Download and Install RKE2

Now we can download and install RKE2. We will be using the latest release, which at the time of writing is v1.28.3+rke2r2. You can find the latest release here. To download and install RKE2, run the following commands:

curl -sfL https://get.rke2.io | sudo INSTALL_RKE2_CHANNEL=latest INSTALL_RKE2_TYPE="server" sh -

Note: you can specify the exact version by setting INSTALL_RKE2_VERSION to the version you want to install. After installing RKE2, make sure to add the following to the dnf configuration to prevent RKE2 from being updated by dnf:

sudo su -
echo "exclude=rke2-*" >> /etc/dnf/dnf.conf
exit

Start RKE2

Now we can start RKE2:

sudo systemctl enable rke2-server.service --now

Installation Cilium CLI

For this guide, we will be using the Cilium CLI to install Cilium. Note that this can be done via a Helm chart as well. To install the Cilium CLI, run the following commands:

CILIUM_CLI_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/cilium-cli/main/stable.txt)
CLI_ARCH=amd64
curl -L --fail --remote-name-all https://github.com/cilium/cilium-cli/releases/download/${CILIUM_CLI_VERSION}/cilium-linux-${CLI_ARCH}.tar.gz{,.sha256sum}
sha256sum --check cilium-linux-${CLI_ARCH}.tar.gz.sha256sum
sudo tar xzvfC cilium-linux-${CLI_ARCH}.tar.gz /usr/local/bin
rm cilium-linux-${CLI_ARCH}.tar.gz{,.sha256sum}

Verify RKE2

To verify that RKE2 is running, run the following command:

sudo systemctl status rke2-server.service

We should also get the KUBECONFIG file that is generated by RKE2. This will be used to access the cluster. To get the KUBECONFIG file, run the following command:

mkdir ~/.kube
sudo cp /etc/rancher/rke2/rke2.yaml ~/.kube/config
sudo chown $(id -u):$(id -g) ~/.kube/config
chmod 600 ~/.kube/config

sudo cp /var/lib/rancher/rke2/bin/kubectl /usr/local/bin
sudo chown $(id -u):$(id -g) /usr/local/bin/kubectl

Now we can verify that the cluster is running:

kubectl get nodes

Note: Because no CNI is installed, we should see that the nodes are in NotReady state.

Install Cilium

Now lets install Cilium. We will be using the Cilium CLI to install Cilium. In order to move all configuration options into a single file, we will be using a cilium.yaml file. This file will be used to configure Cilium. To create the cilium.yaml file, run the following command:

cat <<EOF > cilium.yaml
cluster:
  name: smig-cluster1
  id: 11
prometheus:
  enabled: true
  serviceMonitor:
    enabled: false
dashboards:
  enabled: true
hubble:
  metrics:
    enabled:
    - dns:query;ignoreAAAA
    - drop
    - tcp
    - flow
    - icmp
    - http
    dashboards:
      enabled: true
  relay:
    enabled: true
    prometheus:
      enabled: true
  ui:
    enabled: true
    baseUrl: "/"
version: 1.14.3
operator:
  prometheus:
    enabled: true
  dashboards:
    enabled: true
# clustermesh:
#   # -- Deploy clustermesh-apiserver for clustermesh
#   useAPIServer: false
EOF

Now we can install Cilium:

cilium install -f cilium.yaml

Verify Cilium

Now we can verify that Cilium is running:

cilium status

And we can verify that the nodes are ready:

kubectl get nodes

Now you should see the node as Ready:

NAME     STATUS   ROLES    AGE   VERSION
c1cp1    Ready    master   10m   v1.28.3+rke2r2

It is important that you also test connectivity with Cilum:

cilium hubble port-forward & #3855849
cilium connectivity test --force-deploy

Worker Nodes

Once the controlplane is functional, we can install the and configure the RKE2 agent on the worker node(s). Until the exact confiruation/prerequisites are exported to a VHDX or encapsulated in Terraform/Ansible automation, please ensure that all aforementioned prerequisite steps are done on the worker VMs.

First, we need to get the token from the controlplane node:

TOKEN=$(ssh -i ~/.ssh/rke2 -o IdentitiesOnly=yes master@192.168.7.11 sudo cat /var/lib/rancher/rke2/server/node-token)

Now we need to create the configuration file for the RKE2 agent. To do this, run the following commands:

```yaml
sudo su -
mkdir -p /etc/rancher/rke2/
cat <<EOF > /etc/rancher/rke2/config.yaml
server: https://192.168.1.175:9345
token: ${TOKEN}
EOF
exit

Now intall RKE2:

curl -sfL https://get.rke2.io | sudo INSTALL_RKE2_CHANNEL=latest INSTALL_RKE2_TYPE="agent" sh -

Note: you can specify the exact version by setting INSTALL_RKE2_VERSION to the version you want to install. After installing RKE2, make sure to add the following to the dnf configuration to prevent RKE2 from being updated by dnf:

sudo su -
echo "exclude=rke2-*" >> /etc/dnf/dnf.conf
exit

Now we can start RKE2:

sudo systemctl enable rke2-agent.service --now

Verify RKE2 Agent

To verify that RKE2 is running, run the following command:

sudo systemctl status rke2-agent.service

You can check the pods on the controlplane node to see if the agent has joined the cluster, and in the kube-system namespace the status of the cilium-operator. This operator will perform the installation of Cilium on the agent node. Once complete, the agent node will be ready.

kubectl get nodes

MetalLB

What is MetalLB?

MetalLB is an open-source, community-driven project that provides a load balancer implementation for bare-metal Kubernetes clusters. In a typical Kubernetes deployment, cloud providers often offer load balancing services that can be easily integrated with Kubernetes to distribute traffic to the appropriate pods. However, when running Kubernetes on bare metal, which means without the assistance of a cloud provider’s load balancing service, an external load balancer is needed to expose services to the external network.

Here are some key aspects of MetalLB in relation to Kubernetes:

  • Load Balancing for Bare Metal: MetalLB is specifically designed to address the need for load balancing in bare-metal Kubernetes clusters. It provides a network load balancer implementation that can be used to expose services externally, just like you would in a cloud environment.
  • Layer 2 and BGP Modes: MetalLB supports two operation modes: Layer 2 (L2) mode and Border Gateway Protocol (BGP) mode. In L2 mode, MetalLB operates in the data link layer, using ARP (Address Resolution Protocol) to respond to service IP requests. In BGP mode, MetalLB advertises service IP addresses to the network using the BGP routing protocol. Integration with Kubernetes Services: MetalLB integrates with Kubernetes services and automatically assigns and manages external IP addresses for services of type LoadBalancer. This allows services to be accessed from outside the cluster using the assigned external IP. Configuration and Customization: MetalLB is configurable, allowing users to customize the behavior based on their specific requirements. Users can define pools of IP addresses that MetalLB can allocate from, and they can choose between the Layer 2 and BGP modes based on their network setup.
  • High Availability: MetalLB can be configured for high availability by running multiple instances in the cluster, ensuring that if one instance goes down, another can take over.

Using MetalLB in a bare-metal Kubernetes environment enables users to take advantage of load balancing for their services, which is essential for applications that need to be accessed from outside the cluster. It’s particularly useful in scenarios where a cloud provider’s load balancing services are not available.

Intall MetalLB

To install MetalLB, we will use Helm. First, we need to add the Helm repository:

helm repo add metallb https://metallb.github.io/metallb
helm install metallb metallb/metallb

You can now install MetalLB:

helm install --namespace metallb-system metallb metallb/metallb

The easiest way to use MetalLB is to configure networking at layer 2. Under this approach, you simply assign a range of IP addresses to MetalLB. It then automatically assigns them nodes and manages traffic between them and your endpoints.

To define the address pool, open the MetalLB ConfigMap with:

kubectl edit configmap config -n metallb-system

Then define the address-pools and addresses values as desired. For example, to use the range 192.168.255.1–192.168.255.255, set the configuration to:

apiVersion: v1
kind: ConfigMap
metadata:
  namespace: metallb-system
  name: config
data:
  config:
    address-pools:
    - name: default
    protocol: layer2
      addresses:
      - 192.168.8.1-192.168.8.255

After making changes, apply them with:

kubectl rollout restart deployment controller -n metallb-system

Verify MetalLB

To verify that MetalLB is running, run the following command:

kubectl get pods -n metallb-system

Cilium Cluster Mesh

Cilium Cluster Mesh allows you to connect the networks of multiple clusters in such as way that pods in each cluster can discover and access services in all other clusters of the mesh, provided all the clusters run Cilium as their CNI. This allows effectively joining multiple clusters into a large unified network, regardless of the Kubernetes distribution or location each of them is running. This is achieved by leveraging the Cilium Identity feature to assign a unique identity to each pod, and using that identity to establish secure connections between pods in different clusters. Furthermore, global services can be defined to expose services across the entire mesh to further improve HA. This is a great way to connect multiple clusters together, and it’s very easy to set up.

In this guide we have two clusters, smig-cluster1 and smig-cluster2. We will use smig-cluster1 as the primary cluster, and smig-cluster2 as the secondary cluster. We will connect smig-cluster2 to smig-cluster1. To do this, we need to copy over the KUBECONFIG file from smig-cluster1 to our local machine, and then copy it over to smig-cluster2. Note that you when merging the KUBECONFIG files, you need to make sure that the clusters and contexts sections are unique, but more importantly update the server value in the clusters section to point to the IP address of the corresponding cluster control plane.

CLUSTER1=smig-cluster1
CLUSTER2=smig-cluster2

# In order for cluster mesh to work properly, the same CA secret must exist in both clusters. Therefore delete the secret in cluster 2 and recreate it from cluster 1.
kubectl --context=$CLUSTER2 delete secret -n kube-system cilium-ca
kubectl --context=$CLUSTER1 get secret -n kube-system cilium-ca -o yaml | \
  kubectl --context $CLUSTER2 create -f -

cilium clustermesh enable --context $CLUSTER1 --service-type ClusterIP
cilium clustermesh enable --context $CLUSTER2 --service-type ClusterIP


cilium clustermesh connect --context $CLUSTER1 --destination-context $CLUSTER2
cilium clustermesh connect --context $CLUSTER2 --destination-context $CLUSTER1

Create ServiceMonitors

When you have Prometheus installed, you can enable the Cilium ServiceMonitor to scrape metrics from Cilium. This is done by creating a ServiceMonitor resource in the monitoring-system namespace. The ServiceMonitor resource is defined in the monitoring-system namespace, and it will automatically discover the Cilium pods and scrape metrics from them. To create the ServiceMonitor, use the cilium CLI (or Helm chart) to update the Cilium installation, using the below configuration:

cat <<EOF > cilium-upgrade.yaml
hubble:
  metrics:
    serviceMonitor:
      enabled: true
    dashboards:
      enabled: true
      namespace: monitoring-system
  relay:
    prometheus:
      enabled: true
      serviceMonitor:
        enabled: true
        namespace: monitoring-system
prometheus:
  enabled: true
  serviceMonitor:
    enabled: true
    namespace: monitoring-system
envoy:
  prometheus:
    enabled: true
    serviceMonitor:
      enabled: true
      namespace: monitoring-system
operator:
  prometheus:
    enabled: true
    serviceMonitor:
      enabled: true
      namespace: monitoring-system
  dashboards:
    enabled: true
    namespace: monitoring-system
clustermesh:
  apiserver:
    metrics:
      serviceMonitor:
        enabled: true
        namespace: monitoring-system
dashboards:
  enabled: true
  namespace: monitoring-system
EOF

Then run the following command to apply the changes:

cilium upgrade -f cilium-upgrade.yaml

Verify Cilium ServiceMonitor

To verify that the Cilium ServiceMonitor is running, run the following command:

kubectl get servicemonitor -n monitoring-system

You should see the following ServiceMonitor resources (among others)):

  • cilium-agent
  • cilium-operator
  • hubble

You can also verify this if you go to the Prometheus UI and click on Status -> Targets. You should see the above resources listed and ensure that they are in the Up state.

This will also create a couple Grafana dashboard for Cilium and Hubble. They will be stored in ConfigMaps in the monitoring-system namespace. To view the dashboards, you can use port-forwarding to access the Grafana UI:

kubectl port-forward -n monitoring-system svc/prometheus-stack-grafana 3000:80

Then open your browser and go to http://localhost:3000. You should see the Grafana UI. Click on Explore and you should see the Cilium and Hubble dashboards.

You may also enjoy