Title: Cluster Mesh with Cilium Overview
Author: Mitch Murphy
Date: 2024-09-18

This will be a series of blog posts, first discussing some high-level background on a few of the technologies used. A future post will detail the technical implementation details for reproducing most of what was covered in a Lunch-N-Learn held on November 19, 2024. This demo session discussed configuring, deploying, testing and demoing Cilium Cluster Mesh. Specifically, we demonstrated the use of global services (and shared ingress) to significantly improve resiliency/availability, as well as splitting out stateless and stateful workloads (in this case observability metrics/logs using LGTM). We will also briefly discuss deploying a lightweight service mesh using Cilium 1.16.1, securing workloads via SPIRE/SPIFFE as well as node encryption via Wireguard.

Pre-requisites

Before configuring Cilium Cluster Mesh, ensure the following prerequisites are met:

• Unique CIDR Ranges for Pods and Services: Each Kubernetes cluster must have unique POD and SERVICES CIDR ranges to avoid conflicts and enable seamless cross-cluster communication via IP addresses.
• Unique Node CIDRs and IP Connectivity: Nodes in all clusters must have unique CIDR ranges, and IP connectivity between nodes in different clusters is essential. In my lab environment, I use direct connections between clusters, but per Cilium documentation, it is also possible to achieve connectivity using a VPN or other networking solutions like BGP (Border Gateway Protocol) or VXLAN encapsulation for workloads spread across clouds and regions.
• Clustermesh API Service Exposure: Cilium pods need to connect to the Clustermesh API service exposed on each interconnected cluster. While NodePort can be used to expose this service, this is not recommended for production environments. Instead, using Kubernetes ingress or LoadBalancer services is preferred for production setups to ensure robustness and security. I will be using a LoadBalancer service in this demo, and because I am using on-prem hardware this is possible via MetalLB. In a future article I plan to replace certain MetalLB functionality with Cilium.
• Consistent Certificate Authority (CA): To configure Cilium Cluster Mesh using the Helm chart without additional manual steps, the Cilium CNI in all clusters must be configured with the same Certificate Authority (CA). This ensures mutual trust between clusters, eliminating the need to manually generate or distribute certificates. This configuration can be specified in the Helm chart values file, allowing for seamless Cluster Mesh setup across clusters. Here, Ansible plays are resposible for creating this CA and “templating” the values file with this information.

Configuration

• RKE2: v1.31.0
• Cilium: v1.16.1
• Rocky Linux 9.4
• Kernel 6.10.11-1
• 4 vCPU’s and 8GB RAM per node
• Currently deployed on Windows Data Center 2022, using:
  • Ansible
  • Packer
• Kuberenetes resources:
  • Helm
  • Kustomize
  • ArgoCD
  • MetalLB
  • NGINX Ingress Controller
  • Prometheus CRDs
  • kube-state-metrics
  • LGTM-distributed stack
  • Grafana Alloy
  • nfs-subdir-external-provisioner

Cilium Introduction

Cilium is an advanced Container Network Interface (CNI) solution that leverages eBPF (Extended Berkeley Packet Filter), a powerful Linux kernel technology, to provide highly scalable and efficient networking, security, and observability for containerized workloads in Kubernetes.

In Kubernetes, networking is critical because it allows communication between different components like pods, services, and nodes. Traditionally, Kubernetes networking has relied on various CNIs like Flannel, Calico, or Weave to handle the connectivity between pods and services. However, Cilium stands out because it uses eBPF, which allows it to dynamically program the Linux kernel to handle packet filtering, forwarding, and security enforcement at the kernel level, offering several benefits:

1. Fine-Grained Network Security: Cilium provides a powerful Network Policy implementation that is capable of enforcing security policies based on the rich context of application-layer metadata. This goes beyond basic IP-based policies, allowing for policies that are aware of protocols, services, and even individual application processes.
2. Scalable Performance: Leveraging eBPF, Cilium can deliver networking with significantly lower overhead compared to traditional iptables-based approaches. eBPF runs inside the Linux kernel without needing to go back to user space, ensuring high performance and reduced latency.
3. Layer 7 Policies and Security: Cilium supports layer 7 (L7) visibility and filtering, which allows for enforcing security policies based on application-level protocols such as HTTP, gRPC, or Kafka. This is particularly useful in microservices architectures where services are constantly communicating over well-defined APIs.
4. Built-In Observability: Cilium offers Hubble, a built-in observability tool that provides deep insights into the networking behavior of Kubernetes clusters. With Hubble, administrators can visualize traffic flows, monitor security policies, and debug networking issues across their clusters in real time.
5. Service Mesh Integration: Cilium can integrate with service meshes like Istio or Envoy to provide networking and security enforcement without the need for sidecar proxies. By using eBPF, Cilium can perform many of the functions traditionally handled by service meshes more efficiently, reducing the overhead typically associated with sidecar containers.
6. Native Kubernetes Integration: Cilium is fully compatible with Kubernetes and supports essential features such as pod-to-pod communication, service discovery, and load balancing. It can also integrate with Kubernetes-native constructs like Ingress controllers and CoreDNS, making it a versatile and Kubernetes-first networking solution.
7. Multi-Cluster Networking (Cluster Mesh): In addition to single-cluster setups, Cilium allows for seamless inter-cluster networking through its Cluster Mesh feature. This means that multiple Kubernetes clusters can be interconnected, and services or pods from different clusters can communicate as if they were part of the same network. Cilium automates much of the complexity associated with this multi-cluster networking, such as managing IP routing, load balancing, and network policies.

Cilium provides a robust, scalable, and secure solution for networking in Kubernetes environments. By leveraging eBPF, it offers more advanced networking capabilities compared to traditional CNIs, while also improving observability and security. For organizations running cloud-native applications at scale, Cilium’s rich feature set makes it an attractive choice for Kubernetes networking.

eBPF

eBPF (Extended Berkeley Packet Filter) is a powerful feature within the Linux kernel that allows for efficient, programmable control over various kernel activities. It originated from the classical BPF (cBPF), which was developed in 1992 to filter network packets by avoiding unnecessary copying of data from the kernel to userspace. eBPF expands on this concept, providing enhanced functionality and flexibility.

While cBPF only allowed forward jumps and was mainly focused on network packet filtering, eBPF introduced several significant improvements:

• Increased Resources: eBPF provides 10 registers and supports load/store instructions for 1–8 bytes, greatly enhancing its capabilities.
• Control Flow Enhancements: Unlike cBPF, eBPF supports both forward and backward jumps, allowing for loops, though the kernel ensures these loops terminate to maintain system safety.
• Data Structures: eBPF introduced more complex data structures such as hash tables and arrays, allowing programs to handle more sophisticated operations beyond simple packet filtering.

eBPF can be used at various hook points within the kernel, such as at incoming and outgoing network packets, enabling real-time inspection, filtering, and manipulation. This flexibility allows eBPF to perform tasks like packet mangling, forwarding, and encapsulation.

The safety of eBPF programs is guaranteed by an in-kernel verifier, ensuring that the programs are safe to run and won’t crash the system. Additionally, a Just-In-Time (JIT) compiler translates eBPF bytecode into CPU-specific machine code, optimizing performance for native execution.

Modern systems, such as Cilium, leverage eBPF to enhance networking, security, and observability. Cilium can detect kernel features dynamically, using the latest kernel capabilities when available, demonstrating the continuous evolution and expanding use cases of eBPF beyond its original networking focus.

eBPF is widely used at large tech companies, such as Netflix and Palantir, to not only optimize performance by reducing latecy, but also for deep visibility into processes that would otherwise be near impossible. You can read such a technique on the Netflix blog that details how they use eBPF to solve the “noise neighbors” problem. Read more at ebpf.io

Pod Routing

Cilium offers a powerful enhancement to Kubernetes networking by replacing the default kube-proxy component with an eBPF-based solution. The kube-proxy replacement significantly optimizes pod-to-pod communication, service load balancing, and network policy enforcement in Kubernetes environments. Here’s an in-depth look at how pod routing works in Cilium with kube-proxy replacement, using publicly available information and insights from the official Cilium documentation on Cluster Mesh.

Direct-routing mode

Tunneling mode

What Is Kube-Proxy and Why Replace It?

In a standard Kubernetes setup, kube-proxy is responsible for implementing service networking by managing IP tables or IPVS rules that define how traffic flows between pods, services, and external clients. While kube-proxy is functional, it has limitations, particularly around scalability and performance. Kube-proxy uses a user-space model to route traffic, which introduces overhead in terms of latency and CPU utilization.

Cilium’s kube-proxy replacement addresses these issues by using eBPF to handle networking functions directly in the Linux kernel, removing the need for iptables-based routing. This results in faster, more efficient pod-to-pod communication, and improved load balancing.

When using Cilium with kube-proxy replacement, the primary difference in pod routing is that service load balancing and traffic routing are handled entirely in the kernel using eBPF, without relying on kube-proxy’s traditional user-space or iptables mechanism. Here’s how it works:

Pod-to-Pod Communication

In Kubernetes, when a pod wants to communicate with another pod, it typically does so by referencing the service that fronts the destination pod. With Cilium’s kube-proxy replacement:

eBPF Programs in the Kernel: Cilium installs eBPF programs directly in the Linux kernel to handle all traffic flows. When a packet originates from a pod, Cilium’s eBPF program intercepts the packet at the kernel level, determining the most efficient route based on the pod’s destination. Direct Routing via Cilium BPF Maps: Cilium maintains an internal eBPF map that keeps track of all pod IPs and their locations. This allows Cilium to bypass the kube-proxy and the entire iptables layer, performing direct routing to the destination pod. It can route the packet to the correct pod without going through a complex chain of network rules. Low-Latency Routing: The direct routing enabled by eBPF eliminates unnecessary hops or processing delays introduced by user-space proxies, significantly reducing latency and overhead.

Cross-Node Traffic Routing

In multi-node Kubernetes clusters, traffic often needs to be routed between pods located on different nodes. Cilium, using eBPF and kube-proxy replacement, handles cross-node traffic in a much more efficient way:

NodePort Without Kube-Proxy: Cilium supports NodePort functionality even without kube-proxy. When traffic hits a NodePort, Cilium can directly route the packet to the correct backend pod across nodes using the eBPF map. This ensures that NodePort services continue to function as expected, with reduced latency. Cilium BPF Node Encapsulation: For cross-node traffic, Cilium leverages an eBPF-based node encapsulation mechanism (using VXLAN or Geneve) to route traffic efficiently between nodes. This allows pods on different nodes to communicate as if they were on the same node, maintaining network transparency across the cluster.

Benefits of Cilium’s Kube-Proxy Replacement

Improved Performance: By bypassing kube-proxy and handling traffic directly in the kernel, Cilium reduces the overhead associated with user-space proxies and iptables rules. This results in lower latency, reduced CPU usage, and faster pod-to-pod communication.

Scalability: Traditional kube-proxy setups can struggle with scalability, especially in large clusters with thousands of services. Cilium’s eBPF-based approach scales much better because it does not require the management of large sets of iptables rules. Instead, eBPF maps are highly efficient and can handle large-scale clusters with ease.

Enhanced Observability and Security: Cilium’s eBPF programs provide rich observability into network traffic, enabling better monitoring and debugging. This also allows for advanced security features, such as network policies that operate at both the layer 3/4 and layer 7 levels, something not possible with kube-proxy.

Seamless Multi-Cluster Networking: With Cilium Cluster Mesh, services and pods across multiple clusters are seamlessly integrated. Cilium handles routing between clusters, using eBPF to efficiently direct traffic between nodes and services regardless of cluster boundaries.

Cluster Mesh

Cilium Cluster Mesh is a powerful solution for multi-cluster environments, providing seamless networking across Kubernetes clusters. With the introduction of Helm-based configuration in Cilium 1.14.0, it has become significantly easier to automate Cluster Mesh setups, making it a viable solution even for environments that require frequent changes or updates. Integrating Cilium Cluster Mesh with a GitOps workflow allows for continuous, declarative management of your cluster networking, making it adaptable to both test and production environments.

Until recently, configuring Cilium Cluster Mesh required several manual steps, such as creating and copying certificates between clusters to establish secure connections. However, starting from Cilium version 1.14.0, administrators can configure Cluster Mesh using a GitOps approach by leveraging the Cilium Helm chart. This eliminates the need for manual certificate distribution or bash scripting. By installing the Cilium Helm chart with the correct values, Cluster Mesh becomes operational immediately.

This article outlines the steps to deploy and configure interconnected Kubernetes clusters with Cilium Cluster Mesh, using the Cilium Helm Chart (version 1.16.1).

Global Services

The Global Services feature of Cilium Cluster Mesh allows services deployed across multiple Kubernetes clusters to be exposed under a single DNS name or IP address, enabling pods in one cluster to seamlessly access services in another cluster. This creates a unified service discovery mechanism across clusters, providing seamless connectivity and availability of services, no matter where they are physically located.

In a multi-cluster setup, this feature provides significant advantages, particularly for improving resiliency and high availability (HA) in the following scenarios:

Single Availability Zone (Single AZ)

In a single AZ, Cilium’s Global Services feature enhances resilience by enabling load balancing and failover capabilities between clusters. Even though both clusters reside within the same AZ, the Global Services feature ensures that the workload is distributed across the clusters to prevent single points of failure. If one cluster experiences an issue (such as a node failure or a pod crash), traffic can automatically be redirected to another cluster where the service is available. This ensures service continuity, even within a single AZ environment.

How It Works:

• Global services rely on Cilium’s service load-balancing capabilities, which utilize eBPF to distribute traffic at the kernel level.
• A service deployed in one cluster (Cluster A) can be made available in another cluster (Cluster B), and vice versa.
• DNS-based Service Discovery: Cilium integrates with Kubernetes’ service discovery (via CoreDNS) to provide a unified DNS name that routes requests to all instances of a service across clusters. Pods querying the service by DNS get directed to any of the available service instances, regardless of the cluster.
• If a pod in Cluster A tries to access a service that is unavailable within its own cluster, Cilium automatically reroutes the request to the healthy service endpoint in Cluster B.

Benefits:

• Resiliency: If a cluster experiences service degradation, traffic is routed to another healthy cluster without downtime.
• Load Balancing: Requests can be balanced across all clusters, ensuring optimal resource utilization and performance.

Multi-AZ (Multiple Availability Zones)

In a multi-AZ setup, clusters are spread across different availability zones within the same region. The Global Services feature of Cilium Cluster Mesh provides cross-AZ failover capabilities, enhancing resiliency against the failure of an entire AZ, which is a more significant failure scenario compared to intra-cluster or single-node failures.

How It Works:

• Similar to the single AZ scenario, Global Services allow traffic to be load-balanced between clusters in different AZs.
• In case one AZ becomes completely unavailable due to a failure or maintenance activity, traffic is automatically directed to services in clusters located in the healthy AZs.
• eBPF-driven Load Balancing ensures minimal latency by selecting the most optimal service endpoint, regardless of whether it’s in the same AZ or a different AZ.
• DNS-based service discovery continues to work across AZs, and clients are oblivious to the underlying infrastructure details, as they simply use the same DNS name to access services.

Benefits:

• High Availability: If an entire AZ goes down, the Global Services feature ensures that services remain available in other AZs.
• Fault Tolerance: Traffic is intelligently routed between AZs, avoiding the risk of downtime even in major failure scenarios.
• Load Balancing Across AZs: Distributing traffic across multiple AZs ensures better resource utilization and performance across the entire infrastructure.

Multi-Region

In a multi-region setup, clusters are distributed across entirely different geographical regions. This architecture can provide the highest level of availability, as regions are isolated from each other in terms of geography, power, networking, and other factors. Cilium Cluster Mesh, with its Global Services feature, supports cross-region service discovery and load balancing, ensuring resilience and disaster recovery in case of large-scale regional failures.

How It Works:

• Similar to multi-AZ scenarios, the Global Services feature allows services to be exposed across clusters that are deployed in different regions.
• A pod in one region (Region A) can seamlessly communicate with a service in another region (Region B), thanks to the unified service discovery and routing capabilities of Cilium.
• Traffic is routed between regions based on DNS responses and service load balancing.
• Geo-Proximity Routing: Depending on the service configuration, Cilium can prefer service endpoints within the same region (to reduce latency) and only route to another region in case of service unavailability within the local region.
• Global services in a multi-region setup also enable disaster recovery strategies where traffic can be entirely re-routed to another region in case of catastrophic failure in one region.

Benefits:

• Resiliency Across Regions: Even if an entire region becomes unavailable, the Global Services feature allows traffic to be routed to another region, ensuring business continuity.
• Geographic Load Balancing: Cilium can balance traffic not only across clusters but also across regions, allowing for reduced latency by selecting the nearest service endpoint.
• Disaster Recovery: Multi-region setups ensure that a catastrophic event affecting one region doesn’t result in a complete service outage. Cilium seamlessly reroutes traffic to clusters in unaffected regions.

Service Failover and Traffic Management

One of the core strengths of Cilium’s Global Services feature is its intelligent traffic management and automatic failover. By using eBPF at the kernel level, Cilium can quickly detect failures and reroute traffic without introducing significant latency or downtime. Additionally, the multi-cluster architecture allows for sophisticated traffic shaping policies, such as:

• Weighted Load Balancing: Distribute traffic unevenly across clusters or regions based on their capacity or health.
• Service Affinity: Prefer local service instances over remote ones to reduce latency while still maintaining availability if the local instance fails.

Summary of Benefits

The Global Services feature of Cilium Cluster Mesh plays a critical role in improving resiliency and high availability across various cluster architectures, including single AZ, multi-AZ, and multi-region deployments. Here’s a summary of how it achieves this:

• Resilience: Global Services ensures that service requests are not limited to a single cluster. In case of cluster or AZ failure, services remain accessible by rerouting traffic to healthy clusters.
• High Availability: With service failover across AZs and regions, Cilium ensures that services remain up even in the face of significant infrastructure failures.
• Performance Optimization: Cilium’s eBPF-powered load balancing intelligently routes traffic based on proximity and service health, ensuring optimal performance across clusters and regions.
• Disaster Recovery: In multi-region deployments, Cilium allows services to recover from a complete regional failure by rerouting traffic to healthy regions, maintaining business continuity.

Service Discovery

In Kubernetes, service discovery is the process through which one service can discover and communicate with another, usually through DNS. Cilium integrates seamlessly with Kubernetes’ built-in service discovery mechanisms while extending its capabilities through eBPF.

Key Components of Service Discovery in Cilium:

• Kube-DNS Integration: Cilium integrates with Kubernetes’ native DNS system (typically CoreDNS) to allow pods to resolve services using standard DNS names (e.g., service.namespace.svc.cluster.local). This integration ensures that Kubernetes’ service discovery process remains consistent with Cilium’s networking layer.
• eBPF-based DNS Inspection: Cilium leverages eBPF to inspect DNS traffic in the kernel, enabling faster service discovery and the ability to enforce network policies based on DNS lookups. This allows for DNS-aware security policies, where network access can be restricted or permitted based on the domain name being queried.
• Multi-Cluster Service Discovery: In multi-cluster environments with Cilium Cluster Mesh, Cilium extends service discovery across clusters. Services from different clusters can be discovered and accessed seamlessly as if they were part of the same cluster. Cilium automatically handles the cross-cluster service discovery using DNS-based service records (e.g., for Global Services in Cluster Mesh).
• Service Awareness: Cilium can provide deeper visibility into service-to-service traffic flows by using Hubble, its observability platform. Hubble allows operators to see how services are discovered and how traffic flows between them in real time, making it easier to debug network issues or monitor service-level performance.

DNS Resolution Example:

• When a pod queries a service (e.g., myapp.default.svc.cluster.local), Cilium captures the DNS request and resolves it via CoreDNS.
• If the service spans multiple clusters (in the case of Cluster Mesh), Cilium returns the correct IP of the service, potentially in a remote cluster, using DNS-based load balancing.

Shared Ingress

Now let’s explore the concept of configuring the NGINX ingress controller to balance traffic across all available pods deployed across multiple Kubernetes clusters, interconnected via Cilium Cluster Mesh. A key aspect of this discussion will be how CiliumEndpoints help streamline traffic routing and enable enhanced visibility of network traffic in the cluster.

By default, most Kubernetes ingress controllers route traffic to services by directly using the list of endpoints (i.e., pod IPs) associated with a service. This allows traffic to bypass the Kubernetes service IP and directly reach the pods, optimizing performance and minimizing bottlenecks. However, this approach can be limiting when working with shared services across multiple clusters in a Cilium Cluster Mesh:

• If the deployment is scaled down in one cluster, the ingress controller will see that no pod endpoints are available in the local cluster, and it will fail to route traffic correctly, causing the application to become unavailable.
• Traditional ingress controllers like Traefik rely entirely on endpoints, meaning they can’t route traffic to other clusters when no local endpoints exist.

NGINX Ingress Controller and Shared Services

The NGINX Ingress Controller, however, offers a solution to this problem through the use of the nginx.ingress.kubernetes.io/service-upstream annotation. This annotation configures the ingress controller to route traffic to the service IP instead of directly to the pod IPs (endpoints). In a multi-cluster setup using Cilium Cluster Mesh, this feature is particularly valuable because it allows the NGINX ingress controller to balance traffic across services running in different clusters, even when no local pod endpoints exist.

Role of CiliumEndpoints in Shared Ingress

CiliumEndpoints are an essential part of Cilium’s architecture, providing detailed metadata about the state of pods and their networking configuration. Each pod is represented as a CiliumEndpoint object, which stores critical information about the pod’s identity, IP addresses, policies, and connection status. Cilium uses these objects to efficiently manage traffic routing and enforce network security policies.

In the context of shared ingress and NGINX, CiliumEndpoints enhance visibility and provide deeper control over pod-to-pod and service-to-service communication, especially in a multi-cluster environment.

Key Points About CiliumEndpoints:

• Network Awareness: CiliumEndpoints provide a rich set of metadata about each pod’s networking state, enabling advanced traffic routing decisions based on security policies, IP management, and connectivity status.
• Endpoint Visibility: Unlike traditional Kubernetes endpoints that only expose basic IP addresses, CiliumEndpoints offer more granular insight into the pod’s status, facilitating more efficient traffic routing and policy enforcement across clusters.
• Cross-Cluster Integration: Cilium Cluster Mesh can use CiliumEndpoints to track the network state of pods across different clusters, ensuring that NGINX can balance traffic to services running in other clusters, even when local endpoints are unavailable.

By leveraging CiliumEndpoints, Cilium’s networking layer can provide ingress controllers with enhanced traffic intelligence. This is particularly useful when dealing with cross-cluster traffic routing, as CiliumEndpoints ensure that each cluster is aware of pod and service availability in other clusters, further optimizing the way ingress controllers route traffic.

Conclusion

This article has provided a high-level overview of the capabilities and advantages of using Cilium Cluster Mesh within Kubernetes environments. By leveraging the cutting-edge eBPF technology, Cilium not only simplifies multi-cluster networking but also enhances security and observability across clusters.

While today’s discussion focused on the theoretical framework and potential benefits, a forthcoming post will delve into the practical aspects. We will provide a step-by-step guide (with accompanying code) on the actual implementation and a live demonstration of Cilium Cluster Mesh in action. This upcoming post will ensure you have the hands-on knowledge and tools to effectively deploy Cilium Cluster Mesh, empowering you to fully harness its capabilities in your own Kubernetes deployments.

Stay tuned for our next installment, where we’ll transition from theory to practice, demonstrating the tangible impacts of Cilium’s advanced networking features in real-world scenarios.

Title: Use Vagrant to provision and install RKE2 and Cilium
Author: Mitch Murphy
Date: 2024-06-09

• Use Vagrant to provision and install RKE2 and Cilium
  • Introduction
    • Pros
    • Cons
  • Provision RKE2
    • Benefits of Using Vagrant for Local Kubernetes Cluster:
    • Challenges:
  • Installation
    • Install Virtualization Provider
    • Install Vagrant
    • Install Required Plugins (Optional)
    • Ensure Kernel Modules for Virtualization are Loaded
    • Verify Installation
  • Configuration

Introduction

Vagrant is an open-source tool designed to create and manage virtualized development environments. It allows developers to define and configure lightweight, reproducible, and portable virtual environments using a simple configuration file called a Vagrantfile. Vagrant integrates with various virtualization technologies like VirtualBox, VMware, and Docker, enabling users to set up and tear down development environments quickly and consistently. By using Vagrant, developers can ensure that their development environments are identical across different machines, reducing the “works on my machine” problem and facilitating collaboration among team members.

Pros

• Consistency and Reproducibility: Vagrant ensures that development environments are consistent across all team members’ machines, reducing discrepancies between development, testing, and production environments.
• Easy Setup and Configuration: With Vagrant, setting up a development environment is straightforward and automated, often requiring just a single command to get started.
• Isolation: Vagrant environments are isolated from the host system, minimizing the risk of conflicts with other applications and dependencies.
• Flexibility: Vagrant supports multiple virtualization providers and can be used with various provisioning tools like Ansible, Puppet, and Chef, providing flexibility in how environments are configured and managed.

Cons

• Resource Intensive: Running multiple Vagrant environments can be resource-intensive, requiring significant CPU, memory, and disk space, especially on less powerful machines.
• Learning Curve: For new users, there can be a learning curve associated with understanding Vagrant and its configuration files, particularly when integrating with complex provisioning tools.
• Performance Overheads: Virtualized environments may introduce performance overheads compared to running applications natively on the host machine, which can impact development and testing speeds.
• Dependency on Virtualization Providers: Vagrant relies on external virtualization providers, and any issues or limitations with these providers can affect the performance and reliability of Vagrant environments.

Provision RKE2

Vagrant can be utilized to configure and provision a local Kubernetes cluster by automating the setup of virtual machines (VMs) that act as the nodes of the cluster. Here’s a step-by-step outline of how this can be achieved:

1. Define Vagrantfile:
   • Create a Vagrantfile to define the configuration for the VMs. This file specifies the number of VMs, their resources (CPU, memory, etc.), and the base image to use.
   • Example configuration might include one VM as the master node and two or more VMs as worker nodes.
2. Provisioning the VMs:
   • Use provisioning scripts or configuration management tools (like Ansible, Puppet, or Chef) within the Vagrantfile to install necessary dependencies on the VMs. This typically includes Docker, kubeadm, kubectl, and kubelet.
   • The provisioning script also sets up networking between the VMs to ensure they can communicate with each other.
3. Initialize the Kubernetes Master Node:
   • After the VMs are up and running, SSH into the master node and run kubeadm init to initialize the Kubernetes master. This command sets up the control plane and generates a command (with a token) to join the worker nodes to the cluster.
4. Configure kubectl:
   • Configure the kubectl command-line tool on the master node to interact with the cluster by copying the kubeconfig file to the appropriate location.
5. Join Worker Nodes:
   • SSH into each worker node and run the join command provided by kubeadm init. This command joins the worker nodes to the Kubernetes cluster, making them part of the cluster’s node pool.
6. Network Setup:
   • Deploy a network plugin (such as Flannel, Calico, or Weave) to handle pod networking. This can be done by applying the relevant YAML file using kubectl apply -f .
7. Verify the Cluster: -Once all nodes are joined and the network plugin is configured, verify the cluster status by running kubectl get nodes on the master node. All nodes should be in a “Ready” state.

Benefits of Using Vagrant for Local Kubernetes Cluster:

• Reproducibility: Easily recreate the cluster environment for consistent development and testing.
• Isolation: The cluster runs in isolated VMs, preventing interference with the host system.
• Automation: Automated setup reduces manual configuration time and potential errors.

Challenges:

• Resource Consumption: Multiple VMs can be resource-intensive.
• Configuration Complexity: Initial setup and configuration can be complex, especially for users unfamiliar with Vagrant or Kubernetes.

Installation

For the purposes of this tutorial, we will be using a Debian based system (Pop!_OS). To run Vagrant on a Debian-based system, you need to ensure that several prerequisites are met. Here is a comprehensive list of the prerequisites and steps to get Vagrant up and running:

Install Virtualization Provider

Vagrant requires a virtualization provider to create and manage virtual machines. The most commonly used provider is VirtualBox, but you can also use others like VMware or Docker. Here, we will focus on VirtualBox.

sudo apt update
sudo apt install -y virtualbox

Install Vagrant

Next, you need to install Vagrant. You can download the .deb package from the official Vagrant website or use the package manager.

curl -fsSL https://apt.releases.hashicorp.com/gpg | sudo apt-key add -
sudo apt-add-repository "deb [arch=amd64] https://apt.releases.hashicorp.com $(lsb_release -cs) main"
sudo apt update
sudo apt install -y vagrant

There are a few other packages that you will need:

sudo apt update && sudo apt install -y libvirt-dev qemu-kvm libvirt-daemon-system bridge-utils virtinst libvirt-daemon virt-manager libvirt-doc ruby-libvirt qemu libvirt-clients ebtables dnsmasq-base libxslt-dev libxml2-dev zlib1g-dev ruby-dev libguestfs-tools

Install Required Plugins (Optional)

Depending on your use case, you might need to install additional Vagrant plugins. For example, if you’re using a different provider or need specific functionalities.

PLUGINS=vagrant-vboxmanage vagrant-vbguest vagrant-libvirt vagrant-reload
for PLUGIN in $PLUGINS; do
  vagrant plugin install $PLUGIN
done

Ensure Kernel Modules for Virtualization are Loaded

For VirtualBox to work correctly, ensure that the necessary kernel modules are loaded.

sudo modprobe vboxdrv
sudo modprobe vboxnetflt
sudo modprobe vboxnetadp

Verify Installation

To confirm that Vagrant and VirtualBox are installed correctly, you can check their versions.

vagrant --version
vboxmanage --version

Configuration

I have prepared a simple Vagrantfile for you to get started with standing up an RKE2 v1.30.1 cluster, Cilium, K9s and Helm. Stay tuned for more details regarding this configuration (this will more than likely be reflected in the README of the repository). Link to repository.

Title: Update Cilium to work with Istio and create a self-signed certificate for Istio
Author: Mitch Murphy
Date: 2024-03-16

• Update Cilium to work with Istio and create a self-signed certificate for Istio
  • Introduction
  • Pre-requisites
  • Update Cilium chart
  • Create Istio Namespace
  • Create self-signed certificate

Introduction

In this post we are going to update Cilium to work with Istio and create self-signed certificates for Istio.

As discussed in my previous post, Cilium, when replaceing kube-proxy, provides very powerful features which can increase performance for large Kubernetes clusters. While Cilium does offer a service mesh (with mTLS), it is still in beta and other solutions such as Istio are quite mature and offer a lot of capabilities.

When utilizing alternative networking infrastructure, the underlying Cilium eBPF implementation that replaces kube-proxy may inadvertently reveal itself and lead to unexpected behaviors. This becomes evident when attempting to integrate Istio service mesh with Cilium’s kube-proxy replacement, as the default replacement of kube-proxy can disrupt Istio’s functionality.

Pre-requisites

• RKE2
• Cilium
• Istio
• Helm
• Make
• Kubectl

This blog post had a profound impact on the creation of the one you’re reading now.

Update Cilium chart

helm upgrade cilium \
    --namespace kube-system \
    --set ipam.mode=kubernetes \
    --set socketLB.hostNamespaceOnly=true

Create Istio Namespace

kubectl create namespace istio-system

Note that you can also specify the creation of the namespace in the below script.

Create self-signed certificate

For this step we are going to use a few scripts from the istio repository. These tools can be found here. The first step is to create the namespace and then generate the self-signed certificate that Istio will be using (here we are assuming 3 clusters). The following script will create the certificates and then create the secrets in each cluster (you must have the kubeconfig for each cluster in order to run the script).

You can find the scripts at the following location: istio-tools, the specific script is make-certs.sh.

TODO: real world example(s)

Title: Bootstrap cluster with Flux: the GitOps way
Author: Mitch Murphy
Date: 2023-12-27

• Bootstrap cluster with Flux: the GitOps way
• Introduction
• Prerequisites
• Install Flux
  • Bootstrap cluster with Flux
• Components
• Conclusion

Introduction

What is GitOps? GitOps is a way to do Continuous Delivery, it works by using Git as a single source of truth for declarative infrastructure and applications. With GitOps, we can make changes to our infrastructure and applications by making pull requests against our Git repository. These changes are then automatically applied to our cluster by a GitOps operator. This allows us to manage our infrastructure and applications in a declarative way, using the same Git workflow that we use for application code.

Flux CD is an open-source continuous delivery (CD) tool designed for Kubernetes applications. It automates the deployment and lifecycle management of applications running on Kubernetes clusters. Flux CD focuses on ensuring that the desired state of your Kubernetes resources matches the actual state, thus enabling GitOps practices.

Key features and concepts of Flux CD include:

1. GitOps: Flux CD follows the GitOps operational model, where the desired state of your Kubernetes cluster is declared in a Git repository. Changes to the cluster are made by updating the Git repository, and Flux CD automatically applies those changes to the cluster, ensuring that the actual state converges with the declared state.
2. Automation: Flux CD automates the deployment of applications by synchronizing the Kubernetes manifests stored in a Git repository with the resources running in the cluster. This allows for a declarative approach to managing infrastructure and applications.
3. Continuous Delivery: Flux CD is commonly used for implementing continuous delivery pipelines on Kubernetes. When changes are pushed to the Git repository, Flux CD automatically detects them and triggers the deployment process.
4. Helm Support: Flux CD has built-in support for Helm charts, a package manager for Kubernetes applications. It can automatically sync and apply changes to Helm releases based on the charts stored in the Git repository.
5. Multi-Environment Support: Flux CD supports multi-environment setups, allowing you to manage and deploy applications across different clusters or namespaces.
6. Notifications: Flux CD provides notifications and alerts to keep users informed about the state of deployments and any issues that may arise.

Flux CD is often used in conjunction with other tools and practices that adhere to the GitOps methodology, such as version control systems (like Git), CI/CD pipelines, and Kubernetes operators.

Prerequisites

Before you begin this guide, you’ll need the following:

1. A Kubernetes cluster running RKE2
2. Flux CLI
3. Flux installed on your cluster
4. A GitHub repository to store your cluster configuration

Install Flux

We will be using Flux CD to bootstrap our cluster. If you haven’t already, you will need to install Flux CD on your cluster. You can find a guide on how to do this here.

Bootstrap cluster with Flux

Once you have Flux CD installed, you are ready to boostrap your cluster. We will be using the Flux CLI to bootstrap our cluster. The Flux CLI is a command-line tool that allows you to interact with Flux CD. It can be used to create and manage Flux CD resources, such as Git repositories, Helm releases, and Kubernetes resources. We will be following this guide.

flux bootstrap github \
    --owner=mkm29 \
    --repository=kubula \
    —-path=clusters/mgmt \
    --token-auth \
    --personal \
    --branch=main

The bootstrap github command creates the GitHub repository if it doesn’t exists and commits the Flux manifests to the specified branch. Then it configures the target cluster to synchronize with that repository.

Components

Flux CD consists of several components that work together to provide the desired functionality. These components include:

1. Flux: The core component of Flux CD, Flux is a Kubernetes operator that runs on the cluster and manages the deployment of applications. It monitors the Git repository for changes and automatically applies them to the cluster.
2. Helm Operator: The Helm Operator is a Kubernetes operator that manages Helm releases. It can automatically sync and apply changes to Helm releases based on the charts stored in the Git repository.
3. Memcached: Memcached is a distributed memory object caching system that is used by Flux CD to store and retrieve data.
4. Notification Controller: The Notification Controller is a Kubernetes controller that sends notifications and alerts to users about the state of deployments and any issues that may arise.
5. Image Automation Controller: The Image Automation Controller is a Kubernetes controller that automates the deployment of container images. It can automatically pull new images from a registry and update the corresponding Kubernetes resources.
6. Kustomize Controller: The Kustomize Controller is a Kubernetes controller that manages Kustomize resources. It can automatically sync and apply changes to Kustomize resources based on the manifests stored in the Git repository.
7. Source Controller: The Source Controller is a Kubernetes controller that manages Git repositories. It can automatically sync and apply changes to Git repositories based on the manifests stored in the Git repository.

We are going to use Flux CD to bootstrap a minimal set of Kubernetes components, including a Prometheus monitoring stack, persistent storage via Longhorn and Argo CD. The structure can be found here

Conclusion

In this guide, we have learned how to bootstrap a Kubernetes cluster with Flux CD using GitOps. We have also learned about the key features and concepts of Flux CD, as well as its components. We have also learned how to use the Flux CLI to bootstrap our cluster. Finally, we have learned how to use Flux CD to bootstrap a minimal set of Kubernetes components, including a Prometheus monitoring stack, persistent storage via Longhorn and Argo CD.

Title: Adding Persistent Storage to RKE2 with Longhorn
Author: Mitch Murphy
Date: 2023-12-27

• Adding Persistent Storage to RKE2 with Longhorn
• Introduction
• Prerequisites
• Install Longhorn
• Configure Longhorn
  • Virtual Hard Disk
• Conclusion

Introduction

What is Longhorn? Longhorn refers to a project that provides persistent storage for Kubernetes. Longhorn aims to simplify and enhance the management of persistent storage in Kubernetes clusters. It is an open-source distributed block storage system that can be deployed as a containerized application.

Some key features of Longhorn include:

1. Distributed Storage: Longhorn distributes block storage across multiple nodes in a Kubernetes cluster, providing redundancy and fault tolerance.
2. Snapshot and Backup: It supports snapshots, allowing you to capture the state of your persistent volumes at a specific point in time. Additionally, Longhorn includes features for backup and restore operations.
3. Replication and Synchronization: Longhorn replicates data across nodes to ensure data integrity and provides mechanisms for data synchronization.
4. User Interface: Longhorn includes a user-friendly web-based UI that allows users to manage and monitor their storage resources easily.
5. Kubernetes Native: Longhorn is designed to work seamlessly with Kubernetes, integrating into the Kubernetes control plane and leveraging custom resources to define and manage persistent volumes.

It’s worth noting that the landscape of Kubernetes storage solutions may evolve, and new developments may have occurred since my last update. Therefore, I recommend checking the official Longhorn project documentation or other reliable sources for the latest information on Longhorn and its features.

Prerequisites

Before you begin this guide, you’ll need the following:

1. A Kubernetes cluster running RKE2
2. Flux CD installed on your cluster
3. A GitHub repository to store your cluster configuration
4. Storage provisioned, mounted and formatted on each node in your cluster
   • For this guide, I will be attaching a Virtual Hard Disk to each node/virtual machine in Hyper-V, mounted at /mnt/storage
   • After attaching the VHD to each node, you will need to format the disk with a filesystem. For this guide, I will be using ext4
   • Stay tuned as I will be writing a guide on how to automate this process using Ansible

Install Longhorn

We will be using Flux CD to install Longhorn on our cluster. If you haven’t already, you will need to install Flux CD on your cluster. You can find a guide on how to do this here. Once you have Flux CD installed, you will need to create a HelmRepository resource to point to the Longhorn Helm chart as well as a HelmRelease resource to install Longhorn on your cluster. Please see the structure in this repo

Configure Longhorn

Virtual Hard Disk

First, we need to attach a Virtual Hard Disk to each node/virtual machine in Hyper-V. This will be used as the storage for Longhorn. To do this, you will need to:

1. Create a Virtual Hard Disk in Hyper-V (follow this guide)
2. Attach it to each node/virtual machine in your cluster
3. Format/partition the disk with a filesystem. For this guide, I will be using ext4
4. Mount the disk to /mnt/storage on each node/virtual machine in your cluster. Make sure to add an entry to /etc/fstab so that the disk is mounted on boot
   • echo "/dev/sdb1 /mnt/storage ext4 defaults 0 0" >> /etc/fstab
5. You can find a guide on how to do this here.

As we will be mounting our storage to /mnt/storage you must set the defaultSettings.defaultDataPath to /mnt/storage in the HelmRelease resource. This will ensure that Longhorn will use the storage we have provisioned on each node in our cluster.

Note: that this will create a default StorageClass on our Kubernetes cluster. This means that you do not need to specify the StorageClass when you create PersistentVolumeClaims, as the default StorageClass will be used to provision the PersistentVolume.

Conclusion

In this guide, we have installed Longhorn on our RKE2 cluster using Flux CD. We have also configured Longhorn to use the storage we have provisioned on each node in our cluster. This means that we can now create PersistentVolumeClaims without specifying the StorageClass as the default StorageClass will be used to provision the PersistentVolume.

Title: Configuring ArgoCD RBAC
Author: Mitch Murphy
Date: 2023-12-03

• How to Guide: Configuring ArgoCD RBAC
  • Introduction
  • Prerequisites
  • Add Cluster
  • RBAC Configuration
  • Deploying Application

Introduction

ArgoCD is a great tool for managing your Kubernetes clusters. It allows you to deploy applications to your clusters and keep them in sync. It also allows you to manage your clusters and applications through a GitOps workflow. This means that you can manage your clusters and applications through Git and have ArgoCD keep them in sync. This is great for managing multiple clusters and applications. We can utilize ArgoCD for managing deployments to multiple clusters by multiple teams. This is where RBAC comes in. RBAC stands for Role Based Access Control. It allows us to define roles and permissions for users and groups. This allows us to control who can access what resources in our clusters. This is great for managing access to our clusters and applications.

Prerequisites

• Kubernetes Cluster(s)
• ArgoCD
• Helm
• Keycloak

Add Cluster

First, lets add a new cluster to ArgoCD. For this, we need to use the CLI:

kubectl -n argocd port-forward svc/argocd-server 8080:80
argocd login localhost:8080 --username admin --password <PASSWD> --insecure

Now, we can add our cluster (the name must be the name of the context in your kubeconfig):

argocd cluster add <CLUSTER_NAME>

We will be using the second cluster that we previously created and added to our Cilium cluster mesh (smig-cluster2).

RBAC Configuration

First, follow this guide on how to configure ArgoCD to use Keycloak for authentication. This will allow us to use Keycloak for authentication and authorization. You must create a group named smig2-team and then add user(s) to this group. This will be the group that we will use for RBAC.

Now we need to update the ArgoCD Helm chart to enable RBAC. We can do this by specifying a values file:

cat <<EOF > argocd-rbac-values.yaml
configs:
  oidc.config: |
    name: SmigAdmins
    issuer: https://auth.smigula.io/realms/smig/
    clientID: argocd
    clientSecret: $oidc.azuread.clientSecret
    requestedIDTokenClaims:
      groups:
        essential: true
    requestedScopes:
      - openid
      - profile
      - email
      - groups
  rbac:
    policy.csv |
      p, role:smig-admin, applications, *, */*, allow
      p, role:smig-admin, clusters, *, smig-cluster2, allow
      p, role:smig-admin, repositories, *, *, allow
      p, role:smig-admin, logs, get, *, allow
      p, role:smig-admin, exec, create, */*, allow
      g, smig2-team, role:smig-admin
EOF

helm upgrade argocd argo/argo-cd -f argocd-rbac-values.yaml

Deploying Application

Now, we can deploy an application to our secondary cluster using ArgoCD. We will be deploying the guestbook. First, we need to create a new project as an admin:

argocd proj create smig2-team \
  --description "Smigula Team 2" \
  --dest https://192.168.7.31:6443,default \
  --allow-cluster-resource \
  --allow-namespaced-resource

Now login to the ArgoCD web UI as a user in the smig2-team group. You should see the smig2-team project. Click on it and then click New App. Fill out the form with the following information:

• Application Name: guestbook
• Repository URL: https://github.com/argoproj/argocd-example-apps.git
• Revision: HEAD
• Path: helm-guestbook

For Destination, set cluster URL to https://192.168.7.31:6443 and namespace to default. Click Create. Now, you should see the application in the smig2-team project. Click on it and then click Sync. This will deploy the application to the cluster. You can now access the application by port forwarding the service:

kubectl port-forward svc/guestbook-ui 8080:80

Now, you can access the application at http://localhost:8080. You can also access the application through the ArgoCD web UI by clicking on the application and then clicking Open in new tab. You should see the guestbook application. You can now make changes to the application and ArgoCD will keep it in sync. You can also add more users to the smig2-team group in Keycloak and they will be able to access the application.

Title: Boostrapping RKE2 with ArgoCD
Author: Mitch Murphy
Date: 2023-12-02

Introduction

This guide will walk you through the steps to bootstrap an RKE2 cluster with ArgoCD. This guide assumes you have a working RKE2 cluster and ArgoCD instance. If you do not have these, please refer to my previous post on installing RKE2 with Cilium: here

Table of Contents

• How to Guide: Boostrapping RKE2 with ArgoCD, Prometheus, and Cert-Manager
  • Introduction
  • Table of Contents
  • Prerequisites
  • Steps
    • 1. Create a new namespace for ArgoCD
    • 2. ArgoCD
      • Argo Components
    • 2.1 Install Helm
    • 3. Login to ArgoCD
    • 4. Update Password
    • 5. Add Repositories
    • 6. Create a new project
    • 7. Install Prometheus
      • Using ArgoCD
    • Cert-Manager
      • CM Components
      • Install Cert-Manager
      • Cluster Issuer
    • Ingress Controller

Prerequisites

• RKE2 Cluster
• kubectl
• ssh access to RKE2 cluster

Steps

1. Create a new namespace for ArgoCD

# first switch to proper context
kubectl config use-context <cluster-name>
kubectl create namespace argocd

2. ArgoCD

Argo CD is a declarative, GitOps continuous delivery tool for Kubernetes applications. In the context of Kubernetes, GitOps is a set of practices that use Git as the source of truth for declarative infrastructure and applications. With Argo CD, you can manage and automate the deployment of your Kubernetes applications using Git repositories as the source of configuration and versioning.

Argo Components

Here are some key aspects of Argo CD in relation to Kubernetes:

• Declarative Configuration: Argo CD uses declarative YAML files to describe the desired state of your Kubernetes applications. These files are typically stored in a Git repository.
• Continuous Delivery (CD): Argo CD automates the deployment of applications to Kubernetes clusters. It continuously monitors the Git repository for changes and ensures that the actual cluster state matches the desired state defined in the Git repository.
• GitOps Workflow: Argo CD follows the GitOps workflow, where the Git repository serves as the single source of truth for both the application configuration and the deployment state. Changes to the Git repository trigger automatic updates to the Kubernetes cluster.
• Application Synchronization: Argo CD maintains the desired state of applications in the Git repository by automatically synchronizing with the current state of the Kubernetes cluster. It detects any discrepancies and reconciles them to ensure that the cluster state matches the configuration in Git.
• Multi-Cluster Support: Argo CD supports managing applications across multiple Kubernetes clusters. This is particularly useful in a multi-environment or multi-region setup.
• Web UI and CLI: Argo CD provides a web-based user interface (UI) and a command-line interface (CLI) for managing and monitoring applications.
• Rollback and Roll Forward: Argo CD allows you to roll back or roll forward to a specific version of your application by changing the desired state in the Git repository.

Using Argo CD simplifies and streamlines the deployment and management of Kubernetes applications, making it easier to maintain consistency and traceability across different environments.

2.1 Install Helm

I prefer to use a separate VM/controller to perform all Helm installations. This is not required, but it is recommended. Note this does not need to be performed on the RKE2 server.

# Use curl to download release and untar
curl -L https://get.helm.sh/helm-v3.13.2-linux-amd64.tar.gz | tar xz
cd linux-amd64
chmod +x helm
sudo mv helm /usr/local/bin/helm
cd ../
rm -rf linux-amd64

Add Argo repo to helm

helm repo add argo https://argoproj.github.io/argo-helm
helm repo update

For this step we are going to use the HelmChart CRD to install ArgoCD. This will allow us to use the HelmRelease CRD to manage the ArgoCD application. You need to add the following file to /var/lib/rancher/rke2/server/manifests/ on the RKE2 server. This will install the HelmRelease CRD and the ArgoCD application.

cat <<EOF > argocd-values.yaml
controller:
  metrics:
    enabled: true
    serviceMonitor:
      enabled: true
      namespace: monitoring-system
      interval: 30s
server:
  service:
    type: ClusterIP
  extensions:
    enabled: true
  metrics:
    enabled: true
    serviceMonitor:
      enabled: true
      namespace: monitoring-system
      interval: 30s
repoServer:
  metrics:
    enabled: true
    serviceMonitor:
      enabled: true
      namespace: monitoring-system
      interval: 30s
applicationSet:
  metrics:
    enabled: true
    serviceMonitor:
      enabled: true
      namespace: monitoring-system
      interval: 30s
notifications:
  metrics:
    enabled: true
    serviceMonitor:
      enabled: true
      namespace: monitoring-system
      interval: 30s
redis:
  metrics:
    enabled: true
    serviceMonitor:
      enabled: true
      namespace: monitoring-system
      interval: 30s
rbac:
  defaultPolicy: 'role:readonly'
dex:
  enabled: false
EOF

Now install the chart:

helm install argocd \
  -n argocd \
  --create-namespace \
  -f argocd-values.yaml argo/argo-cd

Now wait for all pods to be running:

kubectl -n argocd get pods -w

3. Login to ArgoCD

password=$(kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d)
# port forward to argocd server
kubectl -n argocd port-forward svc/argocd-server 8080:80
argocd login localhost:8080 --username admin --password $password --insecure

4. Update Password

argocd account update-password --account admin --current-password $password --new-password <new-password>
# delete initial secret
kubectl -n argocd delete secret argocd-initial-admin-secret

5. Add Repositories

Note steps 5 and 6 have been added to the argocd-setup.sh script.

Let us add the prometheus, jetstack and bitnami repositories to ArgoCD. This will allow us to install applications from these repositories.

argocd repo add https://charts.bitnami.com/bitnami --type helm --name bitnami
argocd repo add https://charts.jetstack.io --type helm --name jetstack
argocd repo add https://kubernetes-charts.storage.googleapis.com --type helm --name stable
argocd repo add https://prometheus-community.github.io/helm-chart --type helm --name prometheus-community

6. Create a new project

argocd proj create monitoring

7. Install Prometheus

I have experienced issues with installing the kube-prometheus-stack chart using ArgoCD. I suspect that the issue is related to the Prometheus CRDs, therefore, I have decided to install the chart using the Helm:

helm repo add prometheus-community https://prometheus-community.github.io/helm-chart
helm repo update
helm install --namespace monitoring-system --create-namespace monitoring prometheus-community/kube-prometheus-stack

Using ArgoCD

argocd app create monitoring --repo https://prometheus-community.github.io/helm-chart \
   --helm-chart kube-prometheus-stack \
   --revision 54.2.0 \
   --dest-namespace monitoring-system \
   --dest-server https://kubernetes.default.svc \
   --project monitoring

Cert-Manager

Cert-manager is a Kubernetes native certificate management controller. Its primary purpose is to automate the management and issuance of TLS certificates within Kubernetes clusters. TLS (Transport Layer Security) certificates are used to secure communication over the web, and they are commonly used to enable HTTPS for web applications.

CM Components

Here are key aspects of cert-manager in relation to Kubernetes:

• Certificate Issuance and Renewal: Cert-manager automates the process of obtaining and renewing TLS certificates from various certificate authorities (CAs) such as Let’s Encrypt. It allows you to define certificate issuers and certificate requests as Kubernetes resources.
• Integration with ACME Protocols: Cert-manager supports the ACME (Automatic Certificate Management Environment) protocol, which is commonly used by Let’s Encrypt. This enables automatic certificate provisioning and renewal.
• Custom Resource Definitions (CRDs): Cert-manager extends Kubernetes by introducing custom resource definitions (CRDs) for defining certificate issuers, certificate requests, and certificates. This allows you to manage certificates as part of your Kubernetes configuration.
• Webhook and Controller Architecture: Cert-manager uses a webhook and controller architecture to interact with Kubernetes resources and respond to changes. The controller monitors the state of certificate-related resources and takes actions to ensure the requested certificates are obtained and renewed as needed.
• Integration with Ingress Controllers: Cert-manager integrates with Kubernetes Ingress controllers, making it easy to secure web applications by automatically provisioning TLS certificates for Ingress resources.
• Support for Multiple Issuers: Cert-manager supports multiple issuers, allowing you to choose different certificate authorities or even self-signed certificates based on the specific requirements of your applications.
• Web UI and CLI: Cert-manager provides a web-based user interface (UI) and a command-line interface (CLI) for managing and monitoring certificates.

By using cert-manager, Kubernetes users can ensure that their applications are using valid and up-to-date TLS certificates without manual intervention. This is crucial for maintaining the security and integrity of communication within a Kubernetes cluster and between external clients and services.

Install Cert-Manager

We will use ArgoCD to manage the installation of cert-manager. You can either use the web UI or the CLI to install the application. I will use the CLI.

echo "Adding cert-manager repo"
argocd repo add https://charts.jetstack.io --type helm --name jetstack
echo "Creating cert-manager namespace"
kubectl create ns cert-manager
echo "Creating cert-manager application"
argocd app create cert-manager --repo https://charts.jetstack.io \
   --helm-chart cert-manager \
   --revision v1.5.3 \
   --dest-namespace cert-manager \
   --dest-server https://kubernetes.default.svc \
   --project cert-manager

Cluster Issuer

In the context of cert-manager and Kubernetes, a ClusterIssuer is a custom resource definition (CRD) used to define a certificate issuer at the cluster level. It is a way to configure and manage certificate authorities (CAs) for obtaining TLS certificates across the entire Kubernetes cluster.

Here are key points about ClusterIssuer:

• Scope: Unlike a regular Issuer, which is specific to a namespace, a ClusterIssuer is designed to have a global scope across the entire Kubernetes cluster. This means that certificates issued by a ClusterIssuer can be used by resources in any namespace.
• Centralized Configuration: ClusterIssuer allows you to define a centralized configuration for a certificate authority, such as the endpoint of the CA server, authentication credentials, and other settings needed to interact with the CA.
• Multiple Issuers: You can have multiple ClusterIssuer resources in a cluster, each configured to interact with a different certificate authority or configuration. This flexibility allows you to use different CAs for different purposes or to meet specific security requirements.
• Used with Cert-manager: ClusterIssuer is part of the cert-manager project and works in conjunction with the cert-manager controller. Cert-manager uses the information specified in a ClusterIssuer resource to manage the lifecycle of TLS certificates issued by the associated CA.

For ease of use, we are going to use letsencrypt as our certificate authority. We will use the ClusterIssuer resource to configure cert-manager to use the Let’s Encrypt staging and production environments.

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: your-email@example.com
    privateKeySecretRef:
      name: letsencrypt-prod
    solvers:
    - http01:
        ingress:
          class: nginx

In this example, the ClusterIssuer is named “letsencrypt-prod,” and it is configured to use the Let’s Encrypt ACME server to issue certificates. The configuration includes the email address for notifications and specifies the http01 challenge for domain validation.

This ClusterIssuer can then be referenced in Certificate resources (another cert-manager CRD) across different namespaces within the cluster. Cert-manager will use the ClusterIssuer configuration to automatically manage the lifecycle of TLS certificates associated with those Certificate resources.

Ingress Controller

An Ingress controller is an application that monitors Ingress resources via the Kubernetes API and updates the configuration of a load balancer in case of any changes. It also handles dynamic configuration changes, such as adding or removing routes, without requiring any manual intervention. We have already configured the NGINX Ingress Controller in the previous section. We will use it to expose our applications to the outside world and will use the ClusterIssuer we created in the previous section to issue TLS certificates for our applications. There are a lot of annotations that you can add to your Ingress resource to configure how the NGINX Ingress Controller handles individual routes. You can find the full list of annotations here.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress
  annotations:
    kubernetes.io/ingress.class: nginx
    cert-manager.io/cluster-issuer: letsencrypt-prod
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
    nginx.ingress.kubernetes.io/rewrite-target: /$2
spec:
  rules:
    - host: hello-world.info
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: web
                port:
                  number: 8080
  tls:
  - hosts:
      - https-example.foo.com
    secretName: helloworld-tls

In this example, we have configured the NGINX Ingress Controller to use the letsencrypt-prod ClusterIssuer to issue TLS certificates for the hello-world.info domain. The Ingress resource also specifies that all requests to the hello-world.info domain should be redirected to HTTPS. Note that by adding the cert-manager annotation to the Ingress resource, we are telling cert-manager to ask lets-encrypt to issue a TLS certificate for this domain, and it will be stored in the helloworld-tls secret.

Stay tuned for the next part of this series where we will configure Longhorn to provide persistent storage for our applications.

Title: Expose Cilium Metrics to Prometheus
Author: Mitch Murphy
Date: 2023-12-15

• Update Cilium to expose Prometheus metrics
• Introduction
  • Prerequisites
  • Prometheus
    • Prometheus Overview
    • Prometheus Installation
  • Cilium
    • Cilium and Prometheus
    • Cilium Upgrade
    • Cilium ServiceMonitor Verification
    • Cilium Grafana Dashboards

Introduction

This guide serves as a follow up to a previous article covering how to configure RKE2 to use Cilium as the CNI (and fully replace kube-proxy)

Prerequisites

• RKE2 Cluster
• cilium
• kubectl
• helm

Prometheus

Prometheus is an open-source monitoring and alerting toolkit designed for reliability and scalability. It is particularly well-suited for monitoring dynamic, cloud-native environments such as Kubernetes. Here’s an overview of Prometheus and how it facilitates monitoring in a Kubernetes environment:

Prometheus Overview

1. Data Model:
   • Prometheus follows a multi-dimensional data model, where time-series data is identified by metric names and key-value pairs called labels. Metrics represent numerical values that can be observed and monitored over time.
2. Data Retrieval:
   • Prometheus uses a pull-based approach to collect metrics from monitored targets. Each target (e.g., a Kubernetes node or container) exposes an HTTP endpoint where Prometheus can fetch metrics.
3. Scalability:
   • Prometheus is designed to be highly scalable, making it suitable for large-scale distributed systems like Kubernetes. It can handle a high volume of metrics from numerous targets.
4. Query Language:
   • Prometheus Query Language (PromQL) allows users to express complex queries to analyze and aggregate collected metrics. PromQL facilitates the creation of custom dashboards and alerts. Monitoring Kubernetes with Prometheus:
5. Exporter Components:
   • Kubernetes exposes a rich set of metrics through its API server, kubelet, and other components. Prometheus exporters are used to convert these metrics into a format that Prometheus can scrape.
6. Service Discovery:
   • Prometheus uses service discovery mechanisms to automatically discover and monitor new instances of services as they are deployed in Kubernetes. Kubernetes Service Discovery and DNS-based service discovery are commonly used.
7. Instrumentation:
   • Kubernetes components are instrumented to expose metrics in a format that Prometheus understands. Prometheus also supports custom instrumentation, allowing users to expose application-specific metrics.
8. Alerting and Monitoring:
   • Prometheus provides a built-in alerting system that allows users to define alerting rules based on metric conditions. Grafana is often used in conjunction with Prometheus to create dashboards for visualizing metrics.
9. Integration with Kubernetes Ecosystem:
   • Prometheus is a CNCF (Cloud Native Computing Foundation) project and is well-integrated with the Kubernetes ecosystem. It works seamlessly with other tools like Grafana, Alertmanager, and Kubernetes itself.

In summary, Prometheus is a powerful monitoring solution for Kubernetes, offering a flexible and scalable approach to collecting, querying, and alerting based on the metrics generated by the Kubernetes ecosystem. It plays a crucial role in ensuring the reliability and performance of applications deployed in Kubernetes clusters.

Prometheus Installation

The Prometheus stack includes a few components that greatly help SREs/administrators monitor/observe deployed Kubernetes resources. These include:

• The Prometheus Operator
• Highly available Prometheus
• Highly available Alertmanager
• Prometheus node-exporter
• Prometheus Adapter for Kubernetes Metrics APIs
• kube-state-metrics
• Grafana

Installing the Prometheus stack on Kubernetes using Helm is a common and convenient approach. Helm is a package manager for Kubernetes that simplifies the deployment and management of applications. Here’s a step-by-step guide for Kubernetes administrators on installing the Prometheus stack using Helm:

1. Add Prometheus Helm Repository:

   • Add the Prometheus Helm repository to your Helm client.

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

1. Create Namespace (Optional):

   • Create a dedicated namespace for Prometheus if desired.

kubectl create namespace monitoring

1. Install Prometheus Operator:

   • Install the Prometheus Operator using Helm.

helm install prometheus prometheus-community/kube-prometheus-stack -n monitoring --create-namespace

1. Verify Installation:

   • Check the status of the deployed resources.

kubectl get pods -n monitoring

1. Access Prometheus Dashboard:

   • Prometheus comes with a web-based dashboard. Expose it for access.

kubectl port-forward -n monitoring svc/prometheus-stack-kube-prom-prometheus 9090

Access the Prometheus dashboard at http://localhost:9090 in your web browser.

1. Access Grafana Dashboard (Optional):

   • Grafana is included in the Prometheus stack. Expose it for access.

kubectl port-forward -n monitoring svc/prometheus-stack-grafana 3000:80

Access the Grafana dashboard at http://localhost:3000 and log in (default credentials are admin/prom-operator).

1. Configure Data Source in Grafana (Optional):

   • Configure Prometheus as a data source in Grafana.
     • URL: http://prometheus-stack-kube-prom-prometheus:9090
     • Save the configuration.

2. Explore and Customize:

   • Explore the Prometheus dashboard and Grafana to monitor your Kubernetes cluster.
   • Customize Grafana dashboards and Prometheus alerting rules as needed.

By following these steps, Kubernetes administrators can easily set up and start using the Prometheus stack for monitoring their Kubernetes clusters. The Helm charts provided by the Prometheus community simplify the installation process and allow for customization based on specific monitoring requirements. Take note of the namespace where you install the prometheus-stack to as we will need to specify that when we upgrade Cilium. Stay tuned when I cover configuring alerts in Prometheus and integration with Alertmanager!

Cilium

Please refer to my previous post to find more information on what Cilium offers.

Cilium and Prometheus

Integrating Cilium and Prometheus in a Kubernetes environment brings several benefits, particularly in terms of exposing advanced metrics and enhancing the observability of your microservices-based architecture. Here are the key advantages of integrating Cilium and Prometheus:

1. Rich Network Visibility:
   • Cilium Network Security:
     • Cilium provides advanced network security features, including API-aware network security policies, load balancing, and encryption.
     • Prometheus integration with Cilium allows you to capture and visualize metrics related to network security, such as the number of allowed/denied connections and data on network policies.
2. Fine-Grained Service Metrics:
   • Service-Aware Metrics:
     • Cilium can enforce policies at the service or application layer, providing insights into the interactions between microservices.
     • Prometheus can scrape and expose fine-grained metrics related to service communication, allowing you to monitor service-level performance and troubleshoot issues.
3. Distributed Tracing:
   • Tracepoint Metrics:
     • Cilium includes tracepoint-based metrics that can be exposed through Prometheus.
     • By integrating Cilium and Prometheus, you can leverage these metrics for distributed tracing, gaining visibility into the flow of requests across microservices and identifying bottlenecks or latency issues.
4. Security Metrics:
   • Security Incident Monitoring:
     • Cilium enhances security by providing metrics related to security incidents and policy enforcement.
     • Prometheus integration enables monitoring of security-related metrics, helping Kubernetes engineers detect and respond to security events in real-time.
5. Efficient Resource Utilization:
   • Resource Consumption Metrics:
     • Prometheus can collect and expose metrics related to resource consumption at the network level.
     • This integration allows Kubernetes engineers to optimize resource allocation, identify performance bottlenecks, and ensure efficient use of network resources.
6. Customizable Dashboards:
   • Grafana Integration:
     • Prometheus integrates seamlessly with Grafana, a popular visualization tool.
     • Engineers can create customizable dashboards in Grafana to visualize Cilium and Kubernetes metrics, providing a comprehensive view of the entire system’s health and performance.
7. Scalability and Performance Monitoring:
   • Scalability Metrics:
     • Cilium and Prometheus integration enables monitoring of metrics related to the scalability and performance of microservices.
     • Engineers can use this information to optimize configurations, scale resources based on demand, and ensure a responsive and scalable application architecture.
8. Alerting and Automation:
   • Alertmanager Integration:
     • Prometheus integrates with Alertmanager to provide alerting capabilities based on predefined rules.
     • Engineers can set up alerts for key metrics, enabling proactive monitoring and automated responses to potential issues.

In summary, integrating Cilium and Prometheus in a Kubernetes environment enhances observability by providing detailed metrics related to network security, service communication, distributed tracing, and resource utilization. This integration empowers Kubernetes engineers to monitor, analyze, and optimize the performance, security, and scalability of their microservices applications.

Cilium Upgrade

In order to integrate our previously installed Cilium/Hubble we need to update a few of our Helm values, particularly that involving ServiceMonitors. Update the file containing the values (detailed here) to:

cluster:
  name: smig-cluster1
  id: 11
prometheus:
  enabled: true
  serviceMonitor:
    enabled: false
dashboards:
  enabled: true
hubble:
  metrics:
    enabled:
    - dns:query;ignoreAAAA
    - drop
    - tcp
    - flow
    - icmp
    - http
    dashboards:
      enabled: true
    serviceMonitor:
      enabled: true
      namespace: monitoring
      labels:
        release: prometheus-stack
  relay:
    enabled: true
    prometheus:
      enabled: true
      serviceMonitor:
        enabled: true
        namespace: monitoring
        labels:
          release: prometheus-stack
  ui:
    enabled: true
    baseUrl: "/"
version: 1.14.3
operator:
  prometheus:
    enabled: true
    serviceMonitor:
      enabled: true
      namespace: monitoring
      labels:
        release: prometheus-stack
  dashboards:
    enabled: true
envoy:
  prometheus:
    enabled: true
    serviceMonitor:
      enabled: true
      namespace: monitoring
      labels:
        release: prometheus-stack
clustermesh:
  apiserver:
    metrics:
      enabled: false
      serviceMonitor:
        enabled: true
        namespace: monitoring
        labels:
          release: prometheus-stack

Take note of the label that is added to each ServiceMonitor, this must be present in order for Prometheus to detect the “target” and start scraping it. Now let’s upgrade Cilium:

cilium upgrade -f cilium.yaml

Cilium ServiceMonitor Verification

n order to verify that Prometheus is picking up these new targets, you need to port-forward and make sure that they show up under Status -> Targets.

Cilium Grafana Dashboards

Because we specified in our values file that we wish to create Grafana dashboards, these get created as ConfigMaps and automatically picked up by Grafana. Until we expose Grafana as an Ingress, we must port-forward to it:

kubectl port-forward svc/prometheus-stack-grafana 8080:80 &

Now visit our dashboards, and click on the Hubble dashboard. Because we have not deployed anything to our cluster we should not see much here, but as you can see Cilium/Hubble provides quite advanced metrics! We will visit this dashboard again after we deploy a few things and test our system!

Title: RKE2 Install with Cilium, Hubble and Cluster Mesh
Author: Mitch Murphy
Date: 2023-11-25

Table of contents

• How to Guide: RKE2 Install with Cilium, Hubble and Cluster Mesh
  • Table of contents
  • Introduction
    • What is Cilium?
    • What is Hubble?
  • Prerequisites
    • Networking Pre-Requisites
    • Kernel Configuration
    • selinux Configuration
    • Firewall Configuration
    • Storage Configuration
  • RKE2 Installation
    • Control Plane Node
      • Download and Install RKE2
      • Start RKE2
      • Installation Cilium CLI
      • Verify RKE2
  • Install Cilium
    • Verify Cilium
  • Worker Nodes
    • Verify RKE2 Agent
  • MetalLB
    • What is MetalLB?
    • Intall MetalLB
    • Verify MetalLB
  • Cilium Cluster Mesh
  • Create ServiceMonitors
  • Verify Cilium ServiceMonitor

sudo modprobe br_netfilter
sudo modprobe overlay
sudo su -
cat <<EOT | sudo tee /etc/modules-load.d/kubernetes.conf
br_netfilter
overlay
EOT
cat <<EOT | sudo tee /etc/sysctl.d/kubernetes.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
EOT
sysctl --system
exit

sudo dnf -y upgrade --refresh
sudo rpm --import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org
sudo dnf install https://www.elrepo.org/elrepo-release-8.el8.elrepo.noarch.rpm -y
sudo dnf --enablerepo=elrepo-kernel install -y kernel-ml kernel-ml-core kernel-ml-headers kernel-ml-modules kernel-ml-modules-extra

sudo reboot now

uname -r

sudo dnf install -y container-selinux

sudo setenforce 0
sudo sed -i 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config

ALLOWED_PORTS=( 6443 9100 8080 4245 9345 6443 6444 10250 10259 10257 2379 2380 9796 19090 9090 6942 9091 4244 4240 80 443 9963 9964 8081 8082 7000 9001 6379 9121 8084 6060 6061 6062 9879 9890 9891 9892 9893 9962 9966 ) 
for i in "${ALLOWED_PORTS[@]}"
do
  sudo firewall-cmd --add-port=$i/tcp --permanent
done

sudo firewall-cmd --add-port=30000-32767/tcp --permanent
sudo firewall-cmd --remove-icmp-block=echo-request --permanent
sudo firewall-cmd --remove-icmp-block=echo-reply --permanent
# Since we are using Cilium with GENEVE as overlay, we need the following port too:
# UDP
UDP_PORTS=( 8472 4789 6081 51871 53 55355 58467 41637 39291 38519 46190 )
for i in "${UDP_PORTS[@]}"
do
  sudo firewall-cmd --add-port=$i/udp --permanent
done
sudo firewall-cmd --reload
### Ingress Controller specific ports

### To get DNS resolution working, simply enable Masquerading.
sudo firewall-cmd --zone=public  --add-masquerade --permanent

sudo firewall-cmd --zone=trusted --permanent --add-source=192.168.0.0/16

### Finally apply all the firewall changes
sudo firewall-cmd --reload

sudo su -
yum install -y nano curl wget git tmux jq vim-common iscsi-initiator-utils
echo "iscsi_tcp" >/etc/modules-load.d/iscsi-tcp.conf
systemctl enable iscsid --now
systemctl start iscsid

cat <<EOF>> /etc/NetworkManager/conf.d/rke2-canal.conf
[keyfile]
unmanaged-devices=interface-name:cali*;interface-name:flannel*
EOF
systemctl reload NetworkManager
exit

sudo su -
mkdir -p /etc/rancher/rke2/
cat <<EOF > /etc/rancher/rke2/config.yaml
write-kubeconfig-mode: "0644"
# profile: "cis-1.5"
selinux: false
# add ips/hostname of hosts and loadbalancer
tls-sans:
  - "c1cp1.kubula.internal"
  - "192.168.7.11"
# Make a etcd snapshot every day at 4am
etcd-snapshot-schedule-cron: "0 4 * * *"
# Keep 14 etcd snapshots
etcd-snapshot-retention: 14
etcd-expose-metrics: true
disable:
  - rke2-canal
  - rke2-kube-proxy
network:
  plugin: none
disable-kube-proxy: true
disable-cloud-controller: true
cluster-cidr: 10.42.0.0/16
service-cidr: 10.96.0.0/16
EOF

curl -sfL https://get.rke2.io | sudo INSTALL_RKE2_CHANNEL=latest INSTALL_RKE2_TYPE="server" sh -

sudo su -
echo "exclude=rke2-*" >> /etc/dnf/dnf.conf
exit

sudo systemctl enable rke2-server.service --now

CILIUM_CLI_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/cilium-cli/main/stable.txt)
CLI_ARCH=amd64
curl -L --fail --remote-name-all https://github.com/cilium/cilium-cli/releases/download/${CILIUM_CLI_VERSION}/cilium-linux-${CLI_ARCH}.tar.gz{,.sha256sum}
sha256sum --check cilium-linux-${CLI_ARCH}.tar.gz.sha256sum
sudo tar xzvfC cilium-linux-${CLI_ARCH}.tar.gz /usr/local/bin
rm cilium-linux-${CLI_ARCH}.tar.gz{,.sha256sum}

sudo systemctl status rke2-server.service

mkdir ~/.kube
sudo cp /etc/rancher/rke2/rke2.yaml ~/.kube/config
sudo chown $(id -u):$(id -g) ~/.kube/config
chmod 600 ~/.kube/config

sudo cp /var/lib/rancher/rke2/bin/kubectl /usr/local/bin
sudo chown $(id -u):$(id -g) /usr/local/bin/kubectl

kubectl get nodes

cat <<EOF > cilium.yaml
cluster:
  name: smig-cluster1
  id: 11
prometheus:
  enabled: true
  serviceMonitor:
    enabled: false
dashboards:
  enabled: true
hubble:
  metrics:
    enabled:
    - dns:query;ignoreAAAA
    - drop
    - tcp
    - flow
    - icmp
    - http
    dashboards:
      enabled: true
  relay:
    enabled: true
    prometheus:
      enabled: true
  ui:
    enabled: true
    baseUrl: "/"
version: 1.14.3
operator:
  prometheus:
    enabled: true
  dashboards:
    enabled: true
# clustermesh:
#   # -- Deploy clustermesh-apiserver for clustermesh
#   useAPIServer: false
EOF

cilium install -f cilium.yaml

cilium status

kubectl get nodes

NAME     STATUS   ROLES    AGE   VERSION
c1cp1    Ready    master   10m   v1.28.3+rke2r2

cilium hubble port-forward & #3855849
cilium connectivity test --force-deploy

TOKEN=$(ssh -i ~/.ssh/rke2 -o IdentitiesOnly=yes master@192.168.7.11 sudo cat /var/lib/rancher/rke2/server/node-token)

```yaml
sudo su -
mkdir -p /etc/rancher/rke2/
cat <<EOF > /etc/rancher/rke2/config.yaml
server: https://192.168.1.175:9345
token: ${TOKEN}
EOF
exit

curl -sfL https://get.rke2.io | sudo INSTALL_RKE2_CHANNEL=latest INSTALL_RKE2_TYPE="agent" sh -

sudo su -
echo "exclude=rke2-*" >> /etc/dnf/dnf.conf
exit

sudo systemctl enable rke2-agent.service --now

sudo systemctl status rke2-agent.service

kubectl get nodes

helm repo add metallb https://metallb.github.io/metallb
helm install metallb metallb/metallb

helm install --namespace metallb-system metallb metallb/metallb

kubectl edit configmap config -n metallb-system

apiVersion: v1
kind: ConfigMap
metadata:
  namespace: metallb-system
  name: config
data:
  config:
    address-pools:
    - name: default
    protocol: layer2
      addresses:
      - 192.168.8.1-192.168.8.255

kubectl rollout restart deployment controller -n metallb-system

kubectl get pods -n metallb-system

CLUSTER1=smig-cluster1
CLUSTER2=smig-cluster2

# In order for cluster mesh to work properly, the same CA secret must exist in both clusters. Therefore delete the secret in cluster 2 and recreate it from cluster 1.
kubectl --context=$CLUSTER2 delete secret -n kube-system cilium-ca
kubectl --context=$CLUSTER1 get secret -n kube-system cilium-ca -o yaml | \
  kubectl --context $CLUSTER2 create -f -

cilium clustermesh enable --context $CLUSTER1 --service-type ClusterIP
cilium clustermesh enable --context $CLUSTER2 --service-type ClusterIP


cilium clustermesh connect --context $CLUSTER1 --destination-context $CLUSTER2
cilium clustermesh connect --context $CLUSTER2 --destination-context $CLUSTER1

cat <<EOF > cilium-upgrade.yaml
hubble:
  metrics:
    serviceMonitor:
      enabled: true
    dashboards:
      enabled: true
      namespace: monitoring-system
  relay:
    prometheus:
      enabled: true
      serviceMonitor:
        enabled: true
        namespace: monitoring-system
prometheus:
  enabled: true
  serviceMonitor:
    enabled: true
    namespace: monitoring-system
envoy:
  prometheus:
    enabled: true
    serviceMonitor:
      enabled: true
      namespace: monitoring-system
operator:
  prometheus:
    enabled: true
    serviceMonitor:
      enabled: true
      namespace: monitoring-system
  dashboards:
    enabled: true
    namespace: monitoring-system
clustermesh:
  apiserver:
    metrics:
      serviceMonitor:
        enabled: true
        namespace: monitoring-system
dashboards:
  enabled: true
  namespace: monitoring-system
EOF

cilium upgrade -f cilium-upgrade.yaml

kubectl get servicemonitor -n monitoring-system

kubectl port-forward -n monitoring-system svc/prometheus-stack-grafana 3000:80

Title: Setting up K3s on Raspberry Pi's and Jetson Nano's
Author: Mitch Murphy
Date: 2021-10-09

dd if=./speedTestFile of=/dev/zero bs=20M count=5 oflag=dsync # for SD card
dd if=/mnt/storage/speedTestFile of=/dev/zero bs=20M count=5 oflag=dsync # for SSD

dd if=/dev/zero of=./speedTestFile bs=20M count=5 oflag=direct # for SD card
sudo dd if=/dev/zero of=/mnt/storage/speedTestFile bs=20M count=5 oflag=direct # for SSD

network:
    ethernets:
        eth0:
            dhcp4: no
            addresses: [<STATIC_IP>/24]
            gateway4: <GATEWAY_ADDRESS>
            nameservers:
              addresses: [8.8.8.8,8.8.4.4]
    version: 2

sudo apt upgrade
sudo apt update -y
curl -fsSL https://get.docker.com -o get-docker.sh
sudo sh get-docker.sh

From:
#PermitRootLogin prohibit-password
#PasswordAuthentication yes
#PubkeyAuthentication yes
To:
PermitRootLogin no
PasswordAuthentication no
PubkeyAuthentication yes

sudo /usr/sbin/sshd -t
sudo systemctl restart sshd.service

Host k3s-master
Hostname k3s-master
User master
IdentityFile ~/.ssh/id_k3s-master

Host k3s-worker1
Hostname k3s-worker1
User worker
IdentityFile ~/.ssh/id_k3s-worker1

Host k3s-worker2
Hostname k3s-worker2
User worker
IdentityFile ~/.ssh/id_k3s-worker2

Host k3s-worker-gpu
Hostname k3s-worker-gpu
User worker
IdentityFile ~/.ssh/id_k3s-worker-gpu

192.168.0.100   k3s-master
192.168.0.101   k3s-worker1
192.168.0.102   k3s-worker2
192.168.0.104   k3s-worker-gpu

dtoverlay=disable-wifi
dtoverlay=disable-bluetooth

net.ipv6.conf.all.disable_ipv6=1
net.ipv6.conf.default.disable_ipv6=1
net.ipv6.conf.lo.disable_ipv6=1

#!/bin/bash
# /etc/rc.local

/etc/sysctl.d
/etc/init.d/procps restart

exit 0

curl -sfL https://get.k3s.io | INSTALL_KUBE_EXEC="--write-kubeconfig-mode 664 \
--bind-address 192.168.0.100 --advertise-address 192.168.0.100 \
--default-local-storage-path /mnt/storage --cluster-init --node-label memory=high" sh -

# define what Helm version and where to install:
export HELM_VERSION=v3.7.0
export HELM_INSTALL_DIR=/usr/local/bin

# download the binary and get into place:
cd /tmp
wget https://get.helm.sh/helm-$HELM_VERSION-linux-arm64.tar.gz
tar xvzf helm-$HELM_VERSION-linux-arm64.tar.gz
sudo mv linux-arm64/helm $HELM_INSTALL_DIR/helm

# clean up:
rm -rf linux-arm64 && rm helm-$HELM_VERSION-linux-arm64.tar.gz

helm repo add stable https://charts.helm.sh/stable
helm repo add bitnami https://charts.bitnami.com/bitnami
helm repo add rancher-latest https://releases.rancher.com/server-charts/latest
helm repo add jetstack https://charts.jetstack.io

# this is necessary to address https://github.com/rancher/k3s/issues/1126 for now:
export KUBECONFIG=/etc/rancher/k3s/k3s.yaml >> ~/.bashrc
source ~/.bashrc

kubectl -n kubernetes-dashboard get secret $(kubectl -n kubernetes-dashboard get sa/admin-user -o jsonpath="{.secrets[0].name}") -o go-template=""

export TOKEN=K10513ec520ffb7ce3d94da39d5a26be5da9324769f035498595c9941d21bcfeb62::server:ed7aefd846db06468a6c78fb91d461d2
curl -sfL https://get.k3s.io | K3S_URL=https://192.168.0.100:6443 K3S_TOKEN=$TOKEN \
  INSTALL_KUBE_EXEC="--node-label memory=high" sh -

mirrors:
  "docker.io":
    endpoint:
      - "https://docker.io"
configs:
  "docker.io":
    auth:
      username: "smigula"
      password: <TOKEN>
    tls:
      insecure_skip_verify: true

git clone https://github.com/JetsonHacksNano/resizeSwapMemory.git
cd resizeSwapMemory
chmod +x setSwapMemorySize.sh
./setSwapMemorySize.sh -g 8

sudo sysctl -w net.ipv6.conf.all.disable_ipv6=1
sudo sysctl -w net.ipv6.conf.default.disable_ipv6=1
sudo sysctl -w net.ipv6.conf.lo.disable_ipv6=1

auto eth0
iface eth0 inet static
  address 192.168.0.104
  netmask 255.255.255.0
  gateway 192.168.0.1

export TOKEN=<TOKEN>
curl -sfL https://get.k3s.io | K3S_URL=https://192.168.0.100:6443 K3S_TOKEN=$TOKEN \
  INSTALL_KUBE_EXEC="--node-label memory=medium --node-label=gpu=nvidia" sh -

[plugins.opt]
  path = ""

[plugins.cri]
  stream_server_address = "127.0.0.1"
  stream_server_port = "10010"
  disable_cgroup = true
  disable_apparmor = true
  restrict_oom_score_adj = true

  sandbox_image = ""

[plugins.cri.cni]
  bin_dir = ""
  conf_dir = ""


[plugins.cri.containerd.runtimes.runc]
  # ---- changed from 'io.containerd.runc.v2' for GPU support
  runtime_type = "io.containerd.runtime.v1.linux"

# ---- added for GPU support
[plugins.linux]
  runtime = "nvidia-container-runtime"



[plugins.cri.registry.mirrors]

[plugins.cri.registry.mirrors.""]
  endpoint = [, ]




[plugins.cri.registry.configs."".auth]
  username = ""
  password = ""
  auth = ""
  identitytoken = ""


[plugins.cri.registry.configs."".tls]
  ca_file = ""
  cert_file = ""
  key_file = ""

FROM nvcr.io/nvidia/l4t-base:r32.5.0
RUN apt-get update && apt-get install -y --no-install-recommends make g++
COPY ./samples /tmp/samples
WORKDIR /tmp/samples/1_Utilities/deviceQuery
RUN make clean && make
CMD ["./deviceQuery"]

deviceQuery, CUDA Driver = CUDART, CUDA Driver Version = 10.2, CUDA Runtime Version = 10.2, NumDevs = 1
Result = PASS

#!/bin/bash

IMAGE=xift/jetson_devicequery:r32.5.0
export KUBECONFIG=/etc/rancher/k3s/k3s.yaml
ctr i pull docker.io/${IMAGE}
ctr run --rm --gpus 0 --tty docker.io/${IMAGE} deviceQuery

apiVersion: v1
kind: Pod
metadata:
  name: devicequery
spec:
  containers:
  - name: nvidia
    image: xift/jetson_devicequery:r32.5.0
    command: [ "./deviceQuery" ]
  nodeSelector:
    gpu: nvidia

docker pull nvcr.io/nvidia/l4t-pytorch:r32.6.1-pth1.9-py3
docker run -it --rm --runtime nvidia nvcr.io/nvidia/l4t-pytorch:r32.6.1-pth1.9-py3 python3 -c "import torch; print(torch.cuda.is_available());"