Kubernetes on Ricky

Container security fundamentals

Wed, 04 Oct 2023 09:06:00 +0800

Tuning EMQX to Scale to One Million Concurrent Connection on Kubernetes

Wed, 27 Sep 2023 10:36:00 +0800

Linux Kernel Tuning

node level, basically the non-namespaced sysctls

# Sets the maximum number of file handles allowed by the kernel
sysctl -w fs.file-max=2097152

# Sets the maximum number of open file descriptors that a process can have
sysctl -w fs.nr_open=2097152

namespaced sysctls

# Sets the maximum number of connections that can be queued for acceptance by the kernel.
sysctl -w net.core.somaxconn=32768

# Sets the maximum number of SYN requests that can be queued by the kernel
sysctl -w net.ipv4.tcp_max_syn_backlog=16384

# Setting the minimum, default and maximum size of TCP Buffer
sysctl -w net.ipv4.tcp_rmem='1024 4096 16777216'
sysctl -w net.ipv4.tcp_wmem='1024 4096 16777216'

# Setting Parameters for TCP Connection Tracking
sysctl -w net.netfilter.nf_conntrack_tcp_timeout_time_wait=30

# Controls the maximum number of entries in the TCP time-wait bucket table
sysctl -w net.ipv4.tcp_max_tw_buckets=1048576

# Controls Timeout for FIN-WAIT-2 Sockets:
sysctl -w net.ipv4.tcp_fin_timeout=15

There are some more namespaced sysctls that will improve the performance but because of an active issue we are not able to set them on the container level

# Sets the size of the backlog queue for the network device
sysctl -w net.core.netdev_max_backlog=16384

# Amount of memory that is allocated for storing incoming and outgoing data for a socket
sysctl -w net.core.rmem_default=262144
sysctl -w net.core.wmem_default=262144

# Setting the maximum amount of memory for the socket buffers
sysctl -w net.core.rmem_max=16777216
sysctl -w net.core.wmem_max=16777216
sysctl -w net.core.optmem_max=16777216

Erlang VM Tuning

## Erlang Process Limit
node.process_limit = 2097152

## Sets the maximum number of simultaneously existing ports for this system
node.max_ports = 2097152

EMQX Broker Tuning

# Other configuration…
EMQX_LISTENER__TCP__EXTERNAL: "0.0.0.0:1883"
EMQX_LISTENER__TCP__EXTERNAL__ACCEPTORS: 64
EMQX_LISTENER__TCP__EXTERNAL__MAX_CONNECTIONS: 1024000

Argo CD ApplicationSet Controller: The World Turns for Me!

Mon, 27 Dec 2021 09:41:03 +0800

# Install kind; see the official docs for other platforms
# Run a lightweight k8s cluster locally
~$ brew install kind
# Install kubectx; see the official docs for other platforms
# Use it to switch between k8s contexts
~$ brew install kubectx
# Install helm; see the official docs for other platforms
# K8s package manager
~$ brew install helm
# Install kubectl; see the official docs for other platforms
# Communicate with the k8s cluster API server
~$ brew install kubectl
# Install argocd cli; see the official docs for other platforms
# Communicate with Argo CD
~$ brew install argocd

Understanding Cilium Series (1): Introduction to Cilium

Tue, 21 Dec 2021 13:04:38 +0800

Understanding Cilium Series (1): Introduction to Cilium

Current status of k8s Service load balancing implementations

Before Cilium, Services were implemented by kube-proxy in three modes: userspace, iptables, and ipvs.

Userspace

In this mode, kube-proxy acts as a reverse proxy and listens on random ports. It redirects traffic to the proxy port via iptables rules, and kube-proxy forwards the traffic to backend pods. Service requests go from user space into kernel iptables and then back to user space, which is costly and has poor performance.

Kubernetes Without kube-proxy

Mon, 20 Dec 2021 17:57:13 +0800

Kubernetes Without kube-proxy

Quick-Start

kubeadm init --skip-phases=addon/kube-proxy

# Setup Helm repository
helm repo add cilium https://helm.cilium.io/


helm install cilium cilium/cilium --version 1.9.18 \
 --namespace kube-system \
 --set kubeProxyReplacement=strict \
 --set k8sServiceHost=REPLACE_WITH_API_SERVER_IP \
 --set k8sServicePort=REPLACE_WITH_API_SERVER_PORT

Linkerd 2.8 - Build a Simple and Secure Multi-Cluster Kubernetes Architecture

Thu, 09 Dec 2021 09:25:44 +0800

Linkerd 2.8 - Build a Simple and Secure Multi-Cluster Kubernetes Architecture

Day 28 - Introduction to Useful Third-Party Kubernetes Tools

Thu, 02 Dec 2021 13:28:09 +0800

Day 28 - Introduction to Useful Third-Party Kubernetes Tools

$ kubectl get pods
NAME READY STATUS RESTARTS AGE
ithome-6564f65698-947rv 1/1 Running 0 84s
ithome-6564f65698-fglr9 1/1 Running 0 84s
ithome-6564f65698-k5wtg 1/1 Running 0 84s
ithome-6564f65698-rrvk4 1/1 Running 0 84s
ithome-6564f65698-zhwlj 1/1 Running 0 84s

Stern/Kail

The names of created pods often contain some unreadable random strings.

If you use kubectl to observe logs for individual pods, you need to switch between different pods.

There are many tools for this, such as Stern, Kube-tail, and Kail.

[Kubernetes] Service Overview

Wed, 24 Nov 2021 14:00:45 +0800

[Kubernetes] Service Overview

Define Service

With selector

Since you need Pods before you define a Service, assume there are Pods in the cluster (exposing TCP port 9376) with the label app=MyApp. You can define a Service as an abstraction layer in front of those pods and provide the service through a domain name.

kind: Service
apiVersion: v1
metadata:
 name: my-service
spec:
 # type has four options (ClusterIP, NodePort, LoadBalancer, ExternalName)
 # default is ClusterIP
 type: ClusterIP
 # select pods with "app=MyApp"
 selector:
 app: MyApp
 # Service port configuration
 ports:
 - protocol: TCP
 port: 80
 # This is the port number exposed by the Pod
 targetPort: 9376

Pod <---> Endpoint(tcp:9376) <---> Service(tcp:80, with VIP)

Basic kubeadm usage notes (by request)

Tue, 24 Mar 2020 16:00:34 +0800

Kubernetes Runtime Explained in Plain Language

Thu, 13 Jun 2019 11:28:26 +0800

Kubernetes Runtime Explained in Plain Language