Tuning EMQX to Scale to One Million Concurrent Connection on Kubernetes

Sun, 21 Jan 2024 12:34:35 +0800

Tuning EMQX to Scale to One Million Concurrent Connection on Kubernetes

Tuning EMQX to Scale to One Million Concurrent Connection on Kubernetes

Wed, 27 Sep 2023 10:36:00 +0800

Linux Kernel Tuning

node level, basically the non-namespaced sysctls

# Sets the maximum number of file handles allowed by the kernel
sysctl -w fs.file-max=2097152

# Sets the maximum number of open file descriptors that a process can have
sysctl -w fs.nr_open=2097152

namespaced sysctls

# Sets the maximum number of connections that can be queued for acceptance by the kernel.
sysctl -w net.core.somaxconn=32768

# Sets the maximum number of SYN requests that can be queued by the kernel
sysctl -w net.ipv4.tcp_max_syn_backlog=16384

# Setting the minimum, default and maximum size of TCP Buffer
sysctl -w net.ipv4.tcp_rmem='1024 4096 16777216'
sysctl -w net.ipv4.tcp_wmem='1024 4096 16777216'

# Setting Parameters for TCP Connection Tracking
sysctl -w net.netfilter.nf_conntrack_tcp_timeout_time_wait=30

# Controls the maximum number of entries in the TCP time-wait bucket table
sysctl -w net.ipv4.tcp_max_tw_buckets=1048576

# Controls Timeout for FIN-WAIT-2 Sockets:
sysctl -w net.ipv4.tcp_fin_timeout=15

There are some more namespaced sysctls that will improve the performance but because of an active issue we are not able to set them on the container level

# Sets the size of the backlog queue for the network device
sysctl -w net.core.netdev_max_backlog=16384

# Amount of memory that is allocated for storing incoming and outgoing data for a socket
sysctl -w net.core.rmem_default=262144
sysctl -w net.core.wmem_default=262144

# Setting the maximum amount of memory for the socket buffers
sysctl -w net.core.rmem_max=16777216
sysctl -w net.core.wmem_max=16777216
sysctl -w net.core.optmem_max=16777216

Erlang VM Tuning

## Erlang Process Limit
node.process_limit = 2097152

## Sets the maximum number of simultaneously existing ports for this system
node.max_ports = 2097152

EMQX Broker Tuning

# Other configuration…
EMQX_LISTENER__TCP__EXTERNAL: "0.0.0.0:1883"
EMQX_LISTENER__TCP__EXTERNAL__ACCEPTORS: 64
EMQX_LISTENER__TCP__EXTERNAL__MAX_CONNECTIONS: 1024000

MQTT on Ricky

Tuning EMQX to Scale to One Million Concurrent Connection on Kubernetes

Tuning EMQX to Scale to One Million Concurrent Connection on Kubernetes

Linux Kernel Tuning

Erlang VM Tuning

EMQX Broker Tuning