Go on Ricky

Articles

Fri, 13 Mar 2026 18:30:42 +0800

It’s 2026, Just Use Postgres

The “Use the Right Tool” Trap

You’ve heard the advice: “Use the right tool for the right job.”

Sounds wise. So you end up with:

Elasticsearch for search
Pinecone for vectors
Redis for caching
MongoDB for documents
Kafka for queues
InfluxDB for time-series
PostgreSQL for… the stuff that’s left

Here’s what most people don’t realize: Postgres extensions use the same or better algorithms as specialized databases (in many cases).

Go Tool Trace

Tue, 27 Jan 2026 09:54:42 +0800

Trace is an “event timeline”, not “sampling statistics”

go tool trace: used to inspect runtime traces. Its essence is a time-ordered event stream, including:
- goroutine lifecycle/states: create / runnable / running / waiting / syscall
- scheduler (G/M/P) related events, scheduling latency
- blocking time distribution (net / sync / syscall)
- GC events, STW, heap changes (visible in the trace viewer)
Use pprof to find hot spots: who burns CPU, who allocates a lot
Use trace to find latency causes: why it’s stuck (scheduling? locks? network? syscalls? GC?)

Quick intuition:

Why is this running?

Tue, 30 Dec 2025 11:33:23 +0800

Why is this running?

Mosdns-X

Sun, 09 Nov 2025 20:32:00 +0800

install

bash <(curl -sL https://raw.githubusercontent.com/lidebyte/bashshell/refs/heads/main/mosdns-x-manager.sh)

config

sudo tee /etc/mosdns-x/config.yaml > /dev/null <<'EOF'
# mosdns-x concurrent query (no split routing) config

log:
 level: info
 file: /var/log/mosdns-x/mosdns-x.log

plugins:
 # Cache plugin
 - tag: cache
 type: cache
 args:
 size: 1024
 lazy_cache_ttl: 1800

 # Concurrent upstreams: take the first usable answer
 - tag: forward_all
 type: fast_forward
 args:
 upstream:
 # AliDNS
 - addr: "udp://223.5.5.5"
 - addr: "tls://dns.alidns.com"

 # DNSPod / doh.pub
 - addr: "udp://119.29.29.29"
 - addr: "tls://dot.pub"

 # Cloudflare
 - addr: "udp://1.1.1.1"
 - addr: "tls://cloudflare-dns.com"

 # Google
 - addr: "udp://8.8.8.8"
 - addr: "tls://dns.google"

 # Main pipeline: small cache -> concurrent selection
 - tag: main
 type: sequence
 args:
 exec:
 - cache
 - forward_all

# Listen on dual-stack UDP/TCP 53
servers:
 - exec: main
 listeners:
 - addr: :53
 protocol: udp
 - addr: :53
 protocol: tcp
EOF

systemd

sudo tee /etc/systemd/system/mosdns.service > /dev/null <<'EOF'
[Unit]
Description=Mosdns-X DNS Accelerator
After=network.target

[Service]
Type=simple
User=root
Group=root
ExecStart=/usr/local/bin/mosdns-x start --as-service -d /usr/local/bin -c /etc/mosdns-x/config.yaml
Restart=always
RestartSec=5
StandardOutput=journal
StandardError=journal
SyslogIdentifier=mosdns

[Install]
WantedBy=multi-user.target
EOF

sudo systemctl daemon-reload
sudo systemctl enable --now mosdns

# Backup system DNS
sudo cp -n /etc/resolv.conf /etc/resolv.conf.mosdns-backup

# Switch to local Mosdns-X
echo -e "nameserver 127.0.0.1\noptions edns0" | sudo tee /etc/resolv.conf

# If port 53 is occupied by systemd-resolved, disable it
sudo systemctl disable --now systemd-resolved 2>/dev/null || true

# If you also want to lock it (prevent DHCP changes), run chattr too:
echo -e "nameserver 127.0.0.1\n" > /etc/resolv.conf && chattr +i /etc/resolv.conf

# Check process status
sudo systemctl status mosdns --no-pager

# Test resolution speed (second run should hit cache)
dig +stats www.google.com
dig +stats www.baidu.com

# View logs in real time
tail -f /var/log/mosdns-x/mosdns-x.log

go-synctest

Fri, 07 Nov 2025 14:06:00 +0800

go-synctest

func TestAfterFunc(t *testing.T) {
 synctest.Test(t, func(*testing.T) {
 ctx, cancel := context.WithCancel(context.Background())

 called := false
 context.AfterFunc(ctx, func() { called = true })

 synctest.Wait() // Wait until all goroutines are blocked
 if called {
 t.Fatal("AfterFunc was called before cancel")
 }

 cancel()

 synctest.Wait() // Wait again
 if !called {
 t.Fatal("AfterFunc was not called after cancel")
 }
 })
}

Go Practical Guide: Execute Lua Scripts with go-redis

Tue, 22 Jul 2025 16:56:00 +0800

Go Practical Guide: Execute Lua Scripts with go-redis

Redis command Eval
Redis package redis.NewScript, script.Run

Go Protobuf: The New Opaque API

Mon, 07 Apr 2025 13:53:00 +0800

[Translated] Go Protobuf: The New Opaque API

protoc --proto_path=. \
 --go_out=./ \
+ --go_opt=default_api_level=API_OPAQUE \

GitLab CI Can Automatically Check Code Quality! How to Use SonarQube for Code Quality Checks?

Thu, 06 Feb 2025 12:01:22 +0800

Supply chain attack

Fri, 13 Sep 2024 09:41:00 +0800

Be Alert: Supply Chain Attacks Are Closer Than You Think

github.com/siruspen/logrus false
github.com/sirupsen/logrus true

Go Articles

Tue, 20 Aug 2024 08:55:00 +0800

Learn gin Parameter Validation with the validator Library - This One Article Is Enough

String Constraints

excludesall: does not contain any of the UNICODE characters in the parameter, e.g. excludesall=ab

excludesrune: does not contain the rune specified by the parameter, e.g. excludesrune=asong

startswith: starts with the parameter substring, e.g. startswith=hi

endswith: ends with the parameter substring, e.g. endswith=bye

contains: contains the parameter substring, e.g. contains=email

containsany: contains any of the UNICODE characters in the parameter, e.g. containsany=ab

Black Hat Go

Thu, 01 Aug 2024 15:10:22 +0800

Illustrating How Data Alignment Affects Memory Usage in Go

Mon, 08 Jul 2024 17:52:37 +0800

Illustrating How Data Alignment Affects Memory Usage in Go

The One Billion Row Challenge in Go: from 1m45s to 3.4s in nine solutions

Mon, 08 Jul 2024 09:50:10 +0800

Baseline

$ time cat measurements.txt >/dev/null
0m1.052s

$ time wc measurements.txt
 1000000000 1179173106 13795293380 measurements.txt
0m55.710s

$ time gawk -b -f 1brc.awk measurements.txt >measurements.out
7m35.567s

BEGIN {
 FS = ";"
}

{
 if (counts[$1]++) {
 mins[$1] = $2 < mins[$1] ? $2 : mins[$1]
 maxs[$1] = $2 > maxs[$1] ? $2 : maxs[$1]
 } else {
 mins[$1] = maxs[$1] = $2 # new entry
 }
 sums[$1] += $2
}

END {
 printf "{"
 n = asorti(mins, sorted)
 for (i = 1; i <= n; i++) {
 station = sorted[i]
 min = mins[station]
 max = maxs[station]
 mean = sums[station] / counts[station]
 printf "%s=%.1f/%.1f/%.1f", station, min, mean, max
 if (i < n) {
 printf ", "
 }
 }
 printf "}\n"
}

Solution 1: simple and idiomatic Go

// r1: simple, idiomatic Go using bufio.Scanner and strconv.ParseFloat
//
// ~1.004s for 10M rows

package main

import (
 "bufio"
 "fmt"
 "io"
 "os"
 "sort"
 "strconv"
 "strings"
)

func r1(inputPath string, output io.Writer) error {
 type stats struct {
 min, max, sum float64
 count int64
 }

 f, err := os.Open(inputPath)
 if err != nil {
 return err
 }
 defer f.Close()

 stationStats := make(map[string]stats)

 scanner := bufio.NewScanner(f)
 for scanner.Scan() {
 line := scanner.Text()
 station, tempStr, hasSemi := strings.Cut(line, ";")
 if !hasSemi {
 continue
 }

 temp, err := strconv.ParseFloat(tempStr, 64)
 if err != nil {
 return err
 }

 s, ok := stationStats[station]
 if !ok {
 s.min = temp
 s.max = temp
 s.sum = temp
 s.count = 1
 } else {
 s.min = min(s.min, temp)
 s.max = max(s.max, temp)
 s.sum += temp
 s.count++
 }
 stationStats[station] = s
 }

 stations := make([]string, 0, len(stationStats))
 for station := range stationStats {
 stations = append(stations, station)
 }
 sort.Strings(stations)

 fmt.Fprint(output, "{")
 for i, station := range stations {
 if i > 0 {
 fmt.Fprint(output, ", ")
 }
 s := stationStats[station]
 mean := s.sum / float64(s.count)
 fmt.Fprintf(output, "%s=%.1f/%.1f/%.1f", station, s.min, mean, s.max)
 }
 fmt.Fprint(output, "}\n")
 return nil
}

Solution 2: map with pointer values

// r2: use stats pointer as map value to avoid double hashing
//
// ~921ms for 10M rows (1.09x as fast as r1)

package main

import (
 "bufio"
 "fmt"
 "io"
 "os"
 "sort"
 "strconv"
 "strings"
)

func r2(inputPath string, output io.Writer) error {
 type stats struct {
 min, max, sum float64
 count int64
 }

 f, err := os.Open(inputPath)
 if err != nil {
 return err
 }
 defer f.Close()

 stationStats := make(map[string]*stats)

 scanner := bufio.NewScanner(f)
 for scanner.Scan() {
 line := scanner.Text()
 station, tempStr, hasSemi := strings.Cut(line, ";")
 if !hasSemi {
 continue
 }

 temp, err := strconv.ParseFloat(tempStr, 64)
 if err != nil {
 return err
 }

 s := stationStats[station]
 if s == nil {
 stationStats[station] = &stats{
 min: temp,
 max: temp,
 sum: temp,
 count: 1,
 }
 } else {
 s.min = min(s.min, temp)
 s.max = max(s.max, temp)
 s.sum += temp
 s.count++
 }
 }

 stations := make([]string, 0, len(stationStats))
 for station := range stationStats {
 stations = append(stations, station)
 }
 sort.Strings(stations)

 fmt.Fprint(output, "{")
 for i, station := range stations {
 if i > 0 {
 fmt.Fprint(output, ", ")
 }
 s := stationStats[station]
 mean := s.sum / float64(s.count)
 fmt.Fprintf(output, "%s=%.1f/%.1f/%.1f", station, s.min, mean, s.max)
 }
 fmt.Fprint(output, "}\n")
 return nil
}

Solution 3: avoid strconv.ParseFloat

// r3: parse temperatures manually instead of using strconv.ParseFloat
//
// ~517ms for 10M rows (1.94x as fast as r1)

package main

import (
 "bufio"
 "bytes"
 "fmt"
 "io"
 "os"
 "sort"
)

func r3(inputPath string, output io.Writer) error {
 type stats struct {
 min, max, sum float64
 count int64
 }

 f, err := os.Open(inputPath)
 if err != nil {
 return err
 }
 defer f.Close()

 stationStats := make(map[string]*stats)

 scanner := bufio.NewScanner(f)
 for scanner.Scan() {
 line := scanner.Bytes()
 station, tempBytes, hasSemi := bytes.Cut(line, []byte(";"))
 if !hasSemi {
 continue
 }

 negative := false
 index := 0
 if tempBytes[index] == '-' {
 index++
 negative = true
 }
 temp := float64(tempBytes[index] - '0')
 index++
 if tempBytes[index] != '.' {
 temp = temp*10 + float64(tempBytes[index]-'0')
 index++
 }
 index++ // skip '.'
 temp += float64(tempBytes[index]-'0') / 10
 if negative {
 temp = -temp
 }

 s := stationStats[string(station)]
 if s == nil {
 stationStats[string(station)] = &stats{
 min: temp,
 max: temp,
 sum: temp,
 count: 1,
 }
 } else {
 s.min = min(s.min, temp)
 s.max = max(s.max, temp)
 s.sum += temp
 s.count++
 }
 }

 stations := make([]string, 0, len(stationStats))
 for station := range stationStats {
 stations = append(stations, station)
 }
 sort.Strings(stations)

 fmt.Fprint(output, "{")
 for i, station := range stations {
 if i > 0 {
 fmt.Fprint(output, ", ")
 }
 s := stationStats[station]
 mean := s.sum / float64(s.count)
 fmt.Fprintf(output, "%s=%.1f/%.1f/%.1f", station, s.min, mean, s.max)
 }
 fmt.Fprint(output, "}\n")
 return nil
}

Solution 4: fixed point integers

// r4: use fixed point int32s (*10) instead of float64s
//
// ~491ms for 10M rows (2.04x as fast as r1)

package main

import (
 "bufio"
 "bytes"
 "fmt"
 "io"
 "os"
 "sort"
)

func r4(inputPath string, output io.Writer) error {
 type stats struct {
 min, max, count int32
 sum int64
 }

 f, err := os.Open(inputPath)
 if err != nil {
 return err
 }
 defer f.Close()

 stationStats := make(map[string]*stats)

 scanner := bufio.NewScanner(f)
 for scanner.Scan() {
 line := scanner.Bytes()
 station, tempBytes, hasSemi := bytes.Cut(line, []byte(";"))
 if !hasSemi {
 continue
 }

 negative := false
 index := 0
 if tempBytes[index] == '-' {
 index++
 negative = true
 }
 temp := int32(tempBytes[index] - '0')
 index++
 if tempBytes[index] != '.' {
 temp = temp*10 + int32(tempBytes[index]-'0')
 index++
 }
 index++ // skip '.'
 temp = temp*10 + int32(tempBytes[index]-'0')
 if negative {
 temp = -temp
 }

 s := stationStats[string(station)]
 if s == nil {
 stationStats[string(station)] = &stats{
 min: temp,
 max: temp,
 sum: int64(temp),
 count: 1,
 }
 } else {
 s.min = min(s.min, temp)
 s.max = max(s.max, temp)
 s.sum += int64(temp)
 s.count++
 }
 }

 stations := make([]string, 0, len(stationStats))
 for station := range stationStats {
 stations = append(stations, station)
 }
 sort.Strings(stations)

 fmt.Fprint(output, "{")
 for i, station := range stations {
 if i > 0 {
 fmt.Fprint(output, ", ")
 }
 s := stationStats[station]
 mean := float64(s.sum) / float64(s.count) / 10
 fmt.Fprintf(output, "%s=%.1f/%.1f/%.1f", station, float64(s.min)/10, mean, float64(s.max)/10)
 }
 fmt.Fprint(output, "}\n")
 return nil
}

Solution 5: avoid bytes.Cut

// r5: avoid bytes.Cut
//
// ~442ms for 10M rows (2.27x as fast as r1)

package main

import (
 "bufio"
 "fmt"
 "io"
 "os"
 "sort"
)

func r5(inputPath string, output io.Writer) error {
 type stats struct {
 min, max, count int32
 sum int64
 }

 f, err := os.Open(inputPath)
 if err != nil {
 return err
 }
 defer f.Close()

 stationStats := make(map[string]*stats)

 scanner := bufio.NewScanner(f)
 for scanner.Scan() {
 line := scanner.Bytes()

 end := len(line)
 tenths := int32(line[end-1] - '0')
 ones := int32(line[end-3] - '0') // line[end-2] is '.'
 var temp int32
 var semicolon int
 if line[end-4] == ';' {
 temp = ones*10 + tenths
 semicolon = end - 4
 } else if line[end-4] == '-' {
 temp = -(ones*10 + tenths)
 semicolon = end - 5
 } else {
 tens := int32(line[end-4] - '0')
 if line[end-5] == ';' {
 temp = tens*100 + ones*10 + tenths
 semicolon = end - 5
 } else { // '-'
 temp = -(tens*100 + ones*10 + tenths)
 semicolon = end - 6
 }
 }

 station := line[:semicolon]
 s := stationStats[string(station)]
 if s == nil {
 stationStats[string(station)] = &stats{
 min: temp,
 max: temp,
 sum: int64(temp),
 count: 1,
 }
 } else {
 s.min = min(s.min, temp)
 s.max = max(s.max, temp)
 s.sum += int64(temp)
 s.count++
 }
 }

 stations := make([]string, 0, len(stationStats))
 for station := range stationStats {
 stations = append(stations, station)
 }
 sort.Strings(stations)

 fmt.Fprint(output, "{")
 for i, station := range stations {
 if i > 0 {
 fmt.Fprint(output, ", ")
 }
 s := stationStats[station]
 mean := float64(s.sum) / float64(s.count) / 10
 fmt.Fprintf(output, "%s=%.1f/%.1f/%.1f", station, float64(s.min)/10, mean, float64(s.max)/10)
 }
 fmt.Fprint(output, "}\n")
 return nil
}

Solution 6: avoid bufio.Scanner

// r6: don't use bufio.Scanner to avoid scanning some bytes twice
//
// ~399ms for 10M rows (2.52x as fast as r1)

package main

import (
 "bytes"
 "fmt"
 "io"
 "os"
 "sort"
)

func r6(inputPath string, output io.Writer) error {
 type stats struct {
 min, max, count int32
 sum int64
 }

 f, err := os.Open(inputPath)
 if err != nil {
 return err
 }
 defer f.Close()

 stationStats := make(map[string]*stats)

 buf := make([]byte, 1024*1024)
 readStart := 0
 for {
 n, err := f.Read(buf[readStart:])
 if err != nil && err != io.EOF {
 return err
 }
 if readStart+n == 0 {
 break
 }
 chunk := buf[:readStart+n]

 newline := bytes.LastIndexByte(chunk, '\n')
 if newline < 0 {
 break
 }
 remaining := chunk[newline+1:]
 chunk = chunk[:newline+1]

 for {
 station, after, hasSemi := bytes.Cut(chunk, []byte(";"))
 if !hasSemi {
 break
 }

 index := 0
 negative := false
 if after[index] == '-' {
 negative = true
 index++
 }
 temp := int32(after[index] - '0')
 index++
 if after[index] != '.' {
 temp = temp*10 + int32(after[index]-'0')
 index++
 }
 index++ // skip '.'
 temp = temp*10 + int32(after[index]-'0')
 index += 2 // skip last digit and '\n'
 if negative {
 temp = -temp
 }
 chunk = after[index:]

 s := stationStats[string(station)]
 if s == nil {
 stationStats[string(station)] = &stats{
 min: temp,
 max: temp,
 sum: int64(temp),
 count: 1,
 }
 } else {
 s.min = min(s.min, temp)
 s.max = max(s.max, temp)
 s.sum += int64(temp)
 s.count++
 }
 }

 readStart = copy(buf, remaining)
 }

 stations := make([]string, 0, len(stationStats))
 for station := range stationStats {
 stations = append(stations, station)
 }
 sort.Strings(stations)

 fmt.Fprint(output, "{")
 for i, station := range stations {
 if i > 0 {
 fmt.Fprint(output, ", ")
 }
 s := stationStats[station]
 mean := float64(s.sum) / float64(s.count) / 10
 fmt.Fprintf(output, "%s=%.1f/%.1f/%.1f", station, float64(s.min)/10, mean, float64(s.max)/10)
 }
 fmt.Fprint(output, "}\n")
 return nil
}

Solution 7: custom hash table

// r7: use custom hash table and hash station name as we look for ';'
//
// ~234ms for 10M rows (4.29x as fast as r1)

package main

import (
 "bytes"
 "fmt"
 "io"
 "os"
 "sort"
)

func r7(inputPath string, output io.Writer) error {
 type stats struct {
 min, max, count int32
 sum int64
 }

 f, err := os.Open(inputPath)
 if err != nil {
 return err
 }
 defer f.Close()

 type item struct {
 key []byte
 stat *stats
 }
 const numBuckets = 1 << 17 // number of hash buckets (power of 2)
 items := make([]item, numBuckets) // hash buckets, linearly probed
 size := 0 // number of active items in items slice

 buf := make([]byte, 1024*1024)
 readStart := 0
 for {
 n, err := f.Read(buf[readStart:])
 if err != nil && err != io.EOF {
 return err
 }
 if readStart+n == 0 {
 break
 }
 chunk := buf[:readStart+n]

 newline := bytes.LastIndexByte(chunk, '\n')
 if newline < 0 {
 break
 }
 remaining := chunk[newline+1:]
 chunk = chunk[:newline+1]

 for {
 const (
 // FNV-1 64-bit constants from hash/fnv.
 offset64 = 14695981039346656037
 prime64 = 1099511628211
 )

 var station, after []byte
 hash := uint64(offset64)
 i := 0
 for ; i < len(chunk); i++ {
 c := chunk[i]
 if c == ';' {
 station = chunk[:i]
 after = chunk[i+1:]
 break
 }
 hash ^= uint64(c) // FNV-1a is XOR then *
 hash *= prime64
 }
 if i == len(chunk) {
 break
 }

 index := 0
 negative := false
 if after[index] == '-' {
 negative = true
 index++
 }
 temp := int32(after[index] - '0')
 index++
 if after[index] != '.' {
 temp = temp*10 + int32(after[index]-'0')
 index++
 }
 index++ // skip '.'
 temp = temp*10 + int32(after[index]-'0')
 index += 2 // skip last digit and '\n'
 if negative {
 temp = -temp
 }
 chunk = after[index:]

 hashIndex := int(hash & uint64(numBuckets-1))
 for {
 if items[hashIndex].key == nil {
 // Found empty slot, add new item (copying key).
 key := make([]byte, len(station))
 copy(key, station)
 items[hashIndex] = item{
 key: key,
 stat: &stats{
 min: temp,
 max: temp,
 sum: int64(temp),
 count: 1,
 },
 }
 size++
 if size > numBuckets/2 {
 panic("too many items in hash table")
 }
 break
 }
 if bytes.Equal(items[hashIndex].key, station) {
 // Found matching slot, add to existing stats.
 s := items[hashIndex].stat
 s.min = min(s.min, temp)
 s.max = max(s.max, temp)
 s.sum += int64(temp)
 s.count++
 break
 }
 // Slot already holds another key, try next slot (linear probe).
 hashIndex++
 if hashIndex >= numBuckets {
 hashIndex = 0
 }
 }
 }

 readStart = copy(buf, remaining)
 }

 stationItems := make([]item, 0, size)
 for _, item := range items {
 if item.key == nil {
 continue
 }
 stationItems = append(stationItems, item)
 }
 sort.Slice(stationItems, func(i, j int) bool {
 return string(stationItems[i].key) < string(stationItems[j].key)
 })

 fmt.Fprint(output, "{")
 for i, item := range stationItems {
 if i > 0 {
 fmt.Fprint(output, ", ")
 }
 s := item.stat
 mean := float64(s.sum) / float64(s.count) / 10
 fmt.Fprintf(output, "%s=%.1f/%.1f/%.1f", item.key, float64(s.min)/10, mean, float64(s.max)/10)
 }
 fmt.Fprint(output, "}\n")
 return nil
}

Solution 8: process chunks in parallel

// r8: add some parallelism (but back to non-optimized r1 version)
//
// ~213ms for 10M rows (4.71x as fast as r1)

package main

import (
 "bufio"
 "bytes"
 "fmt"
 "io"
 "os"
 "sort"
 "strconv"
 "strings"
)

type r8Stats struct {
 min, max, sum float64
 count int64
}

func r8(inputPath string, output io.Writer) error {
 parts, err := splitFile(inputPath, maxGoroutines)
 if err != nil {
 return err
 }

 resultsCh := make(chan map[string]r8Stats)
 for _, part := range parts {
 go r8ProcessPart(inputPath, part.offset, part.size, resultsCh)
 }

 totals := make(map[string]r8Stats)
 for i := 0; i < len(parts); i++ {
 result := <-resultsCh
 for station, s := range result {
 ts, ok := totals[station]
 if !ok {
 totals[station] = r8Stats{
 min: s.min,
 max: s.max,
 sum: s.sum,
 count: s.count,
 }
 continue
 }
 ts.min = min(ts.min, s.min)
 ts.max = max(ts.max, s.max)
 ts.sum += s.sum
 ts.count += s.count
 totals[station] = ts
 }
 }

 stations := make([]string, 0, len(totals))
 for station := range totals {
 stations = append(stations, station)
 }
 sort.Strings(stations)

 fmt.Fprint(output, "{")
 for i, station := range stations {
 if i > 0 {
 fmt.Fprint(output, ", ")
 }
 s := totals[station]
 mean := s.sum / float64(s.count)
 fmt.Fprintf(output, "%s=%.1f/%.1f/%.1f", station, s.min, mean, s.max)
 }
 fmt.Fprint(output, "}\n")
 return nil
}

func r8ProcessPart(inputPath string, fileOffset, fileSize int64, resultsCh chan map[string]r8Stats) {
 file, err := os.Open(inputPath)
 if err != nil {
 panic(err)
 }
 defer file.Close()
 _, err = file.Seek(fileOffset, io.SeekStart)
 if err != nil {
 panic(err)
 }
 f := io.LimitedReader{R: file, N: fileSize}

 stationStats := make(map[string]r8Stats)

 scanner := bufio.NewScanner(&f)
 for scanner.Scan() {
 line := scanner.Text()
 station, tempStr, hasSemi := strings.Cut(line, ";")
 if !hasSemi {
 continue
 }

 temp, err := strconv.ParseFloat(tempStr, 64)
 if err != nil {
 panic(err)
 }

 s, ok := stationStats[station]
 if !ok {
 s.min = temp
 s.max = temp
 s.sum = temp
 s.count = 1
 } else {
 s.min = min(s.min, temp)
 s.max = max(s.max, temp)
 s.sum += temp
 s.count++
 }
 stationStats[station] = s
 }

 resultsCh <- stationStats
}

type part struct {
 offset, size int64
}

func splitFile(inputPath string, numParts int) ([]part, error) {
 const maxLineLength = 100

 f, err := os.Open(inputPath)
 if err != nil {
 return nil, err
 }
 st, err := f.Stat()
 if err != nil {
 return nil, err
 }
 size := st.Size()
 splitSize := size / int64(numParts)

 buf := make([]byte, maxLineLength)

 parts := make([]part, 0, numParts)
 offset := int64(0)
 for i := 0; i < numParts; i++ {
 if i == numParts-1 {
 if offset < size {
 parts = append(parts, part{offset, size - offset})
 }
 break
 }

 seekOffset := max(offset+splitSize-maxLineLength, 0)
 _, err := f.Seek(seekOffset, io.SeekStart)
 if err != nil {
 return nil, err
 }
 n, _ := io.ReadFull(f, buf)
 chunk := buf[:n]
 newline := bytes.LastIndexByte(chunk, '\n')
 if newline < 0 {
 return nil, fmt.Errorf("newline not found at offset %d", offset+splitSize-maxLineLength)
 }
 remaining := len(chunk) - newline - 1
 nextOffset := seekOffset + int64(len(chunk)) - int64(remaining)
 parts = append(parts, part{offset, nextOffset - offset})
 offset = nextOffset
 }
 return parts, nil
}

Solution 9: all optimisations plus parallelisation

// r9: all the previous optimizations plus parallel execution
//
// 43ms for 10M rows (23.3x as fast as r1)

package main

import (
 "bytes"
 "fmt"
 "io"
 "os"
 "sort"
)

type r9Stats struct {
 min, max, count int32
 sum int64
}

func r9(inputPath string, output io.Writer) error {
 parts, err := splitFile(inputPath, maxGoroutines)
 if err != nil {
 return err
 }

 resultsCh := make(chan map[string]*r9Stats)
 for _, part := range parts {
 go r9ProcessPart(inputPath, part.offset, part.size, resultsCh)
 }

 totals := make(map[string]*r9Stats)
 for i := 0; i < len(parts); i++ {
 result := <-resultsCh
 for station, s := range result {
 ts := totals[station]
 if ts == nil {
 totals[station] = s
 continue
 }
 ts.min = min(ts.min, s.min)
 ts.max = max(ts.max, s.max)
 ts.sum += s.sum
 ts.count += s.count
 }
 }

 stations := make([]string, 0, len(totals))
 for station := range totals {
 stations = append(stations, station)
 }
 sort.Strings(stations)

 fmt.Fprint(output, "{")
 for i, station := range stations {
 if i > 0 {
 fmt.Fprint(output, ", ")
 }
 s := totals[station]
 mean := float64(s.sum) / float64(s.count) / 10
 fmt.Fprintf(output, "%s=%.1f/%.1f/%.1f", station, float64(s.min)/10, mean, float64(s.max)/10)
 }
 fmt.Fprint(output, "}\n")
 return nil
}

func r9ProcessPart(inputPath string, fileOffset, fileSize int64, resultsCh chan map[string]*r9Stats) {
 file, err := os.Open(inputPath)
 if err != nil {
 panic(err)
 }
 defer file.Close()
 _, err = file.Seek(fileOffset, io.SeekStart)
 if err != nil {
 panic(err)
 }
 f := io.LimitedReader{R: file, N: fileSize}

 type item struct {
 key []byte
 stat *r9Stats
 }
 const numBuckets = 1 << 17 // number of hash buckets (power of 2)
 items := make([]item, numBuckets) // hash buckets, linearly probed
 size := 0 // number of active items in items slice

 buf := make([]byte, 1024*1024)
 readStart := 0
 for {
 n, err := f.Read(buf[readStart:])
 if err != nil && err != io.EOF {
 panic(err)
 }
 if readStart+n == 0 {
 break
 }
 chunk := buf[:readStart+n]

 newline := bytes.LastIndexByte(chunk, '\n')
 if newline < 0 {
 break
 }
 remaining := chunk[newline+1:]
 chunk = chunk[:newline+1]

 for {
 const (
 // FNV-1 64-bit constants from hash/fnv.
 offset64 = 14695981039346656037
 prime64 = 1099511628211
 )

 var station, after []byte
 hash := uint64(offset64)
 i := 0
 for ; i < len(chunk); i++ {
 c := chunk[i]
 if c == ';' {
 station = chunk[:i]
 after = chunk[i+1:]
 break
 }
 hash ^= uint64(c) // FNV-1a is XOR then *
 hash *= prime64
 }
 if i == len(chunk) {
 break
 }

 index := 0
 negative := false
 if after[index] == '-' {
 negative = true
 index++
 }
 temp := int32(after[index] - '0')
 index++
 if after[index] != '.' {
 temp = temp*10 + int32(after[index]-'0')
 index++
 }
 index++ // skip '.'
 temp = temp*10 + int32(after[index]-'0')
 index += 2 // skip last digit and '\n'
 if negative {
 temp = -temp
 }
 chunk = after[index:]

 hashIndex := int(hash & (numBuckets - 1))
 for {
 if items[hashIndex].key == nil {
 // Found empty slot, add new item (copying key).
 key := make([]byte, len(station))
 copy(key, station)
 items[hashIndex] = item{
 key: key,
 stat: &r9Stats{
 min: temp,
 max: temp,
 sum: int64(temp),
 count: 1,
 },
 }
 size++
 if size > numBuckets/2 {
 panic("too many items in hash table")
 }
 break
 }
 if bytes.Equal(items[hashIndex].key, station) {
 // Found matching slot, add to existing stats.
 s := items[hashIndex].stat
 s.min = min(s.min, temp)
 s.max = max(s.max, temp)
 s.sum += int64(temp)
 s.count++
 break
 }
 // Slot already holds another key, try next slot (linear probe).
 hashIndex++
 if hashIndex >= numBuckets {
 hashIndex = 0
 }
 }
 }

 readStart = copy(buf, remaining)
 }

 result := make(map[string]*r9Stats, size)
 for _, item := range items {
 if item.key == nil {
 continue
 }
 result[string(item.key)] = item.stat
 }
 resultsCh <- result
}

Solution 10: all the previous optimizations plus faster semicolon finding and hashing

// r10: all the previous optimizations plus faster semicolon finding and
// hashing
//
// Translated from Java by Menno Finlay-Smits Ideas with ideas taken from
// this fast Java solution:
//
// https://github.com/gunnarmorling/1brc/blob/main/src/main/java/dev/morling/onebrc/CalculateAverage_mtopolnik.java
//
// On my (Ben's) laptop I get these initial results:
//
// $ ./go-1brc -revision=1 ../1brc/data/measurements.txt >out-r1
// Processed 13156.2MB in 1m39.507011009s
// $ ./go-1brc -revision=9 ../1brc/data/measurements.txt >out-r9
// Processed 13156.2MB in 2.893693843s # 34.4x as fast as the r1 above
// $ ./go-1brc -revision=10 ../1brc/data/measurements.txt >out-r10
// Processed 13156.2MB in 2.497241029s # 39.8x as fast as the r1 above

package main

import (
 "bytes"
 "encoding/binary"
 "fmt"
 "io"
 "math/bits"
 "os"
 "sort"
)

const BroadcastSemicolon = 0x3B3B3B3B3B3B3B3B
const Broadcast0x01 = 0x0101010101010101
const Broadcast0x80 = 0x8080808080808080

type r10Stats struct {
 min, max, count int32
 sum int64
}

func r10(inputPath string, output io.Writer) error {
 parts, err := splitFile(inputPath, maxGoroutines)
 if err != nil {
 return err
 }

 resultsCh := make(chan map[string]*r10Stats)
 for _, part := range parts {
 go r10ProcessPart(inputPath, part.offset, part.size, resultsCh)
 }

 totals := make(map[string]*r10Stats)
 for i := 0; i < len(parts); i++ {
 result := <-resultsCh
 for station, s := range result {
 ts := totals[station]
 if ts == nil {
 totals[station] = s
 continue
 }
 ts.min = min(ts.min, s.min)
 ts.max = max(ts.max, s.max)
 ts.sum += s.sum
 ts.count += s.count
 }
 }

 stations := make([]string, 0, len(totals))
 for station := range totals {
 stations = append(stations, station)
 }
 sort.Strings(stations)

 fmt.Fprint(output, "{")
 for i, station := range stations {
 if i > 0 {
 fmt.Fprint(output, ", ")
 }
 s := totals[station]
 mean := float64(s.sum) / float64(s.count) / 10
 fmt.Fprintf(output, "%s=%.1f/%.1f/%.1f", station, float64(s.min)/10, mean, float64(s.max)/10)
 }
 fmt.Fprint(output, "}\n")

 return nil
}

func r10ProcessPart(inputPath string, fileOffset, fileSize int64, resultsCh chan map[string]*r10Stats) {
 file, err := os.Open(inputPath)
 if err != nil {
 panic(err)
 }
 defer file.Close()
 _, err = file.Seek(fileOffset, io.SeekStart)
 if err != nil {
 panic(err)
 }
 f := io.LimitedReader{R: file, N: fileSize}

 type item struct {
 key []byte
 stat *r10Stats
 }
 const numBuckets = 1 << 17 // number of hash buckets (power of 2)
 items := make([]item, numBuckets) // hash buckets, linearly probed
 size := 0 // number of active items in items slice

 buf := make([]byte, 1024*1024)
 readStart := 0
 for {
 n, err := f.Read(buf[readStart:])
 if err != nil && err != io.EOF {
 panic(err)
 }
 if readStart+n == 0 {
 break
 }
 chunk := buf[:readStart+n]

 newline := bytes.LastIndexByte(chunk, '\n')
 if newline < 0 {
 break
 }
 remaining := chunk[newline+1:]
 chunk = chunk[:newline+1]

 chunkLoop:
 for {
 var hash uint64
 var station, after []byte

 if len(chunk) < 8 {
 break chunkLoop
 }

 nameWord0 := binary.NativeEndian.Uint64(chunk)
 matchBits := semicolonMatchBits(nameWord0)
 if matchBits != 0 {
 // semicolon is in the first 8 bytes
 nameLen := calcNameLen(matchBits)
 nameWord0 = maskWord(nameWord0, matchBits)
 station = chunk[:nameLen]
 after = chunk[nameLen+1:]
 hash = calcHash(nameWord0)
 } else {
 // station name is longer so keep looking for the semicolon in
 // uint64 chunks
 nameLen := 8
 hash = calcHash(nameWord0)
 for {
 if nameLen > len(chunk)-8 {
 break chunkLoop
 }
 lastNameWord := binary.NativeEndian.Uint64(chunk[nameLen:])
 matchBits = semicolonMatchBits(lastNameWord)
 if matchBits != 0 {
 nameLen += calcNameLen(matchBits)
 station = chunk[:nameLen]
 after = chunk[nameLen+1:]
 break
 }
 nameLen += 8
 }
 }
 index := 0
 negative := false
 if after[index] == '-' {
 negative = true
 index++
 }
 temp := int32(after[index] - '0')
 index++
 if after[index] != '.' {
 temp = temp*10 + int32(after[index]-'0')
 index++
 }
 index++ // skip '.'
 temp = temp*10 + int32(after[index]-'0')
 index += 2 // skip last digit and '\n'
 if negative {
 temp = -temp
 }
 chunk = after[index:]

 hashIndex := int(hash & (numBuckets - 1))
 for {
 if items[hashIndex].key == nil {
 // Found empty slot, add new item (copying key).
 key := make([]byte, len(station))
 copy(key, station)
 items[hashIndex] = item{
 key: key,
 stat: &r10Stats{
 min: temp,
 max: temp,
 sum: int64(temp),
 count: 1,
 },
 }
 size++
 if size > numBuckets/2 {
 panic("too many items in hash table")
 }
 break
 }
 if bytes.Equal(items[hashIndex].key, station) {
 // Found matching slot, add to existing stats.
 s := items[hashIndex].stat
 s.min = min(s.min, temp)
 s.max = max(s.max, temp)
 s.sum += int64(temp)
 s.count++
 break
 }
 // Slot already holds another key, try next slot (linear probe).
 hashIndex++
 if hashIndex >= numBuckets {
 hashIndex = 0
 }
 }
 }

 readStart = copy(buf, remaining)
 }

 result := make(map[string]*r10Stats, size)
 for _, item := range items {
 if item.key == nil {
 continue
 }
 result[string(item.key)] = item.stat
 }
 resultsCh <- result
}

func calcNameLen(b uint64) int {
 return (bits.TrailingZeros64(b) >> 3)
}

func calcHash(word uint64) uint64 {
 return bits.RotateLeft64(word*0x51_7c_c1_b7_27_22_0a_95, 17)
}

func semicolonMatchBits(word uint64) uint64 {
 diff := word ^ BroadcastSemicolon
 return (diff - Broadcast0x01) & (^diff & Broadcast0x80)
}

func maskWord(word, matchBits uint64) uint64 {
 mask := matchBits ^ (matchBits - 1)
 return word & mask
}

Alexander Yastrebov

package main

import (
 "bytes"
 "fmt"
 "log"
 "math"
 "os"
 "runtime"
 "sort"
 "sync"
 "syscall"
)

type measurement struct {
 min, max, sum, count int64
}

func main() {
 if len(os.Args) != 2 {
 log.Fatalf("Missing measurements filename")
 }

 measurements := processFile(os.Args[1])

 ids := make([]string, 0, len(measurements))
 for id := range measurements {
 ids = append(ids, id)
 }
 sort.Strings(ids)

 fmt.Print("{")
 for i, id := range ids {
 if i > 0 {
 fmt.Print(", ")
 }
 m := measurements[id]
 fmt.Printf("%s=%.1f/%.1f/%.1f", id, round(float64(m.min)/10.0), round(float64(m.sum)/10.0/float64(m.count)), round(float64(m.max)/10.0))
 }
 fmt.Println("}")
}

func processFile(filename string) map[string]*measurement {
 f, err := os.Open(filename)
 if err != nil {
 log.Fatalf("Open: %v", err)
 }
 defer f.Close()

 fi, err := f.Stat()
 if err != nil {
 log.Fatalf("Stat: %v", err)
 }

 size := fi.Size()
 if size <= 0 || size != int64(int(size)) {
 log.Fatalf("Invalid file size: %d", size)
 }

 data, err := syscall.Mmap(int(f.Fd()), 0, int(size), syscall.PROT_READ, syscall.MAP_SHARED)
 if err != nil {
 log.Fatalf("Mmap: %v", err)
 }

 defer func() {
 if err := syscall.Munmap(data); err != nil {
 log.Fatalf("Munmap: %v", err)
 }
 }()

 return process(data)
}

func process(data []byte) map[string]*measurement {
 nChunks := runtime.NumCPU()

 chunkSize := len(data) / nChunks
 if chunkSize == 0 {
 chunkSize = len(data)
 }

 chunks := make([]int, 0, nChunks)
 offset := 0
 for offset < len(data) {
 offset += chunkSize
 if offset >= len(data) {
 chunks = append(chunks, len(data))
 break
 }

 nlPos := bytes.IndexByte(data[offset:], '\n')
 if nlPos == -1 {
 chunks = append(chunks, len(data))
 break
 } else {
 offset += nlPos + 1
 chunks = append(chunks, offset)
 }
 }

 var wg sync.WaitGroup
 wg.Add(len(chunks))

 results := make([]map[string]*measurement, len(chunks))
 start := 0
 for i, chunk := range chunks {
 go func(data []byte, i int) {
 results[i] = processChunk(data)
 wg.Done()
 }(data[start:chunk], i)
 start = chunk
 }
 wg.Wait()

 measurements := make(map[string]*measurement)
 for _, r := range results {
 for id, rm := range r {
 m := measurements[id]
 if m == nil {
 measurements[id] = rm
 } else {
 m.min = min(m.min, rm.min)
 m.max = max(m.max, rm.max)
 m.sum += rm.sum
 m.count += rm.count
 }
 }
 }
 return measurements
}

func processChunk(data []byte) map[string]*measurement {
 // Use fixed size linear probe lookup table
 const (
 // use power of 2 for fast modulo calculation,
 // should be larger than max number of keys which is 10_000
 entriesSize = 1 << 14

 // use FNV-1a hash
 fnv1aOffset64 = 14695981039346656037
 fnv1aPrime64 = 1099511628211
 )

 type entry struct {
 m measurement
 hash uint64
 vlen int
 value [128]byte // use power of 2 > 100 for alignment
 }
 entries := make([]entry, entriesSize)
 entriesCount := 0

 // keep short and inlinable
 getMeasurement := func(hash uint64, value []byte) *measurement {
 i := hash & uint64(entriesSize-1)
 entry := &entries[i]

 // bytes.Equal could be commented to speedup assuming no hash collisions
 for entry.vlen > 0 && !(entry.hash == hash && bytes.Equal(entry.value[:entry.vlen], value)) {
 i = (i + 1) & uint64(entriesSize-1)
 entry = &entries[i]
 }

 if entry.vlen == 0 {
 entry.hash = hash
 entry.vlen = copy(entry.value[:], value)
 entriesCount++
 }
 return &entry.m
 }

 // assume valid input
 for len(data) > 0 {

 idHash := uint64(fnv1aOffset64)
 semiPos := 0
 for i, b := range data {
 if b == ';' {
 semiPos = i
 break
 }

 // calculate FNV-1a hash
 idHash ^= uint64(b)
 idHash *= fnv1aPrime64
 }

 idData := data[:semiPos]

 data = data[semiPos+1:]

 var temp int64
 // parseNumber
 {
 negative := data[0] == '-'
 if negative {
 data = data[1:]
 }

 _ = data[3]
 if data[1] == '.' {
 // 1.2\n
 temp = int64(data[0])*10 + int64(data[2]) - '0'*(10+1)
 data = data[4:]
 // 12.3\n
 } else {
 _ = data[4]
 temp = int64(data[0])*100 + int64(data[1])*10 + int64(data[3]) - '0'*(100+10+1)
 data = data[5:]
 }

 if negative {
 temp = -temp
 }
 }

 m := getMeasurement(idHash, idData)
 if m.count == 0 {
 m.min = temp
 m.max = temp
 m.sum = temp
 m.count = 1
 } else {
 m.min = min(m.min, temp)
 m.max = max(m.max, temp)
 m.sum += temp
 m.count++
 }
 }

 result := make(map[string]*measurement, entriesCount)
 for i := range entries {
 entry := &entries[i]
 if entry.m.count > 0 {
 result[string(entry.value[:entry.vlen])] = &entry.m
 }
 }
 return result
}

func round(x float64) float64 {
 return roundJava(x*10.0) / 10.0
}

// roundJava returns the closest integer to the argument, with ties
// rounding to positive infinity, see java's Math.round
func roundJava(x float64) float64 {
 t := math.Trunc(x)
 if x < 0.0 && t-x == 0.5 {
 //return t
 } else if math.Abs(x-t) >= 0.5 {
 t += math.Copysign(1, x)
 }

 if t == 0 { // check -0
 return 0.0
 }
 return t
}

// parseNumber reads decimal number that matches "^-?[0-9]{1,2}[.][0-9]" pattern,
// e.g.: -12.3, -3.4, 5.6, 78.9 and return the value*10, i.e. -123, -34, 56, 789.
func parseNumber(data []byte) int64 {
 negative := data[0] == '-'
 if negative {
 data = data[1:]
 }

 var result int64
 switch len(data) {
 // 1.2
 case 3:
 result = int64(data[0])*10 + int64(data[2]) - '0'*(10+1)
 // 12.3
 case 4:
 result = int64(data[0])*100 + int64(data[1])*10 + int64(data[3]) - '0'*(100+10+1)
 }

 if negative {
 return -result
 }
 return result
}

https://github.com/aytechnet/1brc

François Pons

package main

import (
 "bytes"
 "flag"
 "fmt"
 "log"
 "os"
 "runtime"
 "runtime/pprof"
 "slices"
 "sync"
 "syscall"
 "sync/atomic"

 "github.com/aytechnet/decimal"
)

type (
 measurement struct {
 hash atomic.Uint64
 minT atomic.Int64
 maxT atomic.Int64
 sumT atomic.Int64
 countT atomic.Int64
 nameLen int
 nameBuf [208]byte
 }

 measurements struct {
 total atomic.Int64
 numParsers int
 results [capacity]measurement
 }

 job struct {
 maxOffset atomic.Int64
 bufLen int
 buf [bufSize]byte
 }
)

const (
 delta = 439
 capacity = 1 << 16 // must be a power of 2 for modulo calculation

 // buffer size
 bufSize = 512 * 1024 // 1Mb

 // use FNV-1a hash
 fnv1aOffset64 = 14695981039346656037
 fnv1aPrime64 = 1099511628211
)

func main() {
 var mode, filename, cpuprofile string
 var res measurements

 flag.StringVar(&mode, "mode", "default", "Which mode to use among 'mmap', 'seq' and 'default'")
 flag.StringVar(&filename, "file", "", "Measurements file to use")
 flag.StringVar(&cpuprofile, "cpuprofile", "", "Write cpuprofile to file")
 flag.IntVar(&res.numParsers, "parsers", runtime.NumCPU(), "Number of thread to use for parsing")

 flag.Parse()

 if filename == "" {
 flag.Usage()
 os.Exit(1)
 }

 if cpuprofile != "" {
 if f, err := os.Create(cpuprofile); err != nil {
 log.Fatal(err)
 } else {
 pprof.StartCPUProfile(f)
 defer pprof.StopCPUProfile()
 }
 }

 f, err := os.Open(filename)
 if err != nil {
 log.Fatalf("Open: %v", err)
 }
 defer f.Close()

 switch mode {
 case "mmap":
 res.processMmap(f)
 case "seq":
 res.processSeq(f)
 default:
 res.process(f)
 }
}

func (res *measurements) processMmap(f *os.File) {
 jobs := make([]job, res.numParsers)

 fi, err := f.Stat()
 if err != nil {
 log.Fatalf("Stat: %v", err)
 }

 size := fi.Size()
 chunkSize := size/int64(len(jobs))
 if chunkSize <= 100 {
 log.Fatalf("Invalid file size: %d", size)
 }

 data, err := syscall.Mmap(int(f.Fd()), 0, int(size), syscall.PROT_READ, syscall.MAP_SHARED)
 if err != nil {
 log.Fatalf("Mmap: %v", err)
 }

 defer func() {
 if err := syscall.Munmap(data); err != nil {
 log.Fatalf("Munmap: %v", err)
 }
 }()

 offset := chunkSize
 for i := range jobs {
 j := &jobs[i]

 if i == len(jobs)-1 {
 j.maxOffset.Store(size)
 } else {
 j.maxOffset.Store(-offset)
 }
 offset += chunkSize
 }

 var wg sync.WaitGroup

 wg.Add(len(jobs))

 offset = 0
 for i := range jobs {
 go func(i int, offset int64){
 defer wg.Done()

 j := &jobs[i]

 maxOffset := j.maxOffset.Load()
 if maxOffset < 0 {
 maxOffset = -maxOffset

 if nlPos := bytes.IndexByte(data[maxOffset:], '\n'); nlPos >= 0 {
 maxOffset += int64(nlPos+1)
 }
 }

 if offset > 0 {
 if nlPos := bytes.IndexByte(data[offset:maxOffset], '\n'); nlPos >= 0 {
 offset += int64(nlPos+1)
 }
 }

 res.processChunk(data[offset:maxOffset])
 }(i, offset)

 offset += chunkSize
 }

 wg.Wait()

 res.printResults()
}

func (res *measurements) processSeq(f *os.File) {
 jobs := make([]job, res.numParsers)
 free_jobs := make(chan *job, len(jobs))
 ready_jobs := make(chan *job, len(jobs))

 var wg sync.WaitGroup

 wg.Add(len(jobs))

 for i := range jobs {
 j := &jobs[i]

 free_jobs <- j
 go func(){
 for j := range ready_jobs {
 res.processChunk(j.buf[:j.bufLen])
 free_jobs <- j
 }
 wg.Done()
 }()
 }

 var prev_j *job

 pos := 0
 nlPos := 0
 for j := range free_jobs {
 if count, _ := f.Read(j.buf[pos:]); count > 0 {
 // finalize previous line from previous job
 if pos > 0 /* && prev_j != nil */ {
 copy(j.buf[:pos], prev_j.buf[bufSize-pos:])
 }

 // prepare next buffer
 if nlPos = bytes.LastIndexByte(j.buf[:pos+count], '\n'); nlPos < 0 {
 log.Fatalf("buffer too small for a complete line");
 } else {
 pos = pos + count - nlPos - 1
 prev_j = j
 j.bufLen = nlPos + 1
 }

 // spawn a new job on this buffer
 ready_jobs <- j
 } else {
 break
 }
 }

 close(ready_jobs)

 wg.Wait()

 close(free_jobs)

 res.printResults()
}

func (res *measurements) process(f *os.File) {
 jobs := make([]job, res.numParsers)

 fi, err := f.Stat()
 if err != nil {
 log.Fatalf("Stat: %v", err)
 }

 size := fi.Size()
 chunkSize := size/int64(len(jobs))
 if chunkSize <= 100 {
 log.Fatalf("Invalid file size: %d", size)
 }

 offset := chunkSize
 for i := range jobs {
 j := &jobs[i]

 if i == len(jobs)-1 {
 j.maxOffset.Store(size)
 } else {
 j.maxOffset.Store(-offset)
 }
 offset += chunkSize
 }

 var wg sync.WaitGroup

 wg.Add(len(jobs))

 offset = 0
 for i := range jobs {
 go func(i int, offset int64){
 defer wg.Done()

 j := &jobs[i]

 nlSkipFirst := offset > 0

 for {
 maxLen := bufSize
 maxOffset := j.maxOffset.Load()
 if maxOffset < 0 {
 maxOffset = -maxOffset
 }
 if offset + int64(maxLen) > maxOffset {
 maxLen = int(maxOffset - offset)
 }

 if count, _ := f.ReadAt(j.buf[:maxLen], offset); count > 0 {
 pos := 0
 if nlSkipFirst {
 if nlPos := bytes.IndexByte(j.buf[:maxLen], '\n'); nlPos >= 0 {
 pos = nlPos + 1
 jobs[i-1].maxOffset.Store(offset + int64(pos))
 nlSkipFirst = false
 } else {
 log.Fatalf("Unable to seek to next line at job n°%d", i)
 }
 }

 if nlPos := bytes.LastIndexByte(j.buf[pos:maxLen], '\n'); nlPos >= 0 {
 j.bufLen = pos + nlPos + 1
 offset += int64(j.bufLen)

 res.processChunk(j.buf[pos:j.bufLen])
 } else {
 log.Fatalf("Buffer too small at job n°%d", i)
 }
 } else {
 return
 }

 maxOffset = j.maxOffset.Load()
 for maxOffset < 0 {
 maxOffset = j.maxOffset.Load()
 }
 if offset >= maxOffset {
 return
 }
 }
 }(i, offset)

 offset += chunkSize
 }

 wg.Wait()

 res.printResults()
}

func (res *measurements) printResults() {
 log.Printf("Read %d entries", res.total.Load())

 ids := make([]int, 0, capacity)

 for i := range res.results {
 m := &res.results[i]

 if m.nameLen > 0 {
 ids = append(ids, i)
 }
 }

 slices.SortFunc(ids, func(a,b int) int {
 return bytes.Compare(res.results[a].nameBuf[0:res.results[a].nameLen], res.results[b].nameBuf[0:res.results[b].nameLen])
 })

 count := 0
 fmt.Print("{")
 for _, i := range ids {
 m := &res.results[i]

 var buf [128]byte // name is 100 chars max, each 3 decimals is 5 bytes max on output
 b := buf[0:0]

 if count > 0 {
 b = append(b, ',', ' ')
 }

 b = append(b, m.nameBuf[0:m.nameLen]...)
 b = append(b, '=')
 b = decimal.New(m.minT.Load(), -1).BytesToFixed(b, 1)
 b = append(b, '/')
 b = decimal.New(m.sumT.Load(), -1).Div(decimal.NewFromInt(m.countT.Load()-1)).BytesToFixed(b, 1)
 b = append(b, '/')
 b = decimal.New(m.maxT.Load(), -1).BytesToFixed(b, 1)
 count++

 fmt.Print(string(b))
 // fmt.Printf("%s=%s/%s/%s", m.nameBuf[0:m.nameLen], decimal.New(m.minT.Load(), -1).StringFixed(1), decimal.New(m.sumT.Load(), -1).Div(decimal.NewFromInt(m.countT.Load()-1)).StringFixed(1), decimal.New(m.maxT.Load(), -1).StringFixed(1))
 count++
 }
 fmt.Println("}")
}

func (res *measurements) processChunk(data []byte) {
 var total int64

 // assume valid input
 for len(data) > 0 {
 i := 0

 // compute FNV-1a hash
 idHash := uint64(fnv1aOffset64)
 for j, b := range data {
 if b == ';' {
 i = j
 break
 }

 // calculate FNV-1a hash
 idHash ^= uint64(b)
 idHash *= fnv1aPrime64
 }
 if idHash == 0 {
 idHash = uint64(len(data))
 }

 idData := data[:i]

 i++ // now i points to temperature

 var temp int64
 // parseNumber
 {
 negative := data[i] == '-'
 if negative {
 i++
 }

 temp = int64(data[i]-'0')
 i++

 if data[i] != '.' {
 temp = temp*10 + int64(data[i]-'0')
 i++
 }
 i++ // data[i] is '.'
 temp = temp*10 + int64(data[i]-'0')
 if negative {
 temp = -temp
 }

 data = data[i+2:]
 }

 // get measurement
 {
 i := idHash & uint64(capacity-1)
 entry := &res.results[i]

 for {
 if entry.hash.CompareAndSwap(0, idHash) {
 // make sure no race occurs as entry may be updated meanwhile as hash has been updated
 if entry.countT.Add(1) == 1 {
 entry.nameLen = len(idData)
 copy(entry.nameBuf[:], idData)
 entry.minT.Store(temp)
 entry.maxT.Store(temp)
 entry.sumT.Store(temp)
 entry.countT.Add(1) // unlock for update below
 } else {
 // wait for countT to be at least 2 for entry init to be complete
 for entry.countT.Load() < 2 {}

 // update existing entry
 minT := entry.minT.Load()
 for minT > temp {
 if entry.minT.CompareAndSwap(minT, temp) {
 break
 } else {
 minT = entry.minT.Load()
 }
 }
 maxT := entry.maxT.Load()
 for maxT < temp {
 if entry.maxT.CompareAndSwap(maxT, temp) {
 break
 } else {
 maxT = entry.maxT.Load()
 }
 }
 entry.sumT.Add(temp)
 entry.countT.Add(1)
 }
 break
 } else if entry.hash.Load() == idHash {
 // the entry is found and may be being updated by another thread
 // wait for countT to be at least 2 for entry init to be complete
 for entry.countT.Load() < 2 {}

 // now that name has been updated, check it is matching
 if len(idData) == entry.nameLen /* bytes.Compare(idData, entry.nameBuf[0:entry.nameLen]) == 0 */ {
 // update existing entry
 minT := entry.minT.Load()
 for minT > temp {
 if entry.minT.CompareAndSwap(minT, temp) {
 break
 } else {
 minT = entry.minT.Load()
 }
 }
 maxT := entry.maxT.Load()
 for maxT < temp {
 if entry.maxT.CompareAndSwap(maxT, temp) {
 break
 } else {
 maxT = entry.maxT.Load()
 }
 }
 entry.sumT.Add(temp)
 entry.countT.Add(1)

 break
 } else {
 // name does not match idData so jump to next entry
 i = (i + delta) & uint64(capacity-1)
 entry = &res.results[i]
 }
 } else {
 i = (i + delta) & uint64(capacity-1)
 entry = &res.results[i]
 }
 }
 }

 total++
 }

 res.total.Add(total)
}

func (entry *measurement) update(temp int64) {
 minT := entry.minT.Load()
 for minT > temp {
 if entry.minT.CompareAndSwap(minT, temp) {
 break
 } else {
 minT = entry.minT.Load()
 }
 }
 maxT := entry.maxT.Load()
 for maxT < temp {
 if entry.maxT.CompareAndSwap(maxT, temp) {
 break
 } else {
 maxT = entry.maxT.Load()
 }
 }
 entry.sumT.Add(temp)
 entry.countT.Add(1)
}

Go leetcode

Thu, 20 Jun 2024 18:15:00 +0800

https://books.halfrost.com/leetcode/

More Powerful Go Execution Tracing

Tue, 07 May 2024 11:45:57 +0800

More Powerful Go Execution Tracing

runtime/trace

golang.org/x/exp/trace

Flight Recording

// Set up the flight recorder
fr := trace.NewFlightRecorder()
fr.Start()
// Set up and run the HTTP server
var once sync.Once
http.HandleFunc("/my-endpoint", func(w http.ResponseWriter, r *http.Request) {
 start := time.Now()
 // Do some work
 doWork(w, r)
 // Got a long-running request, take a snapshot
 if time.Since(start) > 300*time.Millisecond {
 // We only do this once for simplicity; you can do it multiple times
 once.Do(func() {
 // Capture snapshot
 var b bytes.Buffer
 _, err = fr.WriteTo(&b)
 if err != nil {
 log.Print(err)
 return
 }
 // Write snapshot to file
 if err := os.WriteFile("trace.out", b.Bytes(), 0o755); err != nil {
 log.Print(err)
 return
 }
 })
 }
})
log.Fatal(http.ListenAndServe(":8080", nil))

Trace Reader API

// Start reading trace data from stdin.
r, err := trace.NewReader(os.Stdin)
if err != nil {
 log.Fatal(err)
}
var blocked int
var blockedOnNetwork int
for {
 // Read event
 ev, err := r.ReadEvent()
 if err == io.EOF {
 break
 } else if err != nil {
 log.Fatal(err)
 }
 // Handle it
 if ev.Kind() == trace.EventStateTransition {
 st := ev.StateTransition()
 if st.Resource.Kind == trace.ResourceGoroutine {
 id := st.Resource.Goroutine()
 from, to := st.GoroutineTransition()
 // Find blocked goroutines and count them
 if from.Executing() && to == trace.GoWaiting {
 blocked++
 if strings.Contains(st.Reason, "network") {
 blockedOnNetwork++
 }
 }
 }
 }
}

// Print the values we need
p := 100 * float64(blockedOnNetwork) / float64(blocked)
fmt.Printf("%2.3f%% instances of goroutines blocking were to block on the network\n", p)

Run llama3

Thu, 25 Apr 2024 10:14:00 +0800

Docker-compose

version: "3.8"

services:
 ollama:
 image: ollama/ollama:latest
 container_name: ollama
 restart: unless-stopped
 volumes:
 - ./ollama/ollama:/root/.ollama
 tty: true
 ports:
 - 11434:11434
 networks:
 - ollama-docker
 # deploy:
 # resources:
 # reservations:
 # devices:
 # - driver: nvidia
 # count: 1
 # capabilities: [gpu]
 ollama-webui:
 image: ghcr.io/ollama-webui/ollama-webui:main
 container_name: ollama-webui
 restart: unless-stopped
 volumes:
 - ./ollama/ollama-webui:/app/backend/data
 ports:
 - 8080:8080
 environment:
 - "/ollama/api=http://ollama:11434/api"
 extra_hosts:
 - host.docker.internal:host-gateway
 networks:
 - ollama-docker

networks:
 ollama-docker:

Setup

# Run docker-compose
docker-compose up -d

# Pull model(https://ollama.com/library)
docker exec -it ollama /bin/bash
ollama pull llama3

Chat with Web-UI

port defined in docker-compose.yml ollama-webui.ports

Go 1.21 GA PGO Optimization You Can't Grasp in One Read — A Try on WebP Server Go

Thu, 25 Apr 2024 09:22:00 +0800

Go 1.21 GA PGO Optimization You Can’t Grasp in One Read — A Try on WebP Server Go

Go Style Decisions - Pass values

Sat, 13 Apr 2024 17:38:00 +0800

Go Style Decisions - Pass values

Do not pass pointers as function arguments just to save a few bytes. If a function reads its argument x only as *x throughout, then the argument shouldn’t be a pointer. Common instances of this include passing a pointer to a string (*string) or a pointer to an interface value (*io.Reader). In both cases, the value itself is a fixed size and can be passed directly.

Go Generics Beginner Notes

Wed, 07 Feb 2024 19:52:22 +0800

Go Generics Beginner Notes

Custom Constraints

// Addable allows only int or float64 types
type Addable interface {
 int | float64
}

func Add[T Addable](a, b T) T {
 return a + b
}

What ~ Means ~ represents all types that share the same underlying type as the specified type. When you use ~ in a type parameter constraint, you define a type set that includes all types whose underlying type matches the specified type.

type MyInt int
type YourInt int

func PrintInt[T ~int](t T) {
 fmt.Println(t)
}

func main() {
 var a int = 5
 var b MyInt = 10
 var c YourInt = 15

 PrintInt(a)
 PrintInt(b)
 PrintInt(c)
}

Golang Tips

Tue, 26 Sep 2023 22:15:00 +0800

Go Programming Patterns: Slices, Interfaces, Time, and Performance

Go is a high-performance language, but that does not mean we can ignore performance. Here are some tips related to programming and performance.

If you need to convert numbers to strings, strconv.Itoa() is about twice as fast as fmt.Sprintf().
Avoid converting String to []Byte where possible. This conversion hurts performance.
If you use append() on a slice inside a for-loop, expand the slice capacity ahead of time to avoid reallocations and automatic growth by powers of two, which can waste memory.
Use StringBuffer or StringBuild to concatenate strings; they are three to four orders of magnitude faster than + or +=.
Use concurrent goroutines where possible, and use sync.WaitGroup to synchronize concurrent work.
Avoid allocating in hot paths; it makes the GC busy. Use sync.Pool to reuse objects.
Use lock-free operations and avoid mutexes when possible; use sync/Atomic. (See Lock-Free Queue Implementation or Lock-Free Hashmap Implementation.)
Use I/O buffering; I/O is very slow. bufio.NewWrite() and bufio.NewReader() improve performance.
For fixed regex patterns in a for-loop, use regexp.Compile() to precompile; it improves performance by two orders of magnitude.
If you need higher-performance protocols, consider protobuf or msgp instead of JSON, because JSON serialization/deserialization uses reflection.
When using maps, integer keys are faster than string keys because integer comparisons are faster.

Go Patterns

Sat, 16 Sep 2023 16:08:12 +0800

15 Rules of Channels and Their Implementation

Thu, 22 Jun 2023 21:25:05 +0800

15 Rules of Channels and Their Implementation

Operation rules

Operation	nil	Closed channel	Open buffered channel	Open unbuffered channel
Close	panic	panic	Close succeeds; buffered values can be read. After buffer drains, further reads return the zero value of the channel type.	Close succeeds; further reads return the zero value of the channel type.
Receive	Block	Non-blocking; returns the zero value of the channel type	Non-blocking; reads values normally	Block
Send	Block	panic	Non-blocking; writes values normally	Block

Compile-time rules

Operation	Channel type	Result
Receive	Send-only channel	Compile error
Send	Receive-only channel	Compile error
Close	Receive-only channel	Compile error

Pitfalls When Parsing Binary Data in Go

Fri, 20 Jan 2023 13:48:27 +0800

Pitfalls When Parsing Binary Data in Go

for {
 buf := make([]byte, bufSize)
 n, err := r.Read(buf)
 if err == io.EOF {
 return
 } else if err != nil {
 aerr.Store(err)
 return
 }
 ch <- buf[:n]
}

The code returns immediately when err is io.EOF, which is the most likely problem. The docs say:

Callers should always process the n > 0 bytes returned before considering the error err. Doing so correctly handles I/O errors that happen after reading some bytes and also both of the allowed EOF behaviors.

Take Screenshots with Selenium in Go

Fri, 02 Dec 2022 13:40:19 +0800

Take Screenshots with Selenium in Go

package main

import (
 "fmt"
 log "github.com/sirupsen/logrus"
 "github.com/tebeka/selenium"
 "io/ioutil"
)

const (
 chromeDriverPath = "/usr/local/bin/chromedriver"
 port = 9515
)

func main() {
 var opts []selenium.ServiceOption
 selenium.SetDebug(false)
 service, err := selenium.NewChromeDriverService(chromeDriverPath, port, opts...)
 if err != nil {
 panic(err) // panic is used only as an example and is not otherwise recommended.
 }
 defer service.Stop()

 caps := selenium.Capabilities{"browserName": "chrome"}
 // Full-page screenshot
 // Get page width/height, resize the browser window, then take a screenshot.
 // Note: only headless mode allows arbitrary window sizes; otherwise the max height cannot exceed your screen resolution.
 // chromeCaps := chrome.Capabilities{
 // Path: "",
 // Args: []string{
 // "--headless",
 // },
 // }
 // caps.AddChrome(chromeCaps)

 wd, err := selenium.NewRemote(caps, fmt.Sprintf("http://localhost:%d/wd/hub", port))
 if err != nil {
 panic(err)
 }
 defer wd.Quit()

 // ExecuteScript can run JavaScript. The return value is interface{}, so assert to float then convert to int.
 // height, _ := wd.ExecuteScript("return document.body.parentNode.scrollHeight", nil)
 // var realHeight = int(height.(float64))
 // Then set the window size.
 // wd.ResizeWindow("", 1920, realHeight)

 if err := wd.Get("https://github.com/tgbot-collection/archiver"); err != nil {
 panic(err)
 }

 screenshot, err := wd.Screenshot()
 if err != nil {
 log.Errorln(err)

 }
 ioutil.WriteFile("screenshot.png", screenshot, 0644)

}

Full-page screenshot

Get the page width/height, resize the browser window accordingly, and then take the screenshot. Note that only headless mode allows arbitrary window sizes; otherwise the maximum height cannot exceed your display resolution.

Use Go Fuzzing to Write More Complete Unit Tests

Thu, 03 Nov 2022 16:57:29 +0800

Use Go Fuzzing to Write More Complete Unit Tests

func Pow(base uint, exponent uint) uint {
 if exponent == 0 {
 return 1
 }

 return base * Pow(base, exponent-1)
}

func FuzzPow(f *testing.F) {
 f.Fuzz(func(t *testing.T, x, y uint) {
 assert := assert.New(t)

 assert.Equal(Pow(x, 0), uint(1))
 assert.Equal(Pow(x, 1), x)
 assert.Equal(Pow(x, 2), x*x)

 if x > 0 && y > 0 {
 assert.Equal(Pow(x, y)/x, Pow(x, y-1))
 }
 })
}

go test -fuzz=Fuzz -fuzztime 20s

When a fuzz test fails, Go records the input in testcase/
You will find that when x=6 and y=30, the assert fails, i.e., pow(6, 30)/6 is not equal to pow(6, 29). That seems odd, but after testing you will see it is because pow(6, 30) overflows.
The max.MaxUint in Go is about 18 * 10^¹⁸, while 6^²⁹ is about 7 * 10^¹⁸. If you multiply 6^²⁹ by 6, it overflows and yields 8 * 10^¹⁸. It is like running two laps and ending up near the starting point.

var ErrOverflow = fmt.Errorf("overflow")

func Pow(base uint, exponent uint) (uint, error) {
 if exponent == 0 {
 return 1
 }

 prevResult, err := Pow(base, exponent-1)
 if math.MaxUint/base < prevResult {
 return 0, ErrOverflow
 }

 return base * prevResult, nil
}

func FuzzPow(f *testing.F) {
 f.Fuzz(func(t *testing.T, x, y uint) {
 assert := assert.New(t)
 if result, err := Pow(x, 1); err != ErrOverflow {
 assert.Equal(result, x)
 }

 if result, err := Pow(x, y); x > 0 && y > 0 && err != ErrOverflow {
 resultDivX, _ := Pow(x, y-1)
 assert.Equal(result/x, resultDivX)
 }
 })
}

Go string format

Thu, 08 Sep 2022 11:32:00 +0800

The verb at the end defines the type and the interpretation of its corresponding argument.

d - decimal integer
o - octal integer
O - octal integer with 0o prefix
b - binary integer
x - hexadecimal integer lowercase
X - hexadecimal integer uppercase
f - decimal floating point, lowercase
F - decimal floating point, uppercase
e - scientific notation (mantissa/exponent), lowercase
E - scientific notation (mantissa/exponent), uppercase
g - the shortest representation of %e or %f
G - the shortest representation of %E or %F
c - a character represented by the corresponding Unicode code point
q - a quoted character
U - Unicode escape sequence
t - the word true or false
s - a string
v - default format
#v - Go-syntax representation of the value
T - a Go-syntax representation of the type of the value
p - pointer address
% - a double %% prints a single %

Go string format indexing

package main

import (
 "fmt"
)

func main() {

 n1 := 2
 n2 := 3
 n3 := 4

 res := fmt.Sprintf("There are %d oranges %d apples %d plums", n1, n2, n3)
 fmt.Println(res) // There are 2 oranges 3 apples 4 plums

 res2 := fmt.Sprintf("There are %[2]d oranges %d apples %[1]d plums", n1, n2, n3)
 fmt.Println(res2) // There are 3 oranges 4 apples 2 plums
}

Go string format precision

package main

import (
 "fmt"
)

func main() {

 fmt.Printf("%0.f\n", 16.540) // 17
 fmt.Printf("%0.2f\n", 16.540) // 16.54
 fmt.Printf("%0.3f\n", 16.540) // 16.540
 fmt.Printf("%0.5f\n", 16.540) // 16.54000
}

Go string format flags

package main

import (
 "fmt"
)

func main() {

 fmt.Printf("%+d\n", 1691) // +1691

 fmt.Printf("%#x\n", 1691) // 0x69b
 fmt.Printf("%#X\n", 1691) // 0X69B
 fmt.Printf("%#b\n", 1691) // 0b11010011011

 fmt.Printf("%10d\n", 1691) // 1691
 fmt.Printf("%-10d\n", 1691) // 1691
 fmt.Printf("%010d\n", 1691) // 0000001691
}

Go string format width

package main

import (
 "fmt"
)

func main() {

 w := "falcon"
 n := 122
 h := 455.67

 fmt.Printf("%s\n", w) // falcon
 fmt.Printf("%10s\n", w) // falcon

 fmt.Printf("%d\n", n). // 122
 fmt.Printf("%7d\n", n) // 122
 fmt.Printf("%07d\n", n) // 0000122

 fmt.Printf("%10f\n", h) // 455.670000
 fmt.Printf("%11f\n", h) // 455.670000
 fmt.Printf("%12f\n", h) // 455.670000
}

Build a Lightweight Docker Image for Your Go App? | IT Man

Mon, 25 Jul 2022 17:33:47 +0800

Build a Lightweight Docker Image for Your Go App? | IT Man

go build

# default
$ go build -o test1 main.go
$ du -sh test1
14M test1


# You can add `-ldflags "-s -w"` during compilation to reduce the binary size by stripping some link and debug info. Details:
# -a: force rebuilding all dependencies
# -s: drop symbol table info; stack traces in panic will lose file/line info
# -w: drop DWARF debug info; you cannot debug with gdb
# If you don't need the symbol table, you can just use "-s"
# Note: it is not recommended to use -w and -s together
$ go build -ldflags "-s -w" -o test2 main.go
$ du -sh test2
11M test2

upx(`brew/yum install upx`)

$ upx test2
 Ultimate Packer for eXecutables
 Copyright (C) 1996 - 2020
UPX 3.96 Markus Oberhumer, Laszlo Molnar & John Reiser Jan 23rd 2020
 File size Ratio Format Name
 -------------------- ------ ----------- -----------
 11490768 -> 4063248 35.36% macho/amd64 test2
Packed 1 file.

$ upx --brute test2
$ du -sh test2
4.6M test2

UPX compression options

[Golang] Use build tags for different build configs

Mon, 18 Jul 2022 14:21:34 +0800

[Golang] Use build tags for different build configs

Go build -ldflags

package main

import (
 "fmt"
)

var flagString string

func main() {
 fmt.Println("This build with ldflag:", flagString)
}

$ go build -ldflags '-X main.flagString "test"'

$ ./main
This build with ldflag: test

Go build -tags

debug_config.go

//+build debug

package main

var TestString string = "test debug"
var TestString2 string = " and it will run every module with debug tag."

func GetConfigString() string {
 return "it is debug....."
}

//+build debug needs a blank line before and after it (unless it is on the first line).

release_config.go

//+build !debug

package main

var TestString string = "test release"
var TestString2 string = " and it will run every module as no debug tag."

func GetConfigString() string {
 return "it is release....."
}

main

package main

import (
 "fmt"
)

func main() {
 fmt.Println("This build is ", TestString, TestString2, GetConfigString())
}

$ go build -tags debug .
$ ./main
This build is test debug and it will run every module with debug tag. it is debug.....

$ go build .
$ ./main
This build is test release and it will run every module as no debug tag. it is release.....

A Concise Go embed Tutorial

Tue, 12 Jul 2022 13:57:03 +0800

A Concise Go embed Tutorial

Embedding

For a single file, you can embed it as string or []byte
For multiple files and directories, you can embed them as a new filesystem (FS)
Import the “embed” package even if it is not explicitly used
The go:embed directive is used for embedding and must be followed by the target variable name
Only string, []byte, and embed.FS are supported. Aliases and named types (e.g., type S string) are not allowed
Uses relative paths

string/[]byte

package main

import (
 _ "embed"
 "fmt"
)

//go:embed hello.txt
var s string

//go:embed hello.txt
var b []byte

func main() {
 fmt.Println(s)
 fmt.Println(b)
}

embed.FS

package main

import (
 "embed"
 "fmt"
)

//go:embed hello.txt hello2.txt
//go:embed "p/he llo.txt"
//go:embed `p/hello-2.txt`
//go:embed dir
var f embed.FS
func main() {
 data, _ := f.ReadFile("hello.txt")
 fmt.Println(string(data))
 data, _ = f.ReadFile("hello2.txt")
 fmt.Println(string(data))
 data, _ := f.ReadFile("p/he llo.txt")
 fmt.Println(string(data))
 data, _ = f.ReadFile("p/hello-2.txt")
 fmt.Println(string(data))
}

Matching patterns

go:embed can take just a directory name. All files and directories inside, except those starting with . and _, are embedded. Subdirectories are embedded recursively to form a filesystem for that directory.

Go pprof in Practice

Wed, 22 Jun 2022 15:29:02 +0800

package main

import (
 // omitted
 _ "net/http/pprof" // auto-register handlers to the HTTP server for profiling via HTTP
 // omitted
)

func main() {
 // omitted

 runtime.GOMAXPROCS(1) // limit CPU usage to avoid overload
 runtime.SetMutexProfileFraction(1) // enable lock contention profiling
 runtime.SetBlockProfileRate(1) // enable blocking operation profiling

 go func() {
 // start an HTTP server; pprof handlers have already been registered
 // /debug/pprof/
 if err := http.ListenAndServe(":6060", nil); err != nil {
 log.Fatal(err)
 }
 os.Exit(0)
 }()

 // omitted
}

http://localhost:6060/debug/pprof/

Type	Description	Notes
allocs	Samples of memory allocations	Can open in a browser, but readability is low
blocks	Samples of blocking operations	Can open in a browser, but readability is low
cmdline	Show program startup command/args	Can open in a browser; shows ./go-pprof-practice
goroutine	Stacks of all current goroutines	Can open in a browser, but readability is low
heap	Samples of heap memory usage	Can open in a browser, but readability is low
mutex	Samples of lock contention	Can open in a browser, but readability is low
profile	Samples of CPU usage	Opening in a browser downloads a file
threadcreate	Samples of system thread creation	Can open in a browser, but readability is low
trace	Program execution traces	Opening in a browser downloads a file; not covered here, see Go trace

Investigate high CPU usage

Go code refactoring : the 23x performance hunt

Tue, 07 Jun 2022 13:42:07 +0800

Go code refactoring : the 23x performance hunt

$ go test -bench=. -cpuprofile cpu.prof
$ go tool pprof -svg cpu.prof > cpu.svg

$ go test -bench=. -trace trace.out
$ go tool trace trace.out

Dockertest: A Fast Way to Set Up Integration Test Environments

Thu, 12 May 2022 16:43:31 +0800

Go High-Performance Programming Handbook (Long Read)

Fri, 22 Apr 2022 18:01:43 +0800

Go High-Performance Programming Handbook (Long Read)

Common data structures

Reflection is nice, but don’t overuse it

Prefer strconv over fmt
A small amount of repetition is not worse than reflection
Use binary.Read and binary.Write with caution
- binary.Read and binary.Write use reflection and are slow. If you need these two functions, implement the required logic manually instead of using them directly.

Avoid repeated conversions from string to byte slice

Specify container (slice/map) capacity

How to Choose a Go File Reading Approach

Tue, 29 Mar 2022 16:15:34 +0800

How to Choose a Go File Reading Approach

Go provides one-shot methods for reading file content: os.ReadFile and ioutil.ReadFile. Starting from Go 1.16, ioutil.ReadFile is equivalent to os.ReadFile.

The pros and cons of loading a file in one shot are obvious: it reduces IO operations, but it loads the entire file into memory. For large files, this risks running out of memory.

Read line by line

In Go, the bufio.Reader type provides ReadLine(), but in practice we more often use ReadBytes(’\n’) or ReadString(’\n’).

Struct Tricks Every Gopher Should Know

Mon, 03 Jan 2022 15:08:31 +0800

Struct Tricks Every Gopher Should Know

NoCopy

$ go vet aaa.go
# command-line-arguments
./aaa.go:7:14: test passes lock by value: sync.WaitGroup contains sync.noCopy
./aaa.go:15:10: call of test copies lock value: sync.WaitGroup contains sync.noCopy

type noCopy struct{}

type WaitGroup struct {
 noCopy noCopy

 // 64-bit value: high 32 bits are counter, low 32 bits are waiter count.
 // 64-bit atomic operations require 64-bit alignment, but 32-bit
 // compilers only guarantee that 64-bit fields are 32-bit aligned.
 // For this reason on 32 bit architectures we need to check in state()
 // if state1 is aligned or not, and dynamically "swap" the field order if
 // needed.
 state1 uint64
 state2 uint32
}

noCopy is very simple: an empty struct. It has zero size and takes no space (as long as it is not the last field of a struct; otherwise there is still an 8-byte space overhead).

Go AES Encryption and Decryption in Three Modes (CBC/ECB/CFB)

Wed, 22 Dec 2021 17:40:52 +0800

How Go Reads a 16GB File in 25 Seconds

Mon, 22 Nov 2021 11:07:17 +0800

After opening the file, we have two options:

Read the file line by line, which reduces memory pressure but takes more time.

Read the entire file into memory and process it, which uses more memory but significantly reduces time.

Because the file is too large (16 GB), we cannot load the entire file into memory. But the first option is also not feasible for us because we want to process the file within seconds.

Bind JSON with jsoniter in Gin

Mon, 22 Nov 2021 10:27:39 +0800

Bind JSON with jsoniter in Gin

simple

go build -tags=jsoniter ./...

custom

implement `BindingBody` interface

// github.com/gin-gonic/gin@v1.6.3/binding/binding.go:36
// Binding describes the interface which needs to be implemented for binding the
// data present in the request such as JSON request body, query parameters or
// the form POST.
type Binding interface {
 Name() string
 Bind(*http.Request, interface{}) error
}
// BindingBody adds BindBody method to Binding. BindBody is similar with Bind,
// but it reads the body from supplied bytes instead of req.Body.
type BindingBody interface {
 Binding
 BindBody([]byte, interface{}) error
}

package custom
import (
 "bytes"
 "fmt"
 "io"
 "net/http"
 jsoniter "github.com/json-iterator/go"
 "github.com/gin-gonic/gin/binding"
)
// BindingJSON replaces Gin's default binding and supports richer JSON features
var BindingJSON = jsonBinding{}
// You can customize jsoniter config or add plugins
var json = jsoniter.ConfigCompatibleWithStandardLibrary
type jsonBinding struct{}
func (jsonBinding) Name() string {
 return "json"
}
func (jsonBinding) Bind(req *http.Request, obj interface{}) error {
 if req == nil || req.Body == nil {
 return fmt.Errorf("invalid request")
 }
 return decodeJSON(req.Body, obj)
}
func (jsonBinding) BindBody(body []byte, obj interface{}) error {
 return decodeJSON(bytes.NewReader(body), obj)
}
func decodeJSON(r io.Reader, obj interface{}) error {
 decoder := json.NewDecoder(r)
 if binding.EnableDecoderUseNumber {
 decoder.UseNumber()
 }
 if binding.EnableDecoderDisallowUnknownFields {
 decoder.DisallowUnknownFields()
 }
 if err := decoder.Decode(obj); err != nil {
 return err
 }
 return validate(obj)
}
func validate(obj interface{}) error {
 if binding.Validator == nil {
 return nil
 }
 return binding.Validator.ValidateStruct(obj)
}

// Replace binding.JSON with the custom one
ctx.ShouldBindWith(ms, binding.JSON)
ctx.ShouldBindBodyWith(ms, binding.JSON)

Golang os/exec usage (notes)

Fri, 12 Nov 2021 11:34:30 +0800

Golang os/exec usage (notes)

Add extra environment variables before running a command

cmd := exec.Command("programToExecute")
additionalEnv := "FOO=bar"
newEnv := append(os.Environ(), additionalEnv)
cmd.Env = newEnv
out, err := cmd.CombinedOutput()
if err != nil {
 log.Fatalf("cmd.Run() failed with %s\n", err)
}
fmt.Printf("%s", out)

Use os/exec with pipes or bash expressions on Linux

rcmd := `iw dev | awk '$1=="Interface"{print $2}'`
cmd := exec.Command("bash", "-c", rcmd)
out, err := cmd.CombinedOutput()
if err != nil {
 log.Println(err.Error())
}
log.Println(string(out))

Use os/exec with batch expressions on Windows

cmd := exec.Command("cmd", "/c", "ffmpeg -i myfile.mp4 myfile.mp3 && del myfile.mp4")
out, err := cmd.CombinedOutput()
if err != nil {
 log.Println(err.Error())
}
log.Println(string(out))

Fix local language issues in Windows CMD with os/exec

cmd := exec.Command("cmd", "/c", "chcp 65001 && netsh WLAN show drivers")
out, err := cmd.CombinedOutput()
if err != nil {
 log.Println(err.Error())
}
log.Println(string(out))

When using this method, remember to remove the first line of output (the chcp response) before further processing.

Golang benchmarks

Thu, 02 Sep 2021 12:50:33 +0800

Golang benchmarks

Basics

Benchmarks are used for performance testing. Functions should import the testing package and define functions that start with Benchmark. The parameter type is testing.B, and the target function is called repeatedly inside the benchmark loop.

➜ go test -bench=. -run=none
goos: darwin
goarch: amd64
pkg: pkg06
cpu: Intel(R) Core(TM) i7-8850H CPU @ 2.60GHz
BenchmarkFib-12 250 4682682 ns/op
PASS
ok pkg06 1.875s
➜ go test -bench=. -benchmem -run=none
goos: darwin
goarch: amd64
pkg: pkg06
cpu: Intel(R) Core(TM) i7-8850H CPU @ 2.60GHz
BenchmarkFib-12 249 4686452 ns/op 0 B/op 0 allocs/op
PASS
ok pkg06 1.854s

How bench works

The benchmark function keeps running until b.N is no longer valid; it is the number of iterations.
b.N starts at 1. If the benchmark completes within 1 second (default), b.N increases and the benchmark runs again.
b.N increases in the sequence 1,2,5,10,20,50,... and the benchmark reruns.
The result above means it ran 250 times in 1 second, with each run taking 4682682 ns.
The -12 suffix relates to GOMAXPROCS. The default value is the number of CPUs visible to the Go process at startup. You can change it with -cpu, and pass multiple values to run multiple benchmarks.

Run with multiple CPU counts

➜ go test -bench=. -cpu=1,2,4 -benchmem -run=none
goos: darwin
goarch: amd64
pkg: pkg06
cpu: Intel(R) Core(TM) i7-8850H CPU @ 2.60GHz
BenchmarkFib 244 4694667 ns/op 0 B/op 0 allocs/op
BenchmarkFib-2 255 4721201 ns/op 0 B/op 0 allocs/op
BenchmarkFib-4 256 4756392 ns/op 0 B/op 0 allocs/op
PASS
ok pkg06 5.826s

Run benchmarks multiple times with count

Due to CPU throttling, memory locality, background work, GC activity, etc., a single run may be noisy. It is common to run benchmarks multiple times.

[Translated] Go inline strategy and limitations

Sat, 14 Aug 2021 20:33:41 +0800

package main

import "fmt"

func main() {
 n := []float32{120.4, -46.7, 32.50, 34.65, -67.45}
 fmt.Printf("The total is %.02f\n", sum(n))
}

func sum(s []float32) float32 {
 var t float32
 for _, v := range s {
 if t < 0 {
 t = add(t, v)
 } else {
 t = sub(t, v)
 }
 }
 return t
}

func add(a, b float32) float32 { return a + b }
func sub(a, b float32) float32 { return a - b }

Run with -gflags="-m" to show which functions are inlined.

Go Modules workflow for private Git repositories

Fri, 23 Jul 2021 09:40:39 +0800

Go Modules workflow for private Git repositories

go env -w GOPRIVATE=github.com/appleboy
1. go env -w GONOPROXY=github.com/appleboy
2. go env -w GONOSUMDB=github.com/appleboy
git config --global url."https://$USERNAME:$ACCESS_TOKEN@github.com".insteadOf "https://github.com"
1. git config --global url.ssh://git@your.private.git/.insteadOf https://your.private.git/

Gin documentation (Chinese)

Thu, 22 Jul 2021 15:55:48 +0800

Customize route log format

default

[GIN-debug] POST /foo --> main.main.func1 (3 handlers)
[GIN-debug] GET /bar --> main.main.func2 (3 handlers)
[GIN-debug] GET /status --> main.main.func3 (3 handlers)

import (
 "log"
 "net/http"

 "github.com/gin-gonic/gin"
)

func main() {
 r := gin.Default()
 gin.DebugPrintRouteFunc = func(httpMethod, absolutePath, handlerName string, nuHandlers int) {
 log.Printf("endpoint %v %v %v %v\n", httpMethod, absolutePath, handlerName, nuHandlers)
 }

 r.POST("/foo", func(c *gin.Context) {
 c.JSON(http.StatusOK, "foo")
 })

 r.GET("/bar", func(c *gin.Context) {
 c.JSON(http.StatusOK, "bar")
 })

 r.GET("/status", func(c *gin.Context) {
 c.JSON(http.StatusOK, "ok")
 })

 // Listen and Server in http://0.0.0.0:8080
 r.Run()
}

Run multiple services with Gin

package main

import (
 "log"
 "net/http"
 "time"

 "github.com/gin-gonic/gin"
 "golang.org/x/sync/errgroup"
)

var (
 g errgroup.Group
)

func router01() http.Handler {
 e := gin.New()
 e.Use(gin.Recovery())
 e.GET("/", func(c *gin.Context) {
 c.JSON(
 http.StatusOK,
 gin.H{
 "code": http.StatusOK,
 "error": "Welcome server 01",
 },
 )
 })

 return e
}

func router02() http.Handler {
 e := gin.New()
 e.Use(gin.Recovery())
 e.GET("/", func(c *gin.Context) {
 c.JSON(
 http.StatusOK,
 gin.H{
 "code": http.StatusOK,
 "error": "Welcome server 02",
 },
 )
 })

 return e
}

func main() {
 server01 := &http.Server{
 Addr: ":8080",
 Handler: router01(),
 ReadTimeout: 5 * time.Second,
 WriteTimeout: 10 * time.Second,
 }

 server02 := &http.Server{
 Addr: ":8081",
 Handler: router02(),
 ReadTimeout: 5 * time.Second,
 WriteTimeout: 10 * time.Second,
 }

 g.Go(func() error {
 return server01.ListenAndServe()
 })

 g.Go(func() error {
 return server02.ListenAndServe()
 })

 if err := g.Wait(); err != nil {
 log.Fatal(err)
 }
}

XML, JSON, YAML, and ProtoBuf rendering (output formats)

SecureJSON

SecureJSON prevents JSON hijacking. If the response is an array, it will prefix the output with “while(1)” by default.

High Performance Go Workshop

Wed, 28 Apr 2021 16:37:05 +0800

High Performance Go Workshop

Golang Service Exceeded File Handle Limit (too many open files)

Mon, 16 Dec 2019 10:14:33 +0800

Golang Service Exceeded File Handle Limit (too many open files)

Check system config: ulimit -a | grep open. The system config is normal.
Check the service file limit: cat /proc/40636/limits. The service did not inherit the system settings and is still limited to 1024.
Check open file count: lsof -p 40636 | wc -l. It exceeds the limit, so the error occurs.
Check which connections are open: lsof -p 40636 > openfiles.log. Many HTTP connections were left open to the alerting service. The root cause is that the app failed to parse config, sent errors to the alerting service, and did not close connections, leading to too many open files. Why it did not inherit system limits needed more investigation.