Go on Ricky

Articles

Fri, 13 Mar 2026 18:30:42 +0800

It’s 2026, Just Use Postgres

The “Use the Right Tool” Trap

You’ve heard the advice: “Use the right tool for the right job.”

Sounds wise. So you end up with:

Elasticsearch for search
Pinecone for vectors
Redis for caching
MongoDB for documents
Kafka for queues
InfluxDB for time-series
PostgreSQL for… the stuff that’s left

Here’s what most people don’t realize: Postgres extensions use the same or better algorithms as specialized databases (in many cases).

Go Tool Trace

Tue, 27 Jan 2026 09:54:42 +0800

trace 是「事件時間軸」，不是「取樣統計」

go tool trace：用來看 runtime trace（執行追蹤），本質是「時間序列事件」：
- goroutine 的建立 / runnable / running / waiting / syscall
- scheduler（G/M/P）相關事件、排程延遲
- blocking（net / sync / syscall）時間分佈
- GC 事件、STW、heap 變化（在 trace viewer 會看到）
用 pprof 是找「熱點」：誰吃 CPU / 誰 alloc 多
用 trace 是找「延遲原因」：為什麼卡（排程？鎖？網路？syscall？GC？）

直覺判斷：

為什麼它正在執行？

Tue, 30 Dec 2025 11:33:23 +0800

為什麼它正在執行？

Mosdns-X

Sun, 09 Nov 2025 20:32:00 +0800

安裝

bash <(curl -sL https://raw.githubusercontent.com/lidebyte/bashshell/refs/heads/main/mosdns-x-manager.sh)

設定

sudo tee /etc/mosdns-x/config.yaml > /dev/null <<'EOF'
# mosdns-x 并发查询（无分流）配置

log:
 level: info
 file: /var/log/mosdns-x/mosdns-x.log

plugins:
 # 缓存插件
 - tag: cache
 type: cache
 args:
 size: 1024
 lazy_cache_ttl: 1800

 # 并发上游：取最先返回的可用答案
 - tag: forward_all
 type: fast_forward
 args:
 upstream:
 # 阿里
 - addr: "udp://223.5.5.5"
 - addr: "tls://dns.alidns.com"

 # DNSPod / doh.pub
 - addr: "udp://119.29.29.29"
 - addr: "tls://dot.pub"

 # Cloudflare
 - addr: "udp://1.1.1.1"
 - addr: "tls://cloudflare-dns.com"

 # Google
 - addr: "udp://8.8.8.8"
 - addr: "tls://dns.google"

 # 主流水线：小缓存 → 并发优选
 - tag: main
 type: sequence
 args:
 exec:
 - cache
 - forward_all

# 监听（双栈 UDP/TCP 53）
servers:
 - exec: main
 listeners:
 - addr: :53
 protocol: udp
 - addr: :53
 protocol: tcp
EOF

systemd

sudo tee /etc/systemd/system/mosdns.service > /dev/null <<'EOF'
[Unit]
Description=Mosdns-X DNS Accelerator
After=network.target

[Service]
Type=simple
User=root
Group=root
ExecStart=/usr/local/bin/mosdns-x start --as-service -d /usr/local/bin -c /etc/mosdns-x/config.yaml
Restart=always
RestartSec=5
StandardOutput=journal
StandardError=journal
SyslogIdentifier=mosdns

[Install]
WantedBy=multi-user.target
EOF

sudo systemctl daemon-reload
sudo systemctl enable --now mosdns

# 备份系统 DNS
sudo cp -n /etc/resolv.conf /etc/resolv.conf.mosdns-backup

# 改为使用本地 Mosdns-X
echo -e "nameserver 127.0.0.1\noptions edns0" | sudo tee /etc/resolv.conf

# 若 53 端口被 systemd-resolved 占用，可禁用它
sudo systemctl disable --now systemd-resolved 2>/dev/null || true

# 如果想顺便加锁（防止被 DHCP 修改），加上 chattr 一起执行：
echo -e "nameserver 127.0.0.1\n" > /etc/resolv.conf && chattr +i /etc/resolv.conf

# 查看进程状态
sudo systemctl status mosdns --no-pager

# 测试解析速度（第二次命中缓存更快）
dig +stats www.google.com
dig +stats www.baidu.com

# 查看实时日志
tail -f /var/log/mosdns-x/mosdns-x.log

go-synctest

Fri, 07 Nov 2025 14:06:00 +0800

go-synctest

func TestAfterFunc(t *testing.T) {
 synctest.Test(t, func(*testing.T) {
 ctx, cancel := context.WithCancel(context.Background())

 called := false
 context.AfterFunc(ctx, func() { called = true })

 synctest.Wait() // 等到所有 goroutine 都卡住
 if called {
 t.Fatal("AfterFunc 在 cancel 前就被呼叫")
 }

 cancel()

 synctest.Wait() // 再等一次
 if !called {
 t.Fatal("AfterFunc 沒有在 cancel 後被呼叫")
 }
 })
}

Go实战指南：使用 go-redis 执行 Lua 脚本

Tue, 22 Jul 2025 16:56:00 +0800

Go 实战指南：使用 go-redis 执行 Lua 脚本

Redis command Eval
Redis package redis.NewScript, script.Run

Go Protobuf：新的 Opaque API

Mon, 07 Apr 2025 13:53:00 +0800

[译]Go Protobuf：新的 Opaque API

protoc --proto_path=. \
 --go_out=./ \
+ --go_opt=default_api_level=API_OPAQUE \

GitLab CI 可以自動檢測程式碼品質！SonarQube 程式碼品質檢測怎麼做？

Thu, 06 Feb 2025 12:01:22 +0800

供應鏈攻擊

Fri, 13 Sep 2024 09:41:00 +0800

提高警惕，供應鏈攻擊就在你我身邊

github.com/siruspen/logrus 假
github.com/sirupsen/logrus 真

Go 文章

Tue, 20 Aug 2024 08:55:00 +0800

學會 gin 參數校驗之 validator 函式庫，看這一篇就夠了

字串約束

excludesall：不包含參數中任意的 UNICODE 字元，例如 excludesall=ab

excludesrune：不包含參數表示的 rune 字元，excludesrune=asong

startswith：以參數子字串為前綴，例如 startswith=hi

endswith：以參數子字串為後綴，例如 endswith=bye。

contains=：包含參數子字串，例如 contains=email

containsany：包含參數中任意的 UNICODE 字元，例如 containsany=ab

containsrune：包含參數表示的 rune 字元，例如 containsrune=asong

excludes：不包含參數子字串，例如 excludes=email

範圍約束

範圍約束的欄位型別分為三種：

對於數值，我們可以約束其值對於切片、陣列和 map，我們可以約束其長度對於字串，我們可以約束其長度

常用 tag 介紹：

ne：不等於參數值，例如 ne=5 gt：大於參數值，例如 gt=5 gte：大於等於參數值，例如 gte=50 lt：小於參數值，例如 lt=50 lte：小於等於參數值，例如 lte=50 oneof：只能是列舉出的值其中之一，這些值必須是數值或字串，以空格分隔；如果字串中有空格，請用單引號包起來，例如 oneof=male female。 eq：等於參數值，注意與 len 不同。對於字串，eq 約束字串本身的值，而 len 約束字串長度。例如 eq=10 len：等於參數值，例如 len=10 max：小於等於參數值，例如 max=10 min：大於等於參數值，例如 min=10

欄位約束

eqfield：定義欄位間相等約束，用於約束同一結構體中的欄位。例如：eqfield=Password eqcsfield：約束同一結構體中欄位等於另一個欄位（相對），確認密碼時可以使用，例如：eqcsfield=ConfirmPassword nefield：用來約束兩個欄位是否不同，確認兩種顏色是否一致時可以使用，例如：nefield=Color1 necsfield：約束兩個欄位是否不同（相對）

常用約束

unique：指定唯一性約束，不同型別處理不同：

黑帽 Go

Thu, 01 Aug 2024 15:10:22 +0800

以 Go 為例，圖解 Data Alignment 如何影響記憶體使用量

Mon, 08 Jul 2024 17:52:37 +0800

以 Go 為例，圖解 Data Alignment 如何影響記憶體使用量

Go 的十億列挑戰：從 1 分 45 秒到 3.4 秒的九種解法

Mon, 08 Jul 2024 09:50:10 +0800

Baseline

$ time cat measurements.txt >/dev/null
0m1.052s

$ time wc measurements.txt
 1000000000 1179173106 13795293380 measurements.txt
0m55.710s

$ time gawk -b -f 1brc.awk measurements.txt >measurements.out
7m35.567s

BEGIN {
 FS = ";"
}

{
 if (counts[$1]++) {
 mins[$1] = $2 < mins[$1] ? $2 : mins[$1]
 maxs[$1] = $2 > maxs[$1] ? $2 : maxs[$1]
 } else {
 mins[$1] = maxs[$1] = $2 # new entry
 }
 sums[$1] += $2
}

END {
 printf "{"
 n = asorti(mins, sorted)
 for (i = 1; i <= n; i++) {
 station = sorted[i]
 min = mins[station]
 max = maxs[station]
 mean = sums[station] / counts[station]
 printf "%s=%.1f/%.1f/%.1f", station, min, mean, max
 if (i < n) {
 printf ", "
 }
 }
 printf "}\n"
}

Solution 1: simple and idiomatic Go

// r1: simple, idiomatic Go using bufio.Scanner and strconv.ParseFloat
//
// ~1.004s for 10M rows

package main

import (
 "bufio"
 "fmt"
 "io"
 "os"
 "sort"
 "strconv"
 "strings"
)

func r1(inputPath string, output io.Writer) error {
 type stats struct {
 min, max, sum float64
 count int64
 }

 f, err := os.Open(inputPath)
 if err != nil {
 return err
 }
 defer f.Close()

 stationStats := make(map[string]stats)

 scanner := bufio.NewScanner(f)
 for scanner.Scan() {
 line := scanner.Text()
 station, tempStr, hasSemi := strings.Cut(line, ";")
 if !hasSemi {
 continue
 }

 temp, err := strconv.ParseFloat(tempStr, 64)
 if err != nil {
 return err
 }

 s, ok := stationStats[station]
 if !ok {
 s.min = temp
 s.max = temp
 s.sum = temp
 s.count = 1
 } else {
 s.min = min(s.min, temp)
 s.max = max(s.max, temp)
 s.sum += temp
 s.count++
 }
 stationStats[station] = s
 }

 stations := make([]string, 0, len(stationStats))
 for station := range stationStats {
 stations = append(stations, station)
 }
 sort.Strings(stations)

 fmt.Fprint(output, "{")
 for i, station := range stations {
 if i > 0 {
 fmt.Fprint(output, ", ")
 }
 s := stationStats[station]
 mean := s.sum / float64(s.count)
 fmt.Fprintf(output, "%s=%.1f/%.1f/%.1f", station, s.min, mean, s.max)
 }
 fmt.Fprint(output, "}\n")
 return nil
}

Solution 2: map with pointer values

// r2: use stats pointer as map value to avoid double hashing
//
// ~921ms for 10M rows (1.09x as fast as r1)

package main

import (
 "bufio"
 "fmt"
 "io"
 "os"
 "sort"
 "strconv"
 "strings"
)

func r2(inputPath string, output io.Writer) error {
 type stats struct {
 min, max, sum float64
 count int64
 }

 f, err := os.Open(inputPath)
 if err != nil {
 return err
 }
 defer f.Close()

 stationStats := make(map[string]*stats)

 scanner := bufio.NewScanner(f)
 for scanner.Scan() {
 line := scanner.Text()
 station, tempStr, hasSemi := strings.Cut(line, ";")
 if !hasSemi {
 continue
 }

 temp, err := strconv.ParseFloat(tempStr, 64)
 if err != nil {
 return err
 }

 s := stationStats[station]
 if s == nil {
 stationStats[station] = &stats{
 min: temp,
 max: temp,
 sum: temp,
 count: 1,
 }
 } else {
 s.min = min(s.min, temp)
 s.max = max(s.max, temp)
 s.sum += temp
 s.count++
 }
 }

 stations := make([]string, 0, len(stationStats))
 for station := range stationStats {
 stations = append(stations, station)
 }
 sort.Strings(stations)

 fmt.Fprint(output, "{")
 for i, station := range stations {
 if i > 0 {
 fmt.Fprint(output, ", ")
 }
 s := stationStats[station]
 mean := s.sum / float64(s.count)
 fmt.Fprintf(output, "%s=%.1f/%.1f/%.1f", station, s.min, mean, s.max)
 }
 fmt.Fprint(output, "}\n")
 return nil
}

Solution 3: avoid strconv.ParseFloat

// r3: parse temperatures manually instead of using strconv.ParseFloat
//
// ~517ms for 10M rows (1.94x as fast as r1)

package main

import (
 "bufio"
 "bytes"
 "fmt"
 "io"
 "os"
 "sort"
)

func r3(inputPath string, output io.Writer) error {
 type stats struct {
 min, max, sum float64
 count int64
 }

 f, err := os.Open(inputPath)
 if err != nil {
 return err
 }
 defer f.Close()

 stationStats := make(map[string]*stats)

 scanner := bufio.NewScanner(f)
 for scanner.Scan() {
 line := scanner.Bytes()
 station, tempBytes, hasSemi := bytes.Cut(line, []byte(";"))
 if !hasSemi {
 continue
 }

 negative := false
 index := 0
 if tempBytes[index] == '-' {
 index++
 negative = true
 }
 temp := float64(tempBytes[index] - '0')
 index++
 if tempBytes[index] != '.' {
 temp = temp*10 + float64(tempBytes[index]-'0')
 index++
 }
 index++ // skip '.'
 temp += float64(tempBytes[index]-'0') / 10
 if negative {
 temp = -temp
 }

 s := stationStats[string(station)]
 if s == nil {
 stationStats[string(station)] = &stats{
 min: temp,
 max: temp,
 sum: temp,
 count: 1,
 }
 } else {
 s.min = min(s.min, temp)
 s.max = max(s.max, temp)
 s.sum += temp
 s.count++
 }
 }

 stations := make([]string, 0, len(stationStats))
 for station := range stationStats {
 stations = append(stations, station)
 }
 sort.Strings(stations)

 fmt.Fprint(output, "{")
 for i, station := range stations {
 if i > 0 {
 fmt.Fprint(output, ", ")
 }
 s := stationStats[station]
 mean := s.sum / float64(s.count)
 fmt.Fprintf(output, "%s=%.1f/%.1f/%.1f", station, s.min, mean, s.max)
 }
 fmt.Fprint(output, "}\n")
 return nil
}

Solution 4: fixed point integers

// r4: use fixed point int32s (*10) instead of float64s
//
// ~491ms for 10M rows (2.04x as fast as r1)

package main

import (
 "bufio"
 "bytes"
 "fmt"
 "io"
 "os"
 "sort"
)

func r4(inputPath string, output io.Writer) error {
 type stats struct {
 min, max, count int32
 sum int64
 }

 f, err := os.Open(inputPath)
 if err != nil {
 return err
 }
 defer f.Close()

 stationStats := make(map[string]*stats)

 scanner := bufio.NewScanner(f)
 for scanner.Scan() {
 line := scanner.Bytes()
 station, tempBytes, hasSemi := bytes.Cut(line, []byte(";"))
 if !hasSemi {
 continue
 }

 negative := false
 index := 0
 if tempBytes[index] == '-' {
 index++
 negative = true
 }
 temp := int32(tempBytes[index] - '0')
 index++
 if tempBytes[index] != '.' {
 temp = temp*10 + int32(tempBytes[index]-'0')
 index++
 }
 index++ // skip '.'
 temp = temp*10 + int32(tempBytes[index]-'0')
 if negative {
 temp = -temp
 }

 s := stationStats[string(station)]
 if s == nil {
 stationStats[string(station)] = &stats{
 min: temp,
 max: temp,
 sum: int64(temp),
 count: 1,
 }
 } else {
 s.min = min(s.min, temp)
 s.max = max(s.max, temp)
 s.sum += int64(temp)
 s.count++
 }
 }

 stations := make([]string, 0, len(stationStats))
 for station := range stationStats {
 stations = append(stations, station)
 }
 sort.Strings(stations)

 fmt.Fprint(output, "{")
 for i, station := range stations {
 if i > 0 {
 fmt.Fprint(output, ", ")
 }
 s := stationStats[station]
 mean := float64(s.sum) / float64(s.count) / 10
 fmt.Fprintf(output, "%s=%.1f/%.1f/%.1f", station, float64(s.min)/10, mean, float64(s.max)/10)
 }
 fmt.Fprint(output, "}\n")
 return nil
}

Solution 5: avoid bytes.Cut

// r5: avoid bytes.Cut
//
// ~442ms for 10M rows (2.27x as fast as r1)

package main

import (
 "bufio"
 "fmt"
 "io"
 "os"
 "sort"
)

func r5(inputPath string, output io.Writer) error {
 type stats struct {
 min, max, count int32
 sum int64
 }

 f, err := os.Open(inputPath)
 if err != nil {
 return err
 }
 defer f.Close()

 stationStats := make(map[string]*stats)

 scanner := bufio.NewScanner(f)
 for scanner.Scan() {
 line := scanner.Bytes()

 end := len(line)
 tenths := int32(line[end-1] - '0')
 ones := int32(line[end-3] - '0') // line[end-2] is '.'
 var temp int32
 var semicolon int
 if line[end-4] == ';' {
 temp = ones*10 + tenths
 semicolon = end - 4
 } else if line[end-4] == '-' {
 temp = -(ones*10 + tenths)
 semicolon = end - 5
 } else {
 tens := int32(line[end-4] - '0')
 if line[end-5] == ';' {
 temp = tens*100 + ones*10 + tenths
 semicolon = end - 5
 } else { // '-'
 temp = -(tens*100 + ones*10 + tenths)
 semicolon = end - 6
 }
 }

 station := line[:semicolon]
 s := stationStats[string(station)]
 if s == nil {
 stationStats[string(station)] = &stats{
 min: temp,
 max: temp,
 sum: int64(temp),
 count: 1,
 }
 } else {
 s.min = min(s.min, temp)
 s.max = max(s.max, temp)
 s.sum += int64(temp)
 s.count++
 }
 }

 stations := make([]string, 0, len(stationStats))
 for station := range stationStats {
 stations = append(stations, station)
 }
 sort.Strings(stations)

 fmt.Fprint(output, "{")
 for i, station := range stations {
 if i > 0 {
 fmt.Fprint(output, ", ")
 }
 s := stationStats[station]
 mean := float64(s.sum) / float64(s.count) / 10
 fmt.Fprintf(output, "%s=%.1f/%.1f/%.1f", station, float64(s.min)/10, mean, float64(s.max)/10)
 }
 fmt.Fprint(output, "}\n")
 return nil
}

Solution 6: avoid bufio.Scanner

// r6: don't use bufio.Scanner to avoid scanning some bytes twice
//
// ~399ms for 10M rows (2.52x as fast as r1)

package main

import (
 "bytes"
 "fmt"
 "io"
 "os"
 "sort"
)

func r6(inputPath string, output io.Writer) error {
 type stats struct {
 min, max, count int32
 sum int64
 }

 f, err := os.Open(inputPath)
 if err != nil {
 return err
 }
 defer f.Close()

 stationStats := make(map[string]*stats)

 buf := make([]byte, 1024*1024)
 readStart := 0
 for {
 n, err := f.Read(buf[readStart:])
 if err != nil && err != io.EOF {
 return err
 }
 if readStart+n == 0 {
 break
 }
 chunk := buf[:readStart+n]

 newline := bytes.LastIndexByte(chunk, '\n')
 if newline < 0 {
 break
 }
 remaining := chunk[newline+1:]
 chunk = chunk[:newline+1]

 for {
 station, after, hasSemi := bytes.Cut(chunk, []byte(";"))
 if !hasSemi {
 break
 }

 index := 0
 negative := false
 if after[index] == '-' {
 negative = true
 index++
 }
 temp := int32(after[index] - '0')
 index++
 if after[index] != '.' {
 temp = temp*10 + int32(after[index]-'0')
 index++
 }
 index++ // skip '.'
 temp = temp*10 + int32(after[index]-'0')
 index += 2 // skip last digit and '\n'
 if negative {
 temp = -temp
 }
 chunk = after[index:]

 s := stationStats[string(station)]
 if s == nil {
 stationStats[string(station)] = &stats{
 min: temp,
 max: temp,
 sum: int64(temp),
 count: 1,
 }
 } else {
 s.min = min(s.min, temp)
 s.max = max(s.max, temp)
 s.sum += int64(temp)
 s.count++
 }
 }

 readStart = copy(buf, remaining)
 }

 stations := make([]string, 0, len(stationStats))
 for station := range stationStats {
 stations = append(stations, station)
 }
 sort.Strings(stations)

 fmt.Fprint(output, "{")
 for i, station := range stations {
 if i > 0 {
 fmt.Fprint(output, ", ")
 }
 s := stationStats[station]
 mean := float64(s.sum) / float64(s.count) / 10
 fmt.Fprintf(output, "%s=%.1f/%.1f/%.1f", station, float64(s.min)/10, mean, float64(s.max)/10)
 }
 fmt.Fprint(output, "}\n")
 return nil
}

Solution 7: custom hash table

// r7: use custom hash table and hash station name as we look for ';'
//
// ~234ms for 10M rows (4.29x as fast as r1)

package main

import (
 "bytes"
 "fmt"
 "io"
 "os"
 "sort"
)

func r7(inputPath string, output io.Writer) error {
 type stats struct {
 min, max, count int32
 sum int64
 }

 f, err := os.Open(inputPath)
 if err != nil {
 return err
 }
 defer f.Close()

 type item struct {
 key []byte
 stat *stats
 }
 const numBuckets = 1 << 17 // number of hash buckets (power of 2)
 items := make([]item, numBuckets) // hash buckets, linearly probed
 size := 0 // number of active items in items slice

 buf := make([]byte, 1024*1024)
 readStart := 0
 for {
 n, err := f.Read(buf[readStart:])
 if err != nil && err != io.EOF {
 return err
 }
 if readStart+n == 0 {
 break
 }
 chunk := buf[:readStart+n]

 newline := bytes.LastIndexByte(chunk, '\n')
 if newline < 0 {
 break
 }
 remaining := chunk[newline+1:]
 chunk = chunk[:newline+1]

 for {
 const (
 // FNV-1 64-bit constants from hash/fnv.
 offset64 = 14695981039346656037
 prime64 = 1099511628211
 )

 var station, after []byte
 hash := uint64(offset64)
 i := 0
 for ; i < len(chunk); i++ {
 c := chunk[i]
 if c == ';' {
 station = chunk[:i]
 after = chunk[i+1:]
 break
 }
 hash ^= uint64(c) // FNV-1a is XOR then *
 hash *= prime64
 }
 if i == len(chunk) {
 break
 }

 index := 0
 negative := false
 if after[index] == '-' {
 negative = true
 index++
 }
 temp := int32(after[index] - '0')
 index++
 if after[index] != '.' {
 temp = temp*10 + int32(after[index]-'0')
 index++
 }
 index++ // skip '.'
 temp = temp*10 + int32(after[index]-'0')
 index += 2 // skip last digit and '\n'
 if negative {
 temp = -temp
 }
 chunk = after[index:]

 hashIndex := int(hash & uint64(numBuckets-1))
 for {
 if items[hashIndex].key == nil {
 // Found empty slot, add new item (copying key).
 key := make([]byte, len(station))
 copy(key, station)
 items[hashIndex] = item{
 key: key,
 stat: &stats{
 min: temp,
 max: temp,
 sum: int64(temp),
 count: 1,
 },
 }
 size++
 if size > numBuckets/2 {
 panic("too many items in hash table")
 }
 break
 }
 if bytes.Equal(items[hashIndex].key, station) {
 // Found matching slot, add to existing stats.
 s := items[hashIndex].stat
 s.min = min(s.min, temp)
 s.max = max(s.max, temp)
 s.sum += int64(temp)
 s.count++
 break
 }
 // Slot already holds another key, try next slot (linear probe).
 hashIndex++
 if hashIndex >= numBuckets {
 hashIndex = 0
 }
 }
 }

 readStart = copy(buf, remaining)
 }

 stationItems := make([]item, 0, size)
 for _, item := range items {
 if item.key == nil {
 continue
 }
 stationItems = append(stationItems, item)
 }
 sort.Slice(stationItems, func(i, j int) bool {
 return string(stationItems[i].key) < string(stationItems[j].key)
 })

 fmt.Fprint(output, "{")
 for i, item := range stationItems {
 if i > 0 {
 fmt.Fprint(output, ", ")
 }
 s := item.stat
 mean := float64(s.sum) / float64(s.count) / 10
 fmt.Fprintf(output, "%s=%.1f/%.1f/%.1f", item.key, float64(s.min)/10, mean, float64(s.max)/10)
 }
 fmt.Fprint(output, "}\n")
 return nil
}

Solution 8: process chunks in parallel

// r8: add some parallelism (but back to non-optimized r1 version)
//
// ~213ms for 10M rows (4.71x as fast as r1)

package main

import (
 "bufio"
 "bytes"
 "fmt"
 "io"
 "os"
 "sort"
 "strconv"
 "strings"
)

type r8Stats struct {
 min, max, sum float64
 count int64
}

func r8(inputPath string, output io.Writer) error {
 parts, err := splitFile(inputPath, maxGoroutines)
 if err != nil {
 return err
 }

 resultsCh := make(chan map[string]r8Stats)
 for _, part := range parts {
 go r8ProcessPart(inputPath, part.offset, part.size, resultsCh)
 }

 totals := make(map[string]r8Stats)
 for i := 0; i < len(parts); i++ {
 result := <-resultsCh
 for station, s := range result {
 ts, ok := totals[station]
 if !ok {
 totals[station] = r8Stats{
 min: s.min,
 max: s.max,
 sum: s.sum,
 count: s.count,
 }
 continue
 }
 ts.min = min(ts.min, s.min)
 ts.max = max(ts.max, s.max)
 ts.sum += s.sum
 ts.count += s.count
 totals[station] = ts
 }
 }

 stations := make([]string, 0, len(totals))
 for station := range totals {
 stations = append(stations, station)
 }
 sort.Strings(stations)

 fmt.Fprint(output, "{")
 for i, station := range stations {
 if i > 0 {
 fmt.Fprint(output, ", ")
 }
 s := totals[station]
 mean := s.sum / float64(s.count)
 fmt.Fprintf(output, "%s=%.1f/%.1f/%.1f", station, s.min, mean, s.max)
 }
 fmt.Fprint(output, "}\n")
 return nil
}

func r8ProcessPart(inputPath string, fileOffset, fileSize int64, resultsCh chan map[string]r8Stats) {
 file, err := os.Open(inputPath)
 if err != nil {
 panic(err)
 }
 defer file.Close()
 _, err = file.Seek(fileOffset, io.SeekStart)
 if err != nil {
 panic(err)
 }
 f := io.LimitedReader{R: file, N: fileSize}

 stationStats := make(map[string]r8Stats)

 scanner := bufio.NewScanner(&f)
 for scanner.Scan() {
 line := scanner.Text()
 station, tempStr, hasSemi := strings.Cut(line, ";")
 if !hasSemi {
 continue
 }

 temp, err := strconv.ParseFloat(tempStr, 64)
 if err != nil {
 panic(err)
 }

 s, ok := stationStats[station]
 if !ok {
 s.min = temp
 s.max = temp
 s.sum = temp
 s.count = 1
 } else {
 s.min = min(s.min, temp)
 s.max = max(s.max, temp)
 s.sum += temp
 s.count++
 }
 stationStats[station] = s
 }

 resultsCh <- stationStats
}

type part struct {
 offset, size int64
}

func splitFile(inputPath string, numParts int) ([]part, error) {
 const maxLineLength = 100

 f, err := os.Open(inputPath)
 if err != nil {
 return nil, err
 }
 st, err := f.Stat()
 if err != nil {
 return nil, err
 }
 size := st.Size()
 splitSize := size / int64(numParts)

 buf := make([]byte, maxLineLength)

 parts := make([]part, 0, numParts)
 offset := int64(0)
 for i := 0; i < numParts; i++ {
 if i == numParts-1 {
 if offset < size {
 parts = append(parts, part{offset, size - offset})
 }
 break
 }

 seekOffset := max(offset+splitSize-maxLineLength, 0)
 _, err := f.Seek(seekOffset, io.SeekStart)
 if err != nil {
 return nil, err
 }
 n, _ := io.ReadFull(f, buf)
 chunk := buf[:n]
 newline := bytes.LastIndexByte(chunk, '\n')
 if newline < 0 {
 return nil, fmt.Errorf("newline not found at offset %d", offset+splitSize-maxLineLength)
 }
 remaining := len(chunk) - newline - 1
 nextOffset := seekOffset + int64(len(chunk)) - int64(remaining)
 parts = append(parts, part{offset, nextOffset - offset})
 offset = nextOffset
 }
 return parts, nil
}

Solution 9: all optimisations plus parallelisation

// r9: all the previous optimizations plus parallel execution
//
// 43ms for 10M rows (23.3x as fast as r1)

package main

import (
 "bytes"
 "fmt"
 "io"
 "os"
 "sort"
)

type r9Stats struct {
 min, max, count int32
 sum int64
}

func r9(inputPath string, output io.Writer) error {
 parts, err := splitFile(inputPath, maxGoroutines)
 if err != nil {
 return err
 }

 resultsCh := make(chan map[string]*r9Stats)
 for _, part := range parts {
 go r9ProcessPart(inputPath, part.offset, part.size, resultsCh)
 }

 totals := make(map[string]*r9Stats)
 for i := 0; i < len(parts); i++ {
 result := <-resultsCh
 for station, s := range result {
 ts := totals[station]
 if ts == nil {
 totals[station] = s
 continue
 }
 ts.min = min(ts.min, s.min)
 ts.max = max(ts.max, s.max)
 ts.sum += s.sum
 ts.count += s.count
 }
 }

 stations := make([]string, 0, len(totals))
 for station := range totals {
 stations = append(stations, station)
 }
 sort.Strings(stations)

 fmt.Fprint(output, "{")
 for i, station := range stations {
 if i > 0 {
 fmt.Fprint(output, ", ")
 }
 s := totals[station]
 mean := float64(s.sum) / float64(s.count) / 10
 fmt.Fprintf(output, "%s=%.1f/%.1f/%.1f", station, float64(s.min)/10, mean, float64(s.max)/10)
 }
 fmt.Fprint(output, "}\n")
 return nil
}

func r9ProcessPart(inputPath string, fileOffset, fileSize int64, resultsCh chan map[string]*r9Stats) {
 file, err := os.Open(inputPath)
 if err != nil {
 panic(err)
 }
 defer file.Close()
 _, err = file.Seek(fileOffset, io.SeekStart)
 if err != nil {
 panic(err)
 }
 f := io.LimitedReader{R: file, N: fileSize}

 type item struct {
 key []byte
 stat *r9Stats
 }
 const numBuckets = 1 << 17 // number of hash buckets (power of 2)
 items := make([]item, numBuckets) // hash buckets, linearly probed
 size := 0 // number of active items in items slice

 buf := make([]byte, 1024*1024)
 readStart := 0
 for {
 n, err := f.Read(buf[readStart:])
 if err != nil && err != io.EOF {
 panic(err)
 }
 if readStart+n == 0 {
 break
 }
 chunk := buf[:readStart+n]

 newline := bytes.LastIndexByte(chunk, '\n')
 if newline < 0 {
 break
 }
 remaining := chunk[newline+1:]
 chunk = chunk[:newline+1]

 for {
 const (
 // FNV-1 64-bit constants from hash/fnv.
 offset64 = 14695981039346656037
 prime64 = 1099511628211
 )

 var station, after []byte
 hash := uint64(offset64)
 i := 0
 for ; i < len(chunk); i++ {
 c := chunk[i]
 if c == ';' {
 station = chunk[:i]
 after = chunk[i+1:]
 break
 }
 hash ^= uint64(c) // FNV-1a is XOR then *
 hash *= prime64
 }
 if i == len(chunk) {
 break
 }

 index := 0
 negative := false
 if after[index] == '-' {
 negative = true
 index++
 }
 temp := int32(after[index] - '0')
 index++
 if after[index] != '.' {
 temp = temp*10 + int32(after[index]-'0')
 index++
 }
 index++ // skip '.'
 temp = temp*10 + int32(after[index]-'0')
 index += 2 // skip last digit and '\n'
 if negative {
 temp = -temp
 }
 chunk = after[index:]

 hashIndex := int(hash & (numBuckets - 1))
 for {
 if items[hashIndex].key == nil {
 // Found empty slot, add new item (copying key).
 key := make([]byte, len(station))
 copy(key, station)
 items[hashIndex] = item{
 key: key,
 stat: &r9Stats{
 min: temp,
 max: temp,
 sum: int64(temp),
 count: 1,
 },
 }
 size++
 if size > numBuckets/2 {
 panic("too many items in hash table")
 }
 break
 }
 if bytes.Equal(items[hashIndex].key, station) {
 // Found matching slot, add to existing stats.
 s := items[hashIndex].stat
 s.min = min(s.min, temp)
 s.max = max(s.max, temp)
 s.sum += int64(temp)
 s.count++
 break
 }
 // Slot already holds another key, try next slot (linear probe).
 hashIndex++
 if hashIndex >= numBuckets {
 hashIndex = 0
 }
 }
 }

 readStart = copy(buf, remaining)
 }

 result := make(map[string]*r9Stats, size)
 for _, item := range items {
 if item.key == nil {
 continue
 }
 result[string(item.key)] = item.stat
 }
 resultsCh <- result
}

Solution 10: all the previous optimizations plus faster semicolon finding and hashing

// r10: all the previous optimizations plus faster semicolon finding and
// hashing
//
// Translated from Java by Menno Finlay-Smits Ideas with ideas taken from
// this fast Java solution:
//
// https://github.com/gunnarmorling/1brc/blob/main/src/main/java/dev/morling/onebrc/CalculateAverage_mtopolnik.java
//
// On my (Ben's) laptop I get these initial results:
//
// $ ./go-1brc -revision=1 ../1brc/data/measurements.txt >out-r1
// Processed 13156.2MB in 1m39.507011009s
// $ ./go-1brc -revision=9 ../1brc/data/measurements.txt >out-r9
// Processed 13156.2MB in 2.893693843s # 34.4x as fast as the r1 above
// $ ./go-1brc -revision=10 ../1brc/data/measurements.txt >out-r10
// Processed 13156.2MB in 2.497241029s # 39.8x as fast as the r1 above

package main

import (
 "bytes"
 "encoding/binary"
 "fmt"
 "io"
 "math/bits"
 "os"
 "sort"
)

const BroadcastSemicolon = 0x3B3B3B3B3B3B3B3B
const Broadcast0x01 = 0x0101010101010101
const Broadcast0x80 = 0x8080808080808080

type r10Stats struct {
 min, max, count int32
 sum int64
}

func r10(inputPath string, output io.Writer) error {
 parts, err := splitFile(inputPath, maxGoroutines)
 if err != nil {
 return err
 }

 resultsCh := make(chan map[string]*r10Stats)
 for _, part := range parts {
 go r10ProcessPart(inputPath, part.offset, part.size, resultsCh)
 }

 totals := make(map[string]*r10Stats)
 for i := 0; i < len(parts); i++ {
 result := <-resultsCh
 for station, s := range result {
 ts := totals[station]
 if ts == nil {
 totals[station] = s
 continue
 }
 ts.min = min(ts.min, s.min)
 ts.max = max(ts.max, s.max)
 ts.sum += s.sum
 ts.count += s.count
 }
 }

 stations := make([]string, 0, len(totals))
 for station := range totals {
 stations = append(stations, station)
 }
 sort.Strings(stations)

 fmt.Fprint(output, "{")
 for i, station := range stations {
 if i > 0 {
 fmt.Fprint(output, ", ")
 }
 s := totals[station]
 mean := float64(s.sum) / float64(s.count) / 10
 fmt.Fprintf(output, "%s=%.1f/%.1f/%.1f", station, float64(s.min)/10, mean, float64(s.max)/10)
 }
 fmt.Fprint(output, "}\n")

 return nil
}

func r10ProcessPart(inputPath string, fileOffset, fileSize int64, resultsCh chan map[string]*r10Stats) {
 file, err := os.Open(inputPath)
 if err != nil {
 panic(err)
 }
 defer file.Close()
 _, err = file.Seek(fileOffset, io.SeekStart)
 if err != nil {
 panic(err)
 }
 f := io.LimitedReader{R: file, N: fileSize}

 type item struct {
 key []byte
 stat *r10Stats
 }
 const numBuckets = 1 << 17 // number of hash buckets (power of 2)
 items := make([]item, numBuckets) // hash buckets, linearly probed
 size := 0 // number of active items in items slice

 buf := make([]byte, 1024*1024)
 readStart := 0
 for {
 n, err := f.Read(buf[readStart:])
 if err != nil && err != io.EOF {
 panic(err)
 }
 if readStart+n == 0 {
 break
 }
 chunk := buf[:readStart+n]

 newline := bytes.LastIndexByte(chunk, '\n')
 if newline < 0 {
 break
 }
 remaining := chunk[newline+1:]
 chunk = chunk[:newline+1]

 chunkLoop:
 for {
 var hash uint64
 var station, after []byte

 if len(chunk) < 8 {
 break chunkLoop
 }

 nameWord0 := binary.NativeEndian.Uint64(chunk)
 matchBits := semicolonMatchBits(nameWord0)
 if matchBits != 0 {
 // semicolon is in the first 8 bytes
 nameLen := calcNameLen(matchBits)
 nameWord0 = maskWord(nameWord0, matchBits)
 station = chunk[:nameLen]
 after = chunk[nameLen+1:]
 hash = calcHash(nameWord0)
 } else {
 // station name is longer so keep looking for the semicolon in
 // uint64 chunks
 nameLen := 8
 hash = calcHash(nameWord0)
 for {
 if nameLen > len(chunk)-8 {
 break chunkLoop
 }
 lastNameWord := binary.NativeEndian.Uint64(chunk[nameLen:])
 matchBits = semicolonMatchBits(lastNameWord)
 if matchBits != 0 {
 nameLen += calcNameLen(matchBits)
 station = chunk[:nameLen]
 after = chunk[nameLen+1:]
 break
 }
 nameLen += 8
 }
 }
 index := 0
 negative := false
 if after[index] == '-' {
 negative = true
 index++
 }
 temp := int32(after[index] - '0')
 index++
 if after[index] != '.' {
 temp = temp*10 + int32(after[index]-'0')
 index++
 }
 index++ // skip '.'
 temp = temp*10 + int32(after[index]-'0')
 index += 2 // skip last digit and '\n'
 if negative {
 temp = -temp
 }
 chunk = after[index:]

 hashIndex := int(hash & (numBuckets - 1))
 for {
 if items[hashIndex].key == nil {
 // Found empty slot, add new item (copying key).
 key := make([]byte, len(station))
 copy(key, station)
 items[hashIndex] = item{
 key: key,
 stat: &r10Stats{
 min: temp,
 max: temp,
 sum: int64(temp),
 count: 1,
 },
 }
 size++
 if size > numBuckets/2 {
 panic("too many items in hash table")
 }
 break
 }
 if bytes.Equal(items[hashIndex].key, station) {
 // Found matching slot, add to existing stats.
 s := items[hashIndex].stat
 s.min = min(s.min, temp)
 s.max = max(s.max, temp)
 s.sum += int64(temp)
 s.count++
 break
 }
 // Slot already holds another key, try next slot (linear probe).
 hashIndex++
 if hashIndex >= numBuckets {
 hashIndex = 0
 }
 }
 }

 readStart = copy(buf, remaining)
 }

 result := make(map[string]*r10Stats, size)
 for _, item := range items {
 if item.key == nil {
 continue
 }
 result[string(item.key)] = item.stat
 }
 resultsCh <- result
}

func calcNameLen(b uint64) int {
 return (bits.TrailingZeros64(b) >> 3)
}

func calcHash(word uint64) uint64 {
 return bits.RotateLeft64(word*0x51_7c_c1_b7_27_22_0a_95, 17)
}

func semicolonMatchBits(word uint64) uint64 {
 diff := word ^ BroadcastSemicolon
 return (diff - Broadcast0x01) & (^diff & Broadcast0x80)
}

func maskWord(word, matchBits uint64) uint64 {
 mask := matchBits ^ (matchBits - 1)
 return word & mask
}

Alexander Yastrebov

package main

import (
 "bytes"
 "fmt"
 "log"
 "math"
 "os"
 "runtime"
 "sort"
 "sync"
 "syscall"
)

type measurement struct {
 min, max, sum, count int64
}

func main() {
 if len(os.Args) != 2 {
 log.Fatalf("Missing measurements filename")
 }

 measurements := processFile(os.Args[1])

 ids := make([]string, 0, len(measurements))
 for id := range measurements {
 ids = append(ids, id)
 }
 sort.Strings(ids)

 fmt.Print("{")
 for i, id := range ids {
 if i > 0 {
 fmt.Print(", ")
 }
 m := measurements[id]
 fmt.Printf("%s=%.1f/%.1f/%.1f", id, round(float64(m.min)/10.0), round(float64(m.sum)/10.0/float64(m.count)), round(float64(m.max)/10.0))
 }
 fmt.Println("}")
}

func processFile(filename string) map[string]*measurement {
 f, err := os.Open(filename)
 if err != nil {
 log.Fatalf("Open: %v", err)
 }
 defer f.Close()

 fi, err := f.Stat()
 if err != nil {
 log.Fatalf("Stat: %v", err)
 }

 size := fi.Size()
 if size <= 0 || size != int64(int(size)) {
 log.Fatalf("Invalid file size: %d", size)
 }

 data, err := syscall.Mmap(int(f.Fd()), 0, int(size), syscall.PROT_READ, syscall.MAP_SHARED)
 if err != nil {
 log.Fatalf("Mmap: %v", err)
 }

 defer func() {
 if err := syscall.Munmap(data); err != nil {
 log.Fatalf("Munmap: %v", err)
 }
 }()

 return process(data)
}

func process(data []byte) map[string]*measurement {
 nChunks := runtime.NumCPU()

 chunkSize := len(data) / nChunks
 if chunkSize == 0 {
 chunkSize = len(data)
 }

 chunks := make([]int, 0, nChunks)
 offset := 0
 for offset < len(data) {
 offset += chunkSize
 if offset >= len(data) {
 chunks = append(chunks, len(data))
 break
 }

 nlPos := bytes.IndexByte(data[offset:], '\n')
 if nlPos == -1 {
 chunks = append(chunks, len(data))
 break
 } else {
 offset += nlPos + 1
 chunks = append(chunks, offset)
 }
 }

 var wg sync.WaitGroup
 wg.Add(len(chunks))

 results := make([]map[string]*measurement, len(chunks))
 start := 0
 for i, chunk := range chunks {
 go func(data []byte, i int) {
 results[i] = processChunk(data)
 wg.Done()
 }(data[start:chunk], i)
 start = chunk
 }
 wg.Wait()

 measurements := make(map[string]*measurement)
 for _, r := range results {
 for id, rm := range r {
 m := measurements[id]
 if m == nil {
 measurements[id] = rm
 } else {
 m.min = min(m.min, rm.min)
 m.max = max(m.max, rm.max)
 m.sum += rm.sum
 m.count += rm.count
 }
 }
 }
 return measurements
}

func processChunk(data []byte) map[string]*measurement {
 // Use fixed size linear probe lookup table
 const (
 // use power of 2 for fast modulo calculation,
 // should be larger than max number of keys which is 10_000
 entriesSize = 1 << 14

 // use FNV-1a hash
 fnv1aOffset64 = 14695981039346656037
 fnv1aPrime64 = 1099511628211
 )

 type entry struct {
 m measurement
 hash uint64
 vlen int
 value [128]byte // use power of 2 > 100 for alignment
 }
 entries := make([]entry, entriesSize)
 entriesCount := 0

 // keep short and inlinable
 getMeasurement := func(hash uint64, value []byte) *measurement {
 i := hash & uint64(entriesSize-1)
 entry := &entries[i]

 // bytes.Equal could be commented to speedup assuming no hash collisions
 for entry.vlen > 0 && !(entry.hash == hash && bytes.Equal(entry.value[:entry.vlen], value)) {
 i = (i + 1) & uint64(entriesSize-1)
 entry = &entries[i]
 }

 if entry.vlen == 0 {
 entry.hash = hash
 entry.vlen = copy(entry.value[:], value)
 entriesCount++
 }
 return &entry.m
 }

 // assume valid input
 for len(data) > 0 {

 idHash := uint64(fnv1aOffset64)
 semiPos := 0
 for i, b := range data {
 if b == ';' {
 semiPos = i
 break
 }

 // calculate FNV-1a hash
 idHash ^= uint64(b)
 idHash *= fnv1aPrime64
 }

 idData := data[:semiPos]

 data = data[semiPos+1:]

 var temp int64
 // parseNumber
 {
 negative := data[0] == '-'
 if negative {
 data = data[1:]
 }

 _ = data[3]
 if data[1] == '.' {
 // 1.2\n
 temp = int64(data[0])*10 + int64(data[2]) - '0'*(10+1)
 data = data[4:]
 // 12.3\n
 } else {
 _ = data[4]
 temp = int64(data[0])*100 + int64(data[1])*10 + int64(data[3]) - '0'*(100+10+1)
 data = data[5:]
 }

 if negative {
 temp = -temp
 }
 }

 m := getMeasurement(idHash, idData)
 if m.count == 0 {
 m.min = temp
 m.max = temp
 m.sum = temp
 m.count = 1
 } else {
 m.min = min(m.min, temp)
 m.max = max(m.max, temp)
 m.sum += temp
 m.count++
 }
 }

 result := make(map[string]*measurement, entriesCount)
 for i := range entries {
 entry := &entries[i]
 if entry.m.count > 0 {
 result[string(entry.value[:entry.vlen])] = &entry.m
 }
 }
 return result
}

func round(x float64) float64 {
 return roundJava(x*10.0) / 10.0
}

// roundJava returns the closest integer to the argument, with ties
// rounding to positive infinity, see java's Math.round
func roundJava(x float64) float64 {
 t := math.Trunc(x)
 if x < 0.0 && t-x == 0.5 {
 //return t
 } else if math.Abs(x-t) >= 0.5 {
 t += math.Copysign(1, x)
 }

 if t == 0 { // check -0
 return 0.0
 }
 return t
}

// parseNumber reads decimal number that matches "^-?[0-9]{1,2}[.][0-9]" pattern,
// e.g.: -12.3, -3.4, 5.6, 78.9 and return the value*10, i.e. -123, -34, 56, 789.
func parseNumber(data []byte) int64 {
 negative := data[0] == '-'
 if negative {
 data = data[1:]
 }

 var result int64
 switch len(data) {
 // 1.2
 case 3:
 result = int64(data[0])*10 + int64(data[2]) - '0'*(10+1)
 // 12.3
 case 4:
 result = int64(data[0])*100 + int64(data[1])*10 + int64(data[3]) - '0'*(100+10+1)
 }

 if negative {
 return -result
 }
 return result
}

https://github.com/aytechnet/1brc

François Pons

package main

import (
 "bytes"
 "flag"
 "fmt"
 "log"
 "os"
 "runtime"
 "runtime/pprof"
 "slices"
 "sync"
 "syscall"
 "sync/atomic"

 "github.com/aytechnet/decimal"
)

type (
 measurement struct {
 hash atomic.Uint64
 minT atomic.Int64
 maxT atomic.Int64
 sumT atomic.Int64
 countT atomic.Int64
 nameLen int
 nameBuf [208]byte
 }

 measurements struct {
 total atomic.Int64
 numParsers int
 results [capacity]measurement
 }

 job struct {
 maxOffset atomic.Int64
 bufLen int
 buf [bufSize]byte
 }
)

const (
 delta = 439
 capacity = 1 << 16 // must be a power of 2 for modulo calculation

 // buffer size
 bufSize = 512 * 1024 // 1Mb

 // use FNV-1a hash
 fnv1aOffset64 = 14695981039346656037
 fnv1aPrime64 = 1099511628211
)

func main() {
 var mode, filename, cpuprofile string
 var res measurements

 flag.StringVar(&mode, "mode", "default", "Which mode to use among 'mmap', 'seq' and 'default'")
 flag.StringVar(&filename, "file", "", "Measurements file to use")
 flag.StringVar(&cpuprofile, "cpuprofile", "", "Write cpuprofile to file")
 flag.IntVar(&res.numParsers, "parsers", runtime.NumCPU(), "Number of thread to use for parsing")

 flag.Parse()

 if filename == "" {
 flag.Usage()
 os.Exit(1)
 }

 if cpuprofile != "" {
 if f, err := os.Create(cpuprofile); err != nil {
 log.Fatal(err)
 } else {
 pprof.StartCPUProfile(f)
 defer pprof.StopCPUProfile()
 }
 }

 f, err := os.Open(filename)
 if err != nil {
 log.Fatalf("Open: %v", err)
 }
 defer f.Close()

 switch mode {
 case "mmap":
 res.processMmap(f)
 case "seq":
 res.processSeq(f)
 default:
 res.process(f)
 }
}

func (res *measurements) processMmap(f *os.File) {
 jobs := make([]job, res.numParsers)

 fi, err := f.Stat()
 if err != nil {
 log.Fatalf("Stat: %v", err)
 }

 size := fi.Size()
 chunkSize := size/int64(len(jobs))
 if chunkSize <= 100 {
 log.Fatalf("Invalid file size: %d", size)
 }

 data, err := syscall.Mmap(int(f.Fd()), 0, int(size), syscall.PROT_READ, syscall.MAP_SHARED)
 if err != nil {
 log.Fatalf("Mmap: %v", err)
 }

 defer func() {
 if err := syscall.Munmap(data); err != nil {
 log.Fatalf("Munmap: %v", err)
 }
 }()

 offset := chunkSize
 for i := range jobs {
 j := &jobs[i]

 if i == len(jobs)-1 {
 j.maxOffset.Store(size)
 } else {
 j.maxOffset.Store(-offset)
 }
 offset += chunkSize
 }

 var wg sync.WaitGroup

 wg.Add(len(jobs))

 offset = 0
 for i := range jobs {
 go func(i int, offset int64){
 defer wg.Done()

 j := &jobs[i]

 maxOffset := j.maxOffset.Load()
 if maxOffset < 0 {
 maxOffset = -maxOffset

 if nlPos := bytes.IndexByte(data[maxOffset:], '\n'); nlPos >= 0 {
 maxOffset += int64(nlPos+1)
 }
 }

 if offset > 0 {
 if nlPos := bytes.IndexByte(data[offset:maxOffset], '\n'); nlPos >= 0 {
 offset += int64(nlPos+1)
 }
 }

 res.processChunk(data[offset:maxOffset])
 }(i, offset)

 offset += chunkSize
 }

 wg.Wait()

 res.printResults()
}

func (res *measurements) processSeq(f *os.File) {
 jobs := make([]job, res.numParsers)
 free_jobs := make(chan *job, len(jobs))
 ready_jobs := make(chan *job, len(jobs))

 var wg sync.WaitGroup

 wg.Add(len(jobs))

 for i := range jobs {
 j := &jobs[i]

 free_jobs <- j
 go func(){
 for j := range ready_jobs {
 res.processChunk(j.buf[:j.bufLen])
 free_jobs <- j
 }
 wg.Done()
 }()
 }

 var prev_j *job

 pos := 0
 nlPos := 0
 for j := range free_jobs {
 if count, _ := f.Read(j.buf[pos:]); count > 0 {
 // finalize previous line from previous job
 if pos > 0 /* && prev_j != nil */ {
 copy(j.buf[:pos], prev_j.buf[bufSize-pos:])
 }

 // prepare next buffer
 if nlPos = bytes.LastIndexByte(j.buf[:pos+count], '\n'); nlPos < 0 {
 log.Fatalf("buffer too small for a complete line");
 } else {
 pos = pos + count - nlPos - 1
 prev_j = j
 j.bufLen = nlPos + 1
 }

 // spawn a new job on this buffer
 ready_jobs <- j
 } else {
 break
 }
 }

 close(ready_jobs)

 wg.Wait()

 close(free_jobs)

 res.printResults()
}

func (res *measurements) process(f *os.File) {
 jobs := make([]job, res.numParsers)

 fi, err := f.Stat()
 if err != nil {
 log.Fatalf("Stat: %v", err)
 }

 size := fi.Size()
 chunkSize := size/int64(len(jobs))
 if chunkSize <= 100 {
 log.Fatalf("Invalid file size: %d", size)
 }

 offset := chunkSize
 for i := range jobs {
 j := &jobs[i]

 if i == len(jobs)-1 {
 j.maxOffset.Store(size)
 } else {
 j.maxOffset.Store(-offset)
 }
 offset += chunkSize
 }

 var wg sync.WaitGroup

 wg.Add(len(jobs))

 offset = 0
 for i := range jobs {
 go func(i int, offset int64){
 defer wg.Done()

 j := &jobs[i]

 nlSkipFirst := offset > 0

 for {
 maxLen := bufSize
 maxOffset := j.maxOffset.Load()
 if maxOffset < 0 {
 maxOffset = -maxOffset
 }
 if offset + int64(maxLen) > maxOffset {
 maxLen = int(maxOffset - offset)
 }

 if count, _ := f.ReadAt(j.buf[:maxLen], offset); count > 0 {
 pos := 0
 if nlSkipFirst {
 if nlPos := bytes.IndexByte(j.buf[:maxLen], '\n'); nlPos >= 0 {
 pos = nlPos + 1
 jobs[i-1].maxOffset.Store(offset + int64(pos))
 nlSkipFirst = false
 } else {
 log.Fatalf("Unable to seek to next line at job n°%d", i)
 }
 }

 if nlPos := bytes.LastIndexByte(j.buf[pos:maxLen], '\n'); nlPos >= 0 {
 j.bufLen = pos + nlPos + 1
 offset += int64(j.bufLen)

 res.processChunk(j.buf[pos:j.bufLen])
 } else {
 log.Fatalf("Buffer too small at job n°%d", i)
 }
 } else {
 return
 }

 maxOffset = j.maxOffset.Load()
 for maxOffset < 0 {
 maxOffset = j.maxOffset.Load()
 }
 if offset >= maxOffset {
 return
 }
 }
 }(i, offset)

 offset += chunkSize
 }

 wg.Wait()

 res.printResults()
}

func (res *measurements) printResults() {
 log.Printf("Read %d entries", res.total.Load())

 ids := make([]int, 0, capacity)

 for i := range res.results {
 m := &res.results[i]

 if m.nameLen > 0 {
 ids = append(ids, i)
 }
 }

 slices.SortFunc(ids, func(a,b int) int {
 return bytes.Compare(res.results[a].nameBuf[0:res.results[a].nameLen], res.results[b].nameBuf[0:res.results[b].nameLen])
 })

 count := 0
 fmt.Print("{")
 for _, i := range ids {
 m := &res.results[i]

 var buf [128]byte // name is 100 chars max, each 3 decimals is 5 bytes max on output
 b := buf[0:0]

 if count > 0 {
 b = append(b, ',', ' ')
 }

 b = append(b, m.nameBuf[0:m.nameLen]...)
 b = append(b, '=')
 b = decimal.New(m.minT.Load(), -1).BytesToFixed(b, 1)
 b = append(b, '/')
 b = decimal.New(m.sumT.Load(), -1).Div(decimal.NewFromInt(m.countT.Load()-1)).BytesToFixed(b, 1)
 b = append(b, '/')
 b = decimal.New(m.maxT.Load(), -1).BytesToFixed(b, 1)
 count++

 fmt.Print(string(b))
 // fmt.Printf("%s=%s/%s/%s", m.nameBuf[0:m.nameLen], decimal.New(m.minT.Load(), -1).StringFixed(1), decimal.New(m.sumT.Load(), -1).Div(decimal.NewFromInt(m.countT.Load()-1)).StringFixed(1), decimal.New(m.maxT.Load(), -1).StringFixed(1))
 count++
 }
 fmt.Println("}")
}

func (res *measurements) processChunk(data []byte) {
 var total int64

 // assume valid input
 for len(data) > 0 {
 i := 0

 // compute FNV-1a hash
 idHash := uint64(fnv1aOffset64)
 for j, b := range data {
 if b == ';' {
 i = j
 break
 }

 // calculate FNV-1a hash
 idHash ^= uint64(b)
 idHash *= fnv1aPrime64
 }
 if idHash == 0 {
 idHash = uint64(len(data))
 }

 idData := data[:i]

 i++ // now i points to temperature

 var temp int64
 // parseNumber
 {
 negative := data[i] == '-'
 if negative {
 i++
 }

 temp = int64(data[i]-'0')
 i++

 if data[i] != '.' {
 temp = temp*10 + int64(data[i]-'0')
 i++
 }
 i++ // data[i] is '.'
 temp = temp*10 + int64(data[i]-'0')
 if negative {
 temp = -temp
 }

 data = data[i+2:]
 }

 // get measurement
 {
 i := idHash & uint64(capacity-1)
 entry := &res.results[i]

 for {
 if entry.hash.CompareAndSwap(0, idHash) {
 // make sure no race occurs as entry may be updated meanwhile as hash has been updated
 if entry.countT.Add(1) == 1 {
 entry.nameLen = len(idData)
 copy(entry.nameBuf[:], idData)
 entry.minT.Store(temp)
 entry.maxT.Store(temp)
 entry.sumT.Store(temp)
 entry.countT.Add(1) // unlock for update below
 } else {
 // wait for countT to be at least 2 for entry init to be complete
 for entry.countT.Load() < 2 {}

 // update existing entry
 minT := entry.minT.Load()
 for minT > temp {
 if entry.minT.CompareAndSwap(minT, temp) {
 break
 } else {
 minT = entry.minT.Load()
 }
 }
 maxT := entry.maxT.Load()
 for maxT < temp {
 if entry.maxT.CompareAndSwap(maxT, temp) {
 break
 } else {
 maxT = entry.maxT.Load()
 }
 }
 entry.sumT.Add(temp)
 entry.countT.Add(1)
 }
 break
 } else if entry.hash.Load() == idHash {
 // the entry is found and may be being updated by another thread
 // wait for countT to be at least 2 for entry init to be complete
 for entry.countT.Load() < 2 {}

 // now that name has been updated, check it is matching
 if len(idData) == entry.nameLen /* bytes.Compare(idData, entry.nameBuf[0:entry.nameLen]) == 0 */ {
 // update existing entry
 minT := entry.minT.Load()
 for minT > temp {
 if entry.minT.CompareAndSwap(minT, temp) {
 break
 } else {
 minT = entry.minT.Load()
 }
 }
 maxT := entry.maxT.Load()
 for maxT < temp {
 if entry.maxT.CompareAndSwap(maxT, temp) {
 break
 } else {
 maxT = entry.maxT.Load()
 }
 }
 entry.sumT.Add(temp)
 entry.countT.Add(1)

 break
 } else {
 // name does not match idData so jump to next entry
 i = (i + delta) & uint64(capacity-1)
 entry = &res.results[i]
 }
 } else {
 i = (i + delta) & uint64(capacity-1)
 entry = &res.results[i]
 }
 }
 }

 total++
 }

 res.total.Add(total)
}

func (entry *measurement) update(temp int64) {
 minT := entry.minT.Load()
 for minT > temp {
 if entry.minT.CompareAndSwap(minT, temp) {
 break
 } else {
 minT = entry.minT.Load()
 }
 }
 maxT := entry.maxT.Load()
 for maxT < temp {
 if entry.maxT.CompareAndSwap(maxT, temp) {
 break
 } else {
 maxT = entry.maxT.Load()
 }
 }
 entry.sumT.Add(temp)
 entry.countT.Add(1)
}

Go LeetCode

Thu, 20 Jun 2024 18:15:00 +0800

https://books.halfrost.com/leetcode/

更強大的 Go 執行追蹤能力

Tue, 07 May 2024 11:45:57 +0800

更強大的 Go 執行追蹤能力

runtime/trace

golang.org/x/exp/trace

飛行記錄（flight recording）

// 設定飛行記錄器
fr := trace.NewFlightRecorder()
fr.Start()
// 設定並啟動 HTTP 伺服器
var once sync.Once
http.HandleFunc("/my-endpoint", func(w http.ResponseWriter, r *http.Request) {
 start := time.Now()
 // 做些事情
 doWork(w, r)
 // 碰到長耗時請求就來個快照
 if time.Since(start) > 300*time.Millisecond {
 // 這裡為了簡化只做一次，實際上你可以做多次
 once.Do(func() {
 // 擷取快照
 var b bytes.Buffer
 _, err = fr.WriteTo(&b)
 if err != nil {
 log.Print(err)
 return
 }
 // 把快照寫入檔案
 if err := os.WriteFile("trace.out", b.Bytes(), 0o755); err != nil {
 log.Print(err)
 return
 }
 })
 }
})
log.Fatal(http.ListenAndServe(":8080", nil))

追蹤讀取器 API

// 開始從標準輸入讀取追蹤資料。
r, err := trace.NewReader(os.Stdin)
if err != nil {
 log.Fatal(err)
}
var blocked int
var blockedOnNetwork int
for {
 // 讀取事件
 ev, err := r.ReadEvent()
 if err == io.EOF {
 break
 } else if err != nil {
 log.Fatal(err)
 }
 // 處理它
 if ev.Kind() == trace.EventStateTransition {
 st := ev.StateTransition()
 if st.Resource.Kind == trace.ResourceGoroutine {
 id := st.Resource.Goroutine()
 from, to := st.GoroutineTransition()
 // 查找阻塞的 goroutine 並統計
 if from.Executing() && to == trace.GoWaiting {
 blocked++
 if strings.Contains(st.Reason, "network") {
 blockedOnNetwork++
 }
 }
 }
 }
}

// 印出所需數值
p := 100 * float64(blockedOnNetwork) / float64(blocked)
fmt.Printf("%2.3f%% instances of goroutines blocking were to block on the network\n", p)

Run llama3

Thu, 25 Apr 2024 10:14:00 +0800

Docker-compose

version: "3.8"

services:
 ollama:
 image: ollama/ollama:latest
 container_name: ollama
 restart: unless-stopped
 volumes:
 - ./ollama/ollama:/root/.ollama
 tty: true
 ports:
 - 11434:11434
 networks:
 - ollama-docker
 # deploy:
 # resources:
 # reservations:
 # devices:
 # - driver: nvidia
 # count: 1
 # capabilities: [gpu]
 ollama-webui:
 image: ghcr.io/ollama-webui/ollama-webui:main
 container_name: ollama-webui
 restart: unless-stopped
 volumes:
 - ./ollama/ollama-webui:/app/backend/data
 ports:
 - 8080:8080
 environment:
 - "/ollama/api=http://ollama:11434/api"
 extra_hosts:
 - host.docker.internal:host-gateway
 networks:
 - ollama-docker

networks:
 ollama-docker:

Setup

# Run docker-compose
docker-compose up -d

# Pull model(https://ollama.com/library)
docker exec -it ollama /bin/bash
ollama pull llama3

Chat with Web-UI

port defined in docker-compose.yml ollama-webui.ports

一文讀不懂的 Go 1.21 GA 的 PGO 優化——一次在 WebP Server Go 上的嘗試

Thu, 25 Apr 2024 09:22:00 +0800

一文讀不懂的 Go 1.21 GA 的 PGO 優化——一次在 WebP Server Go 上的嘗試

Go Style Decisions - Pass values

Sat, 13 Apr 2024 17:38:00 +0800

Go Style Decisions - Pass values

不要僅僅為了節省一些位元組而將指標作為函數參數傳遞。如果函數始終將其參數 x 讀取為 *x，則該參數不應該是指標。常見的實例包括傳遞指向字串的指標 (*string) 或指向介面值的指標 (*io.Reader)。在這兩種情況下，值本身都是固定大小的，可以直接傳遞。

此建議不適用於大型結構，甚至不適用於可能增加大小的小型結構。特別是，協定緩衝區訊息通常應透過指標而不是值來處理。指標類型滿足 proto.Message 介面（被 proto.Marshal 、 protocmp.Transform 等接受），且協定緩衝區訊息可能非常大，並且通常會隨著時間的推移而變得更大。

Go Generics 入門筆記

Wed, 07 Feb 2024 19:52:22 +0800

Go Generics 入門筆記

自訂約束

// Addable 只允許 int 或 float64 類型
type Addable interface {
 int | float64
}

func Add[T Addable](a, b T) T {
 return a + b
}

~ 符號的作用 ~ 符號用於表示與指定類型有相同底層類型的所有類型。當你在類型參數的約束中使用 ~ 符號時，你指定了一個類型集合，這個集合包含所有底層類型與約束中指定的類型相同的類型。

type MyInt int
type YourInt int

func PrintInt[T ~int](t T) {
 fmt.Println(t)
}

func main() {
 var a int = 5
 var b MyInt = 10
 var c YourInt = 15

 PrintInt(a)
 PrintInt(b)
 PrintInt(c)
}

Golang tips

Tue, 26 Sep 2023 22:15:00 +0800

GO 编程模式：切片，接口，时间和性能 Go 语言是一个高性能的语言，但并不是说这样我们就不用关心性能了，我们还是需要关心的。下面是一个在编程方面和性能相关的提示。
如果需要把数字转字符串，使用 strconv.Itoa() 会比 fmt.Sprintf() 要快一倍左右
尽可能地避免把 String 转成[]Byte 。这个转换会导致性能下降。
如果在 for-loop 里对某个 slice 使用 append()请先把 slice 的容量很扩充到位，这样可以避免内存重新分享以及系统自动按 2 的 N 次方幂进行扩展但又用不到，从而浪费内存。
使用 StringBuffer 或是 StringBuild 来拼接字符串，会比使用 + 或 += 性能高三到四个数量级。
尽可能的使用并发的 go routine，然后使用 sync.WaitGroup 来同步分片操作
避免在热代码中进行内存分配，这样会导致 gc 很忙。尽可能的使用 sync.Pool 来重用对象。
使用 lock-free 的操作，避免使用 mutex，尽可能使用 sync/Atomic 包。（关于无锁编程的相关话题，可参看《无锁队列实现》或《无锁 Hashmap 实现》）
使用 I/O 缓冲，I/O 是个非常非常慢的操作，使用 bufio.NewWrite() 和 bufio.NewReader() 可以带来更高的性能。
对于在 for-loop 里的固定的正则表达式，一定要使用 regexp.Compile() 编译正则表达式。性能会得升两个数量级。
如果你需要更高性能的协议，你要考虑使用 protobuf 或 msgp 而不是 JSON，因为 JSON 的序列化和反序列化里使用了反射。

Go 設計模式

Sat, 16 Sep 2023 16:08:12 +0800

channel 的 15 条规则和底层实现

Thu, 22 Jun 2023 21:25:05 +0800

channel 的 15 条规则和底层实现

操作规则

操作	nil	已关闭的 channel	未关闭有缓冲区的 channel	未关闭无缓冲区的 channel
关闭	panic	panic	成功关闭，然后可以读取缓冲区的值，读取完之后，继续读取到的是 channel 类型的默认值	成功关闭，之后读取到的是 channel 类型的默认值
接收	阻塞	不阻塞，读取到的是 channel 类型的默认值	不阻塞，正常读取值	阻塞
发送	阻塞	panic	不阻塞，正常写入值	阻塞

编译规则

操作	channel 類型	結果
接收	只写 channel	编译错误
发送	只读 channel	编译错误
关闭	只读 channel	编译错误

使用Go语言解析二进制数据踩坑总结

Fri, 20 Jan 2023 13:48:27 +0800

使用 Go 语言解析二进制数据踩坑总结

for {
 buf := make([]byte, bufSize)
 n, err := r.Read(buf)
 if err == io.EOF {
 return
 } else if err != nil {
 aerr.Store(err)
 return
 }
 ch <- buf[:n]
}

当 err 的值为 io.EOF 时直接返回了。这是最可能出问题的地方。仔细阅读函数文档发现：

Callers should always process the n > 0 bytes returned before considering the error err. Doing so correctly handles I/O errors that happen after reading some bytes and also both of the allowed EOF behaviors.

所以程序必须先处理 n > 0 的情况，然后再处理 err != nil 的情况！这跟我之前的印象不相符。因为一般来说，当 err 非空时，前面的返回数据一般都是空值。但 io.Reader 并不是这样。

Go 使用 selenium 截图

Fri, 02 Dec 2022 13:40:19 +0800

Go 使用 selenium 截图

package main

import (
 "fmt"
 log "github.com/sirupsen/logrus"
 "github.com/tebeka/selenium"
 "io/ioutil"
)

const (
 chromeDriverPath = "/usr/local/bin/chromedriver"
 port = 9515
)

func main() {
 var opts []selenium.ServiceOption
 selenium.SetDebug(false)
 service, err := selenium.NewChromeDriverService(chromeDriverPath, port, opts...)
 if err != nil {
 panic(err) // panic is used only as an example and is not otherwise recommended.
 }
 defer service.Stop()

 caps := selenium.Capabilities{"browserName": "chrome"}
 // 整页截屏
 // 获取到网页的宽高，然后把浏览器的窗口设置成这么大，然后再截图就好了！需要注意的是只有 headless 模式可以任意设置窗口大小，否则最大高度不能超过你的显示器分辨率
 // chromeCaps := chrome.Capabilities{
 // Path: "",
 // Args: []string{
 // "--headless",
 // },
 // }
 // caps.AddChrome(chromeCaps)

 wd, err := selenium.NewRemote(caps, fmt.Sprintf("http://localhost:%d/wd/hub", port))
 if err != nil {
 panic(err)
 }
 defer wd.Quit()

 // 在 selenium 中使用ExecuteScript就可以执行 JavaScript 代码，需要注意返回值是 interface，所以要类型断言成 float，然后再转成 int
 // height, _ := wd.ExecuteScript("return document.body.parentNode.scrollHeight", nil)
 // var realHeight = int(height.(float64))
 // 然后我们需要设置窗口大小
 // wd.ResizeWindow("", 1920, realHeight)

 if err := wd.Get("https://github.com/tgbot-collection/archiver"); err != nil {
 panic(err)
 }

 screenshot, err := wd.Screenshot()
 if err != nil {
 log.Errorln(err)

 }
 ioutil.WriteFile("screenshot.png", screenshot, 0644)

}

整页截屏

获取到网页的宽高，然后把浏览器的窗口设置成这么大，然后再截图就好了！需要注意的是只有 headless 模式可以任意设置窗口大小，否则最大高度不能超过你的显示器分辨率

善用 Go Fuzzing，幫助你寫出更完整的單元測試

Thu, 03 Nov 2022 16:57:29 +0800

善用 Go Fuzzing，幫助你寫出更完整的單元測試

func Pow(base uint, exponent uint) uint {
 if exponent == 0 {
 return 1
 }

 return base * Pow(base, exponent-1)
}

func FuzzPow(f *testing.F) {
 f.Fuzz(func(t *testing.T, x, y uint) {
 assert := assert.New(t)

 assert.Equal(Pow(x, 0), uint(1))
 assert.Equal(Pow(x, 1), x)
 assert.Equal(Pow(x, 2), x*x)

 if x > 0 && y > 0 {
 assert.Equal(Pow(x, y)/x, Pow(x, y-1))
 }
 })
}

go test -fuzz=Fuzz -fuzztime 20s

在做 Fuzzing Test 的時候如果跑一跑 FAIL 了，Go 會幫忙把那組 input 記在 testcase/ 裡面
看了之後會發現在 x=6、y=30 時 assert 會失敗，也就是說 pow(6, 30)/6 不會等於 pow(6, 29)。但這也太奇怪了吧？仔細實驗之後才發現是因為在計算 pow(6, 30) 的時候會發生 overflow。
因為 Go 定義的 max.MaxUint 大約是 18 * 10¹⁸，但 6²⁹ 大概是 7 * 10¹⁸。如果把 6²⁹ 再乘上 6，就會發生 overflow 得到 8 * 10¹⁸，很像繞了操場兩圈結果在跟原本差不多的位置。

var ErrOverflow = fmt.Errorf("overflow")

func Pow(base uint, exponent uint) (uint, error) {
 if exponent == 0 {
 return 1
 }

 prevResult, err := Pow(base, exponent-1)
 if math.MaxUint/base < prevResult {
 return 0, ErrOverflow
 }

 return base * prevResult, nil
}

func FuzzPow(f *testing.F) {
 f.Fuzz(func(t *testing.T, x, y uint) {
 assert := assert.New(t)
 if result, err := Pow(x, 1); err != ErrOverflow {
 assert.Equal(result, x)
 }

 if result, err := Pow(x, y); x > 0 && y > 0 && err != ErrOverflow {
 resultDivX, _ := Pow(x, y-1)
 assert.Equal(result/x, resultDivX)
 }
 })
}

Go 字串格式化

Thu, 08 Sep 2022 11:32:00 +0800

結尾的動詞（verb）決定對應參數的型別與解讀方式。

d - 十進位整數
o - 八進位整數
O - 帶有 0o 前綴的八進位整數
b - 二進位整數
x - 十六進位整數（小寫）
X - 十六進位整數（大寫）
f - 十進位浮點數（小寫）
F - 十進位浮點數（大寫）
e - 科學記號（尾數/指數，小寫）
E - 科學記號（尾數/指數，大寫）
g - %e 或 %f 的最短表示
G - %E 或 %F 的最短表示
c - 以 Unicode 碼點表示的字元
q - 帶引號的字元
U - Unicode 逸出序列
t - true 或 false 字串
s - 字串
v - 預設格式
#v - 值的 Go 語法表示
T - 值型別的 Go 語法表示
p - 指標位址
% - 雙 %% 會輸出單一 %

Go 字串格式化索引

package main

import (
 "fmt"
)

func main() {

 n1 := 2
 n2 := 3
 n3 := 4

 res := fmt.Sprintf("There are %d oranges %d apples %d plums", n1, n2, n3)
 fmt.Println(res) // There are 2 oranges 3 apples 4 plums

 res2 := fmt.Sprintf("There are %[2]d oranges %d apples %[1]d plums", n1, n2, n3)
 fmt.Println(res2) // There are 3 oranges 4 apples 2 plums
}

Go 字串格式化精度

package main

import (
 "fmt"
)

func main() {

 fmt.Printf("%0.f\n", 16.540) // 17
 fmt.Printf("%0.2f\n", 16.540) // 16.54
 fmt.Printf("%0.3f\n", 16.540) // 16.540
 fmt.Printf("%0.5f\n", 16.540) // 16.54000
}

Go 字串格式化旗標

package main

import (
 "fmt"
)

func main() {

 fmt.Printf("%+d\n", 1691) // +1691

 fmt.Printf("%#x\n", 1691) // 0x69b
 fmt.Printf("%#X\n", 1691) // 0X69B
 fmt.Printf("%#b\n", 1691) // 0b11010011011

 fmt.Printf("%10d\n", 1691) // 1691
 fmt.Printf("%-10d\n", 1691) // 1691
 fmt.Printf("%010d\n", 1691) // 0000001691
}

Go 字串格式化寬度

package main

import (
 "fmt"
)

func main() {

 w := "falcon"
 n := 122
 h := 455.67

 fmt.Printf("%s\n", w) // falcon
 fmt.Printf("%10s\n", w) // falcon

 fmt.Printf("%d\n", n). // 122
 fmt.Printf("%7d\n", n) // 122
 fmt.Printf("%07d\n", n) // 0000122

 fmt.Printf("%10f\n", h) // 455.670000
 fmt.Printf("%11f\n", h) // 455.670000
 fmt.Printf("%12f\n", h) // 455.670000
}

為你的Go應用建立輕量級Docker映象？ | IT人

Mon, 25 Jul 2022 17:33:47 +0800

為你的 Go 應用建立輕量級 Docker 映象？ | IT 人

go build

# default
$ go build -o test1 main.go
$ du -sh test1
14M test1


# 在程式編譯的時候可以加上 `-ldflags "-s -w"` 引數來優化編譯，原理是通過去除部分連結和除錯等資訊來減小編譯生成的可執行程式體積，具體引數如下：
# -a：強制編譯所有依賴包
# -s：去掉符號表資訊，不過 panic 的時候 stack trace 就沒有任何檔名/行號資訊了
# -w：去掉 DWARF 除錯資訊，不過得到的程式就不能使用 gdb 進行除錯了
# 若對符號表無需求，-ldflags 直接新增 "-s" 即可
# 注：不建議-w和-s同時使用
$ go build -ldflags "-s -w" -o test2 main.go
$ du -sh test2
11M test2

upx(`brew/yum install upx`)

$ upx test2
 Ultimate Packer for eXecutables
 Copyright (C) 1996 - 2020
UPX 3.96 Markus Oberhumer, Laszlo Molnar & John Reiser Jan 23rd 2020
 File size Ratio Format Name
 -------------------- ------ ----------- -----------
 11490768 -> 4063248 35.36% macho/amd64 test2
Packed 1 file.

$ upx --brute test2
$ du -sh test2
4.6M test2

upx 的壓縮選項

[Golang] 利用build tags達到不同的build config

Mon, 18 Jul 2022 14:21:34 +0800

[Golang] 利用 build tags 達到不同的 build config

Go build -ldflags

package main

import (
 "fmt"
)

var flagString string

func main() {
 fmt.Println("This build with ldflag:", flagString)
}

$ go build -ldflags '-X main.flagString "test"'

$ ./main
This build with ldflag: test

Go build -tags

debug_config.go

//+build debug

package main

var TestString string = "test debug"
var TestString2 string = " and it will run every module with debug tag."

func GetConfigString() string {
 return "it is debug....."
}

//+build debug 前後需要一個空行(除非你在第一行)

release_config.go

//+build !debug

package main

var TestString string = "test release"
var TestString2 string = " and it will run every module as no debug tag."

func GetConfigString() string {
 return "it is release....."
}

main

package main

import (
 "fmt"
)

func main() {
 fmt.Println("This build is ", TestString, TestString2, GetConfigString())
}

$ go build -tags debug .
$ ./main
This build is test debug and it will run every module with debug tag. it is debug.....

$ go build .
$ ./main
This build is test release and it will run every module as no debug tag. it is release.....

Go embed 简明教程

Tue, 12 Jul 2022 13:57:03 +0800

Go embed 简明教程

嵌入

对于单个的文件，支持嵌入为string和 []byre
对于多个文件和文件夹，支持嵌入为新的文件系统 FS
比如导入 “embed” 包，即使无显式的使用
go:embed 指令用来嵌入，必须紧跟着嵌入后的变量名
只支持嵌入为 string, []byte 和 embed.FS 三种类型，这三种类型的别名(alias)和命名类型(如 type S string)都不可以
使用的是相对路径

string/[]byte

package main

import (
 _ "embed"
 "fmt"
)

//go:embed hello.txt
var s string

//go:embed hello.txt
var b []byte

func main() {
 fmt.Println(s)
 fmt.Println(b)
}

embed.FS

package main

import (
 "embed"
 "fmt"
)

//go:embed hello.txt hello2.txt
//go:embed "p/he llo.txt"
//go:embed `p/hello-2.txt`
//go:embed dir
var f embed.FS
func main() {
 data, _ := f.ReadFile("hello.txt")
 fmt.Println(string(data))
 data, _ = f.ReadFile("hello2.txt")
 fmt.Println(string(data))
 data, _ := f.ReadFile("p/he llo.txt")
 fmt.Println(string(data))
 data, _ = f.ReadFile("p/hello-2.txt")
 fmt.Println(string(data))
}

匹配模式

go:embed 指令中可以只写文件夹名，此文件夹中除了.和_开头的文件和文件夹都会被嵌入，并且子文件夹也会被递归的嵌入，形成一个此文件夹的文件系统。

golang pprof 实战

Wed, 22 Jun 2022 15:29:02 +0800

package main

import (
 // 略
 _ "net/http/pprof" // 会自动注册 handler 到 http server，方便通过 http 接口获取程序运行采样报告
 // 略
)

func main() {
 // 略

 runtime.GOMAXPROCS(1) // 限制 CPU 使用数，避免过载
 runtime.SetMutexProfileFraction(1) // 开启对锁调用的跟踪
 runtime.SetBlockProfileRate(1) // 开启对阻塞操作的跟踪

 go func() {
 // 启动一个 http server，注意 pprof 相关的 handler 已经自动注册过了
 // /debug/pprof/
 if err := http.ListenAndServe(":6060", nil); err != nil {
 log.Fatal(err)
 }
 os.Exit(0)
 }()

 // 略
}

http://localhost:6060/debug/pprof/

类型	描述	备注
allocs	内存分配情况的采样信息	可以用浏览器打开，但可读性不高
blocks	阻塞操作情况的采样信息	可以用浏览器打开，但可读性不高
cmdline	显示程序启动命令及参数	可以用浏览器打开，这里会显示 ./go-pprof-practice
goroutine	当前所有协程的堆栈信息	可以用浏览器打开，但可读性不高
heap	堆上内存使用情况的采样信息	可以用浏览器打开，但可读性不高
mutex	锁争用情况的采样信息	可以用浏览器打开，但可读性不高
profile	CPU 占用情况的采样信息	浏览器打开会下载文件
threadcreate	系统线程创建情况的采样信息	可以用浏览器打开，但可读性不高
trace	程序运行跟踪信息	浏览器打开会下载文件，本文不涉及，可另行参阅《深入浅出 Go trace》

排查 CPU 占用过高

Go 程式碼重構：23 倍效能追擊

Tue, 07 Jun 2022 13:42:07 +0800

Go 程式碼重構：23 倍效能追擊

$ go test -bench=. -cpuprofile cpu.prof
$ go tool pprof -svg cpu.prof > cpu.svg

$ go test -bench=. -trace trace.out
$ go tool trace trace.out

Dockertest 极速搭建集成测试环境神器

Thu, 12 May 2022 16:43:31 +0800

Go语言高性能编程手册（万字长文）

Fri, 22 Apr 2022 18:01:43 +0800

Go 语言高性能编程手册（万字长文）

常用数据结构

反射虽好，切莫贪杯

优先使用 strconv 而不是 fmt
少量的重复不比反射差
慎用 binary.Read 和 binary.Write
- binary.Read 和 binary.Write 使用反射并且很慢。如果有需要用到这两个函数的地方，我们应该手动实现这两个函数的相关功能，而不是直接去使用它们。

避免重复的字符串到字节切片的转换

指定容器(slice/map)容量

字符串拼接方式

行内拼接字符串推荐使用运算符 +
非行内拼接字符串推荐使用 strings.Builder

遍历 []struct{} 使用下标而不是 range

两种通过 index 遍历 []struct 性能没有差别，但是 range 遍历 []struct 中元素时，性能非常差。

range 遍历 []*struct 中元素时，與下标性能没有差别。

内存管理

使用空结构体节省内存

struct 布局要考虑内存对齐

CPU 访问内存时，并不是逐个字节访问，而是以字长（word size）为单位访问。比如 32 位的 CPU ，字长为 4 字节，那么 CPU 访问内存的单位也是 4 字节。这么设计的目的，是减少 CPU 访问内存的次数，加大 CPU 访问内存的吞吐量。比如同样读取 8 个字节的数据，一次读取 4 个字节那么只需要读取 2 次。

怎么选择 Go 文件读取方案

Tue, 29 Mar 2022 16:15:34 +0800

怎么选择 Go 文件读取方案

Go 提供了可一次性读取文件内容的方法：os.ReadFile 与 ioutil.ReadFile。在 Go 1.16 开始，ioutil.ReadFile 就等价于 os.ReadFile。

一次性加载文件的优缺点非常明显，它能减少 IO 次数，但它会将文件内容都加载至内存中，对于大文件，存在内存撑爆的风险。

逐行读取

Go 中 bufio.Reader 对象提供了一个 ReadLine() 方法，但其实我们更多地是使用 ReadBytes(’\n’) 或者 ReadString(’\n’) 代替。

func ReadLines(filename string) {
 fi, err := os.Open(filename)
 if err != nil{
 panic(err)
 }
 defer fi.Close()
 reader := bufio.NewReader(fi)
 for {
 _, err = reader.ReadString('\n')
 if err != nil {
 if err == io.EOF {
 break
 }
 panic(err)
 }
 }
}

块读取

块读取也称为分片读取，这也很好理解，我们可以将内容分成一块块的，每次读取指定大小的块内容。这里，我们将块大小设置为 4KB。

func ReadChunk(filename string) {
 f, err := os.Open(filename)
 if err != nil {
 panic(err)
 }
 defer f.Close()
 buf := make([]byte, 4*1024)
 r := bufio.NewReader(f)
 for {
 _, err = r.Read(buf)
 if err != nil {
 if err == io.EOF {
 break
 }
 panic(err)
 }
 }
}

result

BenchmarkOsReadFile4KB-8 92877 12491 ns/op
BenchmarkOsReadFile4MB-8 1620 744460 ns/op
BenchmarkOsReadFile4GB-8 1 7518057733 ns/op
signal: killed

BenchmarkReadLines4KB-8 90846 13184 ns/op
BenchmarkReadLines4MB-8 493 2338170 ns/op
BenchmarkReadLines4GB-8 1 3072629047 ns/op
BenchmarkReadLines16GB-8 1 12472749187 ns/op

BenchmarkReadChunk4KB-8 99848 12262 ns/op
BenchmarkReadChunk4MB-8 913 1233216 ns/op
BenchmarkReadChunk4GB-8 1 2095515009 ns/op
BenchmarkReadChunk16GB-8 1 8547054349 ns/op

在本文的测试条件下（每行数据 1KB），对于小对象 4KB 的读取，三种方式差距并不大；在 MB 级别的读取中，直接加载最快，但块读取也慢不了多少；上了 GB 后，块读取方式会最快。

Gopher 需要知道的几个结构体骚操作

Mon, 03 Jan 2022 15:08:31 +0800

Gopher 需要知道的几个结构体骚操作

NoCopy

$ go vet aaa.go
# command-line-arguments
./aaa.go:7:14: test passes lock by value: sync.WaitGroup contains sync.noCopy
./aaa.go:15:10: call of test copies lock value: sync.WaitGroup contains sync.noCopy

type noCopy struct{}

type WaitGroup struct {
 noCopy noCopy

 // 64-bit value: high 32 bits are counter, low 32 bits are waiter count.
 // 64-bit atomic operations require 64-bit alignment, but 32-bit
 // compilers only guarantee that 64-bit fields are 32-bit aligned.
 // For this reason on 32 bit architectures we need to check in state()
 // if state1 is aligned or not, and dynamically "swap" the field order if
 // needed.
 state1 uint64
 state2 uint32
}

noCopy 定义非常简单，空结构体，zero size 不占用空间(前提是非结构体的最后一个字段，否则还要是有 8 byte 空间开销)

golang的AES加密和解密的三种模式实现（CBC/ECB/CFB)

Wed, 22 Dec 2021 17:40:52 +0800

25秒读取16GB文件，Go怎么做到的？

Mon, 22 Nov 2021 11:07:17 +0800

打开文件后，我们有以下两个选项可以选择：

逐行读取文件，这有助于减少内存紧张，但需要更多的时间。

一次将整个文件读入内存并处理该文件，这将消耗更多内存，但会显著减少时间。

由于文件太大，即 16 GB，因此无法将整个文件加载到内存中。但是第一种选择对我们来说也是不可行的，因为我们希望在几秒钟内处理文件。

但你猜怎么着，还有第三种选择。瞧……相比于将整个文件加载到内存中，在 Go 语言中，我们还可以使用 bufio.NewReader()将文件分块加载。


func main() {
 s := time.Now()
 args := os.Args[1:]
 if len(args) != 6 { // for format LogExtractor.exe -f "From Time" -t "To Time" -i "Log file directory location"
 fmt.Println("Please give proper command line arguments")
 return
 }
 startTimeArg := args[1]
 finishTimeArg := args[3]
 fileName := args[5]

 file, err := os.Open(fileName)
 if err != nil {
 fmt.Println("cannot able to read the file", err)
 return
 }

 defer file.Close() // close after checking err

 queryStartTime, err := time.Parse("2006-01-02T15:04:05.0000Z", startTimeArg)
 if err != nil {
 fmt.Println("Could not able to parse the start time", startTimeArg)
 return
 }

 queryFinishTime, err := time.Parse("2006-01-02T15:04:05.0000Z", finishTimeArg)
 if err != nil {
 fmt.Println("Could not able to parse the finish time", finishTimeArg)
 return
 }

 filestat, err := file.Stat()
 if err != nil {
 fmt.Println("Could not able to get the file stat")
 return
 }

 fileSize := filestat.Size()
 offset := fileSize - 1
 lastLineSize := 0

 for {
 b := make([]byte, 1)
 n, err := file.ReadAt(b, offset)
 if err != nil {
 fmt.Println("Error reading file ", err)
 break
 }
 char := string(b[0])
 if char == "\n" {
 break
 }
 offset--
 lastLineSize += n
 }

 lastLine := make([]byte, lastLineSize)
 _, err = file.ReadAt(lastLine, offset+1)
 if err != nil {
 fmt.Println("Could not able to read last line with offset", offset, "and lastline size", lastLineSize)
 return
 }

 logSlice := strings.SplitN(string(lastLine), ",", 2)
 logCreationTimeString := logSlice[0]

 lastLogCreationTime, err := time.Parse("2006-01-02T15:04:05.0000Z", logCreationTimeString)
 if err != nil {
 fmt.Println("can not able to parse time : ", err)
 }

 if lastLogCreationTime.After(queryStartTime) && lastLogCreationTime.Before(queryFinishTime) {
 Process(file, queryStartTime, queryFinishTime)
 }

 fmt.Println("\nTime taken - ", time.Since(s))
}

func Process(f *os.File, start time.Time, end time.Time) error {
 linesPool := sync.Pool{New: func() interface{} {
 lines := make([]byte, 250*1024)
 return lines
 }}

 stringPool := sync.Pool{New: func() interface{} {
 lines := ""
 return lines
 }}

 r := bufio.NewReader(f)

 var wg sync.WaitGroup

 for {
 buf := linesPool.Get().([]byte)

 n, err := r.Read(buf)
 buf = buf[:n]

 if n == 0 {
 if err != nil {
 fmt.Println(err)
 break
 }
 if err == io.EOF {
 break
 }
 return err
 }

 nextUntillNewline, err := r.ReadBytes('\n')

 if err != io.EOF {
 buf = append(buf, nextUntillNewline...)
 }

 wg.Add(1)
 go func() {
 ProcessChunk(buf, &linesPool, &stringPool, start, end)
 wg.Done()
 }()

 }

 wg.Wait()
 return nil
}

func ProcessChunk(chunk []byte, linesPool *sync.Pool, stringPool *sync.Pool, start time.Time, end time.Time) {
 var wg2 sync.WaitGroup

 logs := stringPool.Get().(string)
 logs = string(chunk)

 linesPool.Put(chunk)

 logsSlice := strings.Split(logs, "\n")

 stringPool.Put(logs)

 chunkSize := 300
 n := len(logsSlice)
 noOfThread := n / chunkSize

 if n%chunkSize != 0 {
 noOfThread++
 }

 for i := 0; i < (noOfThread); i++ {

 wg2.Add(1)
 go func(s int, e int) {
 defer wg2.Done() // to avaoid deadlocks
 for i := s; i < e; i++ {
 text := logsSlice[i]
 if len(text) == 0 {
 continue
 }
 logSlice := strings.SplitN(text, ",", 2)
 logCreationTimeString := logSlice[0]

 logCreationTime, err := time.Parse("2006-01-02T15:04:05.0000Z", logCreationTimeString)
 if err != nil {
 fmt.Printf("\n Could not able to parse the time :%s for log : %v", logCreationTimeString, text)
 return
 }

 if logCreationTime.After(start) && logCreationTime.Before(end) {
 // fmt.Println(text)
 }
 }
 }(i*chunkSize, int(math.Min(float64((i+1)*chunkSize), float64(len(logsSlice)))))
 }

 wg2.Wait()
 logsSlice = nil
}

Gin 框架绑定 JSON 参数使用 jsoniter

Mon, 22 Nov 2021 10:27:39 +0800

Gin 框架绑定 JSON 参数使用 jsoniter

simple

go build -tags=jsoniter ./...

custom

implement `BindingBody` interface

// github.com/gin-gonic/gin@v1.6.3/binding/binding.go:36
// Binding describes the interface which needs to be implemented for binding the
// data present in the request such as JSON request body, query parameters or
// the form POST.
type Binding interface {
 Name() string
 Bind(*http.Request, interface{}) error
}
// BindingBody adds BindBody method to Binding. BindBody is similar with Bind,
// but it reads the body from supplied bytes instead of req.Body.
type BindingBody interface {
 Binding
 BindBody([]byte, interface{}) error
}

package custom
import (
 "bytes"
 "fmt"
 "io"
 "net/http"
 jsoniter "github.com/json-iterator/go"
 "github.com/gin-gonic/gin/binding"
)
// BindingJSON 替换Gin默认的binding，支持更丰富JSON功能
var BindingJSON = jsonBinding{}
// 可以自定义jsoniter配置或者添加插件
var json = jsoniter.ConfigCompatibleWithStandardLibrary
type jsonBinding struct{}
func (jsonBinding) Name() string {
 return "json"
}
func (jsonBinding) Bind(req *http.Request, obj interface{}) error {
 if req == nil || req.Body == nil {
 return fmt.Errorf("invalid request")
 }
 return decodeJSON(req.Body, obj)
}
func (jsonBinding) BindBody(body []byte, obj interface{}) error {
 return decodeJSON(bytes.NewReader(body), obj)
}
func decodeJSON(r io.Reader, obj interface{}) error {
 decoder := json.NewDecoder(r)
 if binding.EnableDecoderUseNumber {
 decoder.UseNumber()
 }
 if binding.EnableDecoderDisallowUnknownFields {
 decoder.DisallowUnknownFields()
 }
 if err := decoder.Decode(obj); err != nil {
 return err
 }
 return validate(obj)
}
func validate(obj interface{}) error {
 if binding.Validator == nil {
 return nil
 }
 return binding.Validator.ValidateStruct(obj)
}

// binding.JSON 替换成自定义的
ctx.ShouldBindWith(ms, binding.JSON)
ctx.ShouldBindBodyWith(ms, binding.JSON)

Golang os/exec 使用方法（筆記）

Fri, 12 Nov 2021 11:34:30 +0800

Golang os/exec 使用方法（筆記）

加入額外的環境參數再執行指令

cmd := exec.Command("programToExecute")
additionalEnv := "FOO=bar"
newEnv := append(os.Environ(), additionalEnv)
cmd.Env = newEnv
out, err := cmd.CombinedOutput()
if err != nil {
 log.Fatalf("cmd.Run() failed with %s\n", err)
}
fmt.Printf("%s", out)

在 Linux 下使用 os/exec + Pipe 或 Bash 運算式

rcmd := `iw dev | awk '$1=="Interface"{print $2}'`
cmd := exec.Command("bash", "-c", rcmd)
out, err := cmd.CombinedOutput()
if err != nil {
 log.Println(err.Error())
}
log.Println(string(out))

在 Windows 下使用 os/exec + Batch 運算式

cmd := exec.Command("cmd", "/c", "ffmpeg -i myfile.mp4 myfile.mp3 && del myfile.mp4")
out, err := cmd.CombinedOutput()
if err != nil {
 log.Println(err.Error())
}
log.Println(string(out))

解決 Windows CMD 在 os/exec 時會出現本地語言的問題

cmd := exec.Command("cmd", "/c", "chcp 65001 && netsh WLAN show drivers")
out, err := cmd.CombinedOutput()
if err != nil {
 log.Println(err.Error())
}
log.Println(string(out))

使用此方法得出的 output string 請記得把回傳的首行（chcp 的回傳結果）刪除掉再進行其他處理和讀取。

Golang基准测试

Thu, 02 Sep 2021 12:50:33 +0800

Golang 基准测试

基本使用

基准测试常用于代码性能测试，函数需要导入 testing 包，并定义以 Benchmark 开头的函数，参数为 testing.B 指针类型，在测试函数中循环调用函数多次

➜ go test -bench=. -run=none
goos: darwin
goarch: amd64
pkg: pkg06
cpu: Intel(R) Core(TM) i7-8850H CPU @ 2.60GHz
BenchmarkFib-12 250 4682682 ns/op
PASS
ok pkg06 1.875s
➜ go test -bench=. -benchmem -run=none
goos: darwin
goarch: amd64
pkg: pkg06
cpu: Intel(R) Core(TM) i7-8850H CPU @ 2.60GHz
BenchmarkFib-12 249 4686452 ns/op 0 B/op 0 allocs/op
PASS
ok pkg06 1.854s

bench 的工作原理

基准测试函数会被一直调用直到 b.N 无效，它是基准测试循环的次数
b.N 从 1 开始，如果基准测试函数在 1 秒内就完成 (默认值)，则 b.N 增加，并再次运行基准测试函数
b.N 的值会按照序列 1,2,5,10,20,50,... 增加，同时再次运行基准测测试函数
上述结果解读代表 1 秒内运行了 250 次，每次 4682682 ns
-12 后缀和用于运行次测试的 GOMAXPROCS 值有关。与 GOMAXPROCS 一样，此数字默认为启动时 Go 进程可见的 CPU 数。可以使用 -cpu 标识更改此值，可以传入多个值以列表形式来运行基准测试

传入 cpu num 进行测试

➜ go test -bench=. -cpu=1,2,4 -benchmem -run=none
goos: darwin
goarch: amd64
pkg: pkg06
cpu: Intel(R) Core(TM) i7-8850H CPU @ 2.60GHz
BenchmarkFib 244 4694667 ns/op 0 B/op 0 allocs/op
BenchmarkFib-2 255 4721201 ns/op 0 B/op 0 allocs/op
BenchmarkFib-4 256 4756392 ns/op 0 B/op 0 allocs/op
PASS
ok pkg06 5.826s

count 多次运行基准测试

因为热缩放、内存局部性、后台处理、gc 活动等等会导致单次的误差，所以一般会进行多次测试

[译] Go语言inline内联的策略与限制

Sat, 14 Aug 2021 20:33:41 +0800

package main

import "fmt"

func main() {
 n := []float32{120.4, -46.7, 32.50, 34.65, -67.45}
 fmt.Printf("The total is %.02f\n", sum(n))
}

func sum(s []float32) float32 {
 var t float32
 for _, v := range s {
 if t < 0 {
 t = add(t, v)
 } else {
 t = sub(t, v)
 }
 }
 return t
}

func add(a, b float32) float32 { return a + b }
func sub(a, b float32) float32 { return a - b }

使用参数-gflags="-m"运行，可显示被内联的函数

./op.go:3:6: can inline add
./op.go:7:6: can inline sub
./main.go:16:11: inlining call to sub
./main.go:14:11: inlining call to add
./main.go:7:12: inlining call to fmt.Printf

可以看到 add 方法被内联了。但是，为什么 sum 方法没有被内联呢？使用运行参数-gflags="-m -m"可以看到原因：

Go Modules 處理 Private GIT Repository 流程

Fri, 23 Jul 2021 09:40:39 +0800

Go Modules 處理 Private GIT Repository 流程

go env -w GOPRIVATE=github.com/appleboy
1. go env -w GONOPROXY=github.com/appleboy
2. go env -w GONOSUMDB=github.com/appleboy
git config --global url."https://$USERNAME:$ACCESS_TOKEN@github.com".insteadOf "https://github.com"
1. git config --global url.ssh://git@your.private.git/.insteadOf https://your.private.git/

Gin中文文档

Thu, 22 Jul 2021 15:55:48 +0800

自定义路由日志的格式

default

[GIN-debug] POST /foo --> main.main.func1 (3 handlers)
[GIN-debug] GET /bar --> main.main.func2 (3 handlers)
[GIN-debug] GET /status --> main.main.func3 (3 handlers)

import (
 "log"
 "net/http"

 "github.com/gin-gonic/gin"
)

func main() {
 r := gin.Default()
 gin.DebugPrintRouteFunc = func(httpMethod, absolutePath, handlerName string, nuHandlers int) {
 log.Printf("endpoint %v %v %v %v\n", httpMethod, absolutePath, handlerName, nuHandlers)
 }

 r.POST("/foo", func(c *gin.Context) {
 c.JSON(http.StatusOK, "foo")
 })

 r.GET("/bar", func(c *gin.Context) {
 c.JSON(http.StatusOK, "bar")
 })

 r.GET("/status", func(c *gin.Context) {
 c.JSON(http.StatusOK, "ok")
 })

 // Listen and Server in http://0.0.0.0:8080
 r.Run()
}

Gin 运行多个服务

package main

import (
 "log"
 "net/http"
 "time"

 "github.com/gin-gonic/gin"
 "golang.org/x/sync/errgroup"
)

var (
 g errgroup.Group
)

func router01() http.Handler {
 e := gin.New()
 e.Use(gin.Recovery())
 e.GET("/", func(c *gin.Context) {
 c.JSON(
 http.StatusOK,
 gin.H{
 "code": http.StatusOK,
 "error": "Welcome server 01",
 },
 )
 })

 return e
}

func router02() http.Handler {
 e := gin.New()
 e.Use(gin.Recovery())
 e.GET("/", func(c *gin.Context) {
 c.JSON(
 http.StatusOK,
 gin.H{
 "code": http.StatusOK,
 "error": "Welcome server 02",
 },
 )
 })

 return e
}

func main() {
 server01 := &http.Server{
 Addr: ":8080",
 Handler: router01(),
 ReadTimeout: 5 * time.Second,
 WriteTimeout: 10 * time.Second,
 }

 server02 := &http.Server{
 Addr: ":8081",
 Handler: router02(),
 ReadTimeout: 5 * time.Second,
 WriteTimeout: 10 * time.Second,
 }

 g.Go(func() error {
 return server01.ListenAndServe()
 })

 g.Go(func() error {
 return server02.ListenAndServe()
 })

 if err := g.Wait(); err != nil {
 log.Fatal(err)
 }
}

XML、JSON、YAML 和 ProtoBuf 渲染（输出格式）

SecureJSON

使用 SecureJSON 可以防止 json 劫持，如果返回的数据是数组，则会默认在返回值前加上"while(1)"

高效能 Go 工作坊

Wed, 28 Apr 2021 16:37:05 +0800

高效能 Go 工作坊

Golang 服務的檔案句柄超出系統限制（too many open files）

Mon, 16 Dec 2019 10:14:33 +0800

Golang 服務的檔案句柄超出系統限制（too many open files）

查看系統設定：ulimit -a | grep open，系統設定正常。
查看服務的開啟檔案限制：cat /proc/40636/limits，服務沒有繼承系統設定，仍是預設的 1024 限制。
查看服務開啟檔案數（連線數）：lsof -p 40636 | wc -l，已超過限制，因此報錯。
查看開啟了哪些連線：lsof -p 40636 > openfiles.log，發現很多 HTTP 連線未關閉，IP 是告警服務的介面。沿著線索找到原因：程式解析設定時出錯，對告警服務大量連線後未關閉，導致 too many open files。至於為何服務未繼承系統最大限制，還需進一步查看。
最終原因：服務由 supervisor 管理，supervisor 預設 minfds 為 1024。於設定中加入 minfds=81920，並重啟 supervisorctl reload。