Hero Image
怎么选择 Go 文件读取方案

怎么选择 Go 文件读取方案 Go 提供了可一次性读取文件内容的方法：os.ReadFile 与 ioutil.ReadFile。在 Go 1.16 开始，ioutil.ReadFile 就等价于 os.ReadFile。 一次性加载文件的优缺点非常明显，它能减少 IO 次数，但它会将文件内容都加载至内存中，对于大文件，存在内存撑爆的风险。 逐行读取 Go 中 bufio.Reader 对象提供了一个 ReadLine() 方法，但其实我们更多地是使用 ReadBytes(’\n’) 或者 ReadString(’\n’) 代替。 func ReadLines(filename string) { fi, err := os.Open(filename) if err != nil{ panic(err) } defer fi.Close() reader := bufio.NewReader(fi) for { _, err = reader.ReadString('\n') if err != nil { if err == io.EOF { break } panic(err) } } } 块读取 块读取也称为分片读取，这也很好理解，我们可以将内容分成一块块的，每次读取指定大小的块内容。这里，我们将块大小设置为 4KB。 func ReadChunk(filename string) { f, err := os.Open(filename) if err != nil { panic(err) } defer f.Close() buf := make([]byte, 4*1024) r := bufio.NewReader(f) for { _, err = r.Read(buf) if err != nil { if err == io.EOF { break } panic(err) } } } result BenchmarkOsReadFile4KB-8 92877 12491 ns/op BenchmarkOsReadFile4MB-8 1620 744460 ns/op BenchmarkOsReadFile4GB-8 1 7518057733 ns/op signal: killed BenchmarkReadLines4KB-8 90846 13184 ns/op BenchmarkReadLines4MB-8 493 2338170 ns/op BenchmarkReadLines4GB-8 1 3072629047 ns/op BenchmarkReadLines16GB-8 1 12472749187 ns/op BenchmarkReadChunk4KB-8 99848 12262 ns/op BenchmarkReadChunk4MB-8 913 1233216 ns/op BenchmarkReadChunk4GB-8 1 2095515009 ns/op BenchmarkReadChunk16GB-8 1 8547054349 ns/op 在本文的测试条件下（每行数据 1KB），对于小对象 4KB 的读取，三种方式差距并不大；在 MB 级别的读取中，直接加载最快，但块读取也慢不了多少；上了 GB 后，块读取方式会最快。

Tuesday, March 29, 2022 Read
Hero Image
將 List 轉置為 dict 的 list

將 List 轉置為 dict 的 list list my_users: - aaa - bbb - ccc dict my_users: - Name: aaa - Name: bbb - Name: ccc playbook - name: Transform data vars: orig_users: - aaa - bbb - ccc my_users: [] set_fact: my_users: "{{ my_users + [{} | combine({'Name': item})] }}" loop: "{{ orig_users }}" 說明如下： 利用 combine 這個 filter 來組出 dict 利用 loop 去 iterate orig_users 用 my_users + [{} | combine({‘Name’: item})] 來做 List append```

Wednesday, March 9, 2022 Read
Hero Image
Misspelling, missing collection, or incorrect module path for fortios_system_config_backup_restore

Misspelling, missing collection, or incorrect module path for fortios_system_config_backup_restore issue Hi, I’m trying to retrieve the configuration from a Fortigate using the module fortios_system_config_backup_restore and basing on your playbook example. I have ansible 2.10.6. When I use the collection fortinet.fortios:1.1.8, I got the following error: missing 1 required positional argument: 'mod' The full traceback is: Traceback (most recent call last): File "/home/ansible/.ansible/tmp/ansible-local-8668osvgufxk/ansible-tmp-1615481963.0092971-8768-3877705280026/AnsiballZ_fortios_system_config_backup_restore.py", line 102, in <module> _ansiballz_main() File "/home/ansible/.ansible/tmp/ansible-local-8668osvgufxk/ansible-tmp-1615481963.0092971-8768-3877705280026/AnsiballZ_fortios_system_config_backup_restore.py", line 94, in _ansiballz_main invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS) File "/home/ansible/.ansible/tmp/ansible-local-8668osvgufxk/ansible-tmp-1615481963.0092971-8768-3877705280026/AnsiballZ_fortios_system_config_backup_restore.py", line 40, in invoke_module runpy.run_module(mod_name='ansible_collections.fortinet.fortios.plugins.modules.fortios_system_config_backup_restore', init_globals=None, run_name='__main__', alter_sys=True) File "/usr/lib/python3.8/runpy.py", line 207, in run_module return _run_module_code(code, init_globals, run_name, mod_spec) File "/usr/lib/python3.8/runpy.py", line 97, in _run_module_code _run_code(code, mod_globals, init_globals, File "/usr/lib/python3.8/runpy.py", line 87, in _run_code exec(code, run_globals) File "/tmp/ansible_fortinet.fortios.fortios_system_config_backup_restore_payload_mplamrs7/ansible_fortinet.fortios.fortios_system_config_backup_restore_payload.zip/ansible_collections/fortinet/fortios/plugins/modules/fortios_system_config_backup_restore.py", line 472, in <module> File "/tmp/ansible_fortinet.fortios.fortios_system_config_backup_restore_payload_mplamrs7/ansible_fortinet.fortios.fortios_system_config_backup_restore_payload.zip/ansible_collections/fortinet/fortios/plugins/modules/fortios_system_config_backup_restore.py", line 436, in main TypeError: __init__() missing 1 required positional argument: 'mod' fatal: [fortigate-01]: FAILED! => { "changed": false, "module_stderr": "Traceback (most recent call last):\n File \"/home/ansible/.ansible/tmp/ansible-local-8668osvgufxk/ansible-tmp-1615481963.0092971-8768-3877705280026/AnsiballZ_fortios_system_config_backup_restore.py\", line 102, in <module>\n _ansiballz_main()\n File \"/home/ansible/.ansible/tmp/ansible-local-8668osvgufxk/ansible-tmp-1615481963.0092971-8768-3877705280026/AnsiballZ_fortios_system_config_backup_restore.py\", line 94, in _ansiballz_main\n invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)\n File \"/home/ansible/.ansible/tmp/ansible-local-8668osvgufxk/ansible-tmp-1615481963.0092971-8768-3877705280026/AnsiballZ_fortios_system_config_backup_restore.py\", line 40, in invoke_module\n runpy.run_module(mod_name='ansible_collections.fortinet.fortios.plugins.modules.fortios_system_config_backup_restore', init_globals=None, run_name='__main__', alter_sys=True)\n File \"/usr/lib/python3.8/runpy.py\", line 207, in run_module\n return _run_module_code(code, init_globals, run_name, mod_spec)\n File \"/usr/lib/python3.8/runpy.py\", line 97, in _run_module_code\n _run_code(code, mod_globals, init_globals,\n File \"/usr/lib/python3.8/runpy.py\", line 87, in _run_code\n exec(code, run_globals)\n File \"/tmp/ansible_fortinet.fortios.fortios_system_config_backup_restore_payload_mplamrs7/ansible_fortinet.fortios.fortios_system_config_backup_restore_payload.zip/ansible_collections/fortinet/fortios/plugins/modules/fortios_system_config_backup_restore.py\", line 472, in <module>\n File \"/tmp/ansible_fortinet.fortios.fortios_system_config_backup_restore_payload_mplamrs7/ansible_fortinet.fortios.fortios_system_config_backup_restore_payload.zip/ansible_collections/fortinet/fortios/plugins/modules/fortios_system_config_backup_restore.py\", line 436, in main\nTypeError: __init__() missing 1 required positional argument: 'mod'\n", "module_stdout": "", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 1 } Using the recent pre release fortinet.fortios:2.0.0 , I got:

Thursday, January 6, 2022 Read
Hero Image
Gopher 需要知道的几个结构体骚操作

Gopher 需要知道的几个结构体骚操作 NoCopy $ go vet aaa.go # command-line-arguments ./aaa.go:7:14: test passes lock by value: sync.WaitGroup contains sync.noCopy ./aaa.go:15:10: call of test copies lock value: sync.WaitGroup contains sync.noCopy type noCopy struct{} type WaitGroup struct { noCopy noCopy // 64-bit value: high 32 bits are counter, low 32 bits are waiter count. // 64-bit atomic operations require 64-bit alignment, but 32-bit // compilers only guarantee that 64-bit fields are 32-bit aligned. // For this reason on 32 bit architectures we need to check in state() // if state1 is aligned or not, and dynamically "swap" the field order if // needed. state1 uint64 state2 uint32 } noCopy 定义非常简单，空结构体，zero size 不占用空间(前提是非结构体的最后一个字段，否则还要是有 8 byte 空间开销)

Monday, January 3, 2022 Read
Hero Image
Nginx怎样隐藏上游错误

Nginx 怎样隐藏上游错误 Nginx 允许对以下 7 种可以进行重试的错误码启用 next upstream 功能 403 Forbidden 404 Not Found 429 Too Many Requests 500 Internal Server Error 502 Bad Gateway 503 Server Unavailable 504 Gateway Timeout 当上游返回 404 错误时，改为通过 200 返回一张找不到资源的图片 此时，可以通过 proxy_intercept_errors 指令完成这一功能 当 proxy_intercept_errors 开启后，对于上游返回的大于等于 300 响应码的请求，都可以基于 error_page 指令继续处理 location /ih { proxy_pass http://ihBackend; proxy_intercept_errors on; error_page 404 = /404.html; } location = /404.html { alias html/404_not_found.html; }

Monday, December 27, 2021 Read
Hero Image
Argo CD ApplicationSet Controller: 世界為我而轉動！

Argo CD ApplicationSet Controller: 世界為我而轉動！ Argo CD # 安裝 kind，其他平台安裝方式請參閱官方文件 # 用來運行輕量 K8s Cluster 於本地端 ~$ brew install kind # 安裝 kubectx，其他平台安裝方式請參閱官方文件 # 用來方便切換不同 k8s context ~$ brew install kubectx # 安裝 helm，其他平台安裝方式請參閱官方文件 # K8s 套件管理工具 ~$ brew install helm # 安裝 kubectl, 其他平台安裝方式請參閱官方文件 # 用來與 K8s Cluster API Server 溝通 ~$ brew install kubectl # 安裝 argocd cli, 其他平台安裝方式請參閱官方文件 # 用來與 Argo CD 溝通 ~$ brew install argocd

Monday, December 27, 2021 Read
Hero Image
golang的AES加密和解密的三种模式实现（CBC/ECB/CFB)

golang 的 AES 加密和解密的三种模式实现（CBC/ECB/CFB) How to encrypt a file using Go

Wednesday, December 22, 2021 Read
Hero Image
【理解 Cilium 系列文章】(一) 初識 Cilium

【理解 Cilium 系列文章】(一) 初識 Cilium 當前 k8s Service 負載均衡的實現現狀 在 Cilium 出現之前， Service 由 kube-proxy 來實現，實現方式有 userspace ， iptables ， ipvs 三種模式。 Userspace 當前模式下，kube-proxy 作為反向代理,監聽隨機端口，通過 iptables 規則將流量重定向到代理端口，再由 kube-proxy 將流量轉發到 後端 pod。Service 的請求會先從用户空間進入內核 iptables，然後再回到用户空間，代價較大，性能較差。 Iptables 存在的問題： 1.可擴展性差。隨着 service 數據達到數千個，其控制面和數據面的性能都會急劇下降。原因在於 iptables 控制面的接口設計中，每添加一條規則，需要遍歷和修改所有的規則，其控制面性能是 O(n²) 。在數據面，規則是用鏈表組織的，其性能是 O(n) 2.LB 調度算法僅支持隨機轉發 Ipvs 模式 IPVS 是專門為 LB 設計的。它用 hash table 管理 service，對 service 的增刪查找都是 O(1)的時間複雜度。不過 IPVS 內核模塊沒有 SNAT 功能，因此借用了 iptables 的 SNAT 功能。 IPVS 針對報文做 DNAT 後，將連接信息保存在 nf_conntrack 中，iptables 據此接力做 SNAT。該模式是目前 Kubernetes 網絡性能最好的選擇。但是由於 nf_conntrack 的複雜性，帶來了很大的性能損耗。騰訊針對該問題做過相應的優化 【繞過 conntrack，使用 eBPF 增強 IPVS 優化 K8s 網絡性能】

Tuesday, December 21, 2021 Read
Hero Image
Kubernetes Without kube-proxy

Kubernetes Without kube-proxy Quick-Start kubeadm init --skip-phases=addon/kube-proxy # Setup Helm repository helm repo add cilium https://helm.cilium.io/ helm install cilium cilium/cilium --version 1.9.18 \ --namespace kube-system \ --set kubeProxyReplacement=strict \ --set k8sServiceHost=REPLACE_WITH_API_SERVER_IP \ --set k8sServicePort=REPLACE_WITH_API_SERVER_PORT

Monday, December 20, 2021 Read
Hero Image
Zsh tab-completion not working

Zsh tab-completion not working Question Although I’ve used Oh-My-Zsh in the past, I decided this time around (i.e. setting up a new computer) I’d try to avoid installing it to keep things a bit leaner. Right now I’m trying to cherry-pick Oh-My-Zsh’s insensitive tab-completion feature. Digging around its source repo, I found the following line: zstyle ':completion:*' matcher-list 'm:{a-zA-Z}={A-Za-z}' 'r:|=*' 'l:|=* r:|=*' I’m fairly confident this is the line Oh-My-Zsh executes to do what I want, so I tried adding it to my .zshrc. Restarted my terminal. Typed cd desk, then hit [tab]. No dice – I didn’t get cd Desktop/.

Friday, December 10, 2021 Read
Hero Image
Linkerd 2.8 - 實現超級簡單又安全的多叢集(multicluster) Kubernetes 架構

Linkerd 2.8 - 實現超級簡單又安全的多叢集(multicluster) Kubernetes 架構

Thursday, December 9, 2021 Read
Hero Image
Best Practices for Writing Bash Scripts

Best Practices for Writing Bash Scripts Shell Scripting - Best Practices #!/usr/bin/env bash # Bash3 Boilerplate. Copyright (c) 2014, kvz.io set -o errexit set -o pipefail set -o nounset # set -o xtrace # Set magic variables for current file & dir __dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" __file="${__dir}/$(basename "${BASH_SOURCE[0]}")" __base="$(basename ${__file} .sh)" __root="$(cd "$(dirname "${__dir}")" && pwd)" # <-- change this as it depends on your app arg1="${1:-}" Use shift to read function arguments This makes it easier to reorder arguments, if you change your mind later.

Monday, December 6, 2021 Read