Hero Image
[Kubernetes] Service Overview

[Kubernetes] Service Overview Define Service With selector Since you need Pods before you define a Service, assume there are Pods in the cluster (exposing TCP port 9376) with the label app=MyApp. You can define a Service as an abstraction layer in front of those pods and provide the service through a domain name. kind: Service apiVersion: v1 metadata: name: my-service spec: # type has four options (ClusterIP, NodePort, LoadBalancer, ExternalName) # default is ClusterIP type: ClusterIP # select pods with "app=MyApp" selector: app: MyApp # Service port configuration ports: - protocol: TCP port: 80 # This is the port number exposed by the Pod targetPort: 9376 Pod <---> Endpoint(tcp:9376) <---> Service(tcp:80, with VIP)

Wednesday, November 24, 2021 Read
Hero Image
How Go Reads a 16GB File in 25 Seconds

How Go Reads a 16GB File in 25 Seconds Reading 16GB File in Seconds, Golang After opening the file, we have two options: Read the file line by line, which reduces memory pressure but takes more time. Read the entire file into memory and process it, which uses more memory but significantly reduces time. Because the file is too large (16 GB), we cannot load the entire file into memory. But the first option is also not feasible for us because we want to process the file within seconds.

Monday, November 22, 2021 Read
Hero Image
Bind JSON with jsoniter in Gin

Bind JSON with jsoniter in Gin simple go build -tags=jsoniter ./... custom implement BindingBody interface // github.com/gin-gonic/gin@v1.6.3/binding/binding.go:36 // Binding describes the interface which needs to be implemented for binding the // data present in the request such as JSON request body, query parameters or // the form POST. type Binding interface { Name() string Bind(*http.Request, interface{}) error } // BindingBody adds BindBody method to Binding. BindBody is similar with Bind, // but it reads the body from supplied bytes instead of req.Body. type BindingBody interface { Binding BindBody([]byte, interface{}) error } package custom import ( "bytes" "fmt" "io" "net/http" jsoniter "github.com/json-iterator/go" "github.com/gin-gonic/gin/binding" ) // BindingJSON replaces Gin's default binding and supports richer JSON features var BindingJSON = jsonBinding{} // You can customize jsoniter config or add plugins var json = jsoniter.ConfigCompatibleWithStandardLibrary type jsonBinding struct{} func (jsonBinding) Name() string { return "json" } func (jsonBinding) Bind(req *http.Request, obj interface{}) error { if req == nil || req.Body == nil { return fmt.Errorf("invalid request") } return decodeJSON(req.Body, obj) } func (jsonBinding) BindBody(body []byte, obj interface{}) error { return decodeJSON(bytes.NewReader(body), obj) } func decodeJSON(r io.Reader, obj interface{}) error { decoder := json.NewDecoder(r) if binding.EnableDecoderUseNumber { decoder.UseNumber() } if binding.EnableDecoderDisallowUnknownFields { decoder.DisallowUnknownFields() } if err := decoder.Decode(obj); err != nil { return err } return validate(obj) } func validate(obj interface{}) error { if binding.Validator == nil { return nil } return binding.Validator.ValidateStruct(obj) } // Replace binding.JSON with the custom one ctx.ShouldBindWith(ms, binding.JSON) ctx.ShouldBindBodyWith(ms, binding.JSON)

Monday, November 22, 2021 Read
Hero Image
Golang os/exec usage (notes)

Golang os/exec usage (notes) Add extra environment variables before running a command cmd := exec.Command("programToExecute") additionalEnv := "FOO=bar" newEnv := append(os.Environ(), additionalEnv) cmd.Env = newEnv out, err := cmd.CombinedOutput() if err != nil { log.Fatalf("cmd.Run() failed with %s\n", err) } fmt.Printf("%s", out) Use os/exec with pipes or bash expressions on Linux rcmd := `iw dev | awk '$1=="Interface"{print $2}'` cmd := exec.Command("bash", "-c", rcmd) out, err := cmd.CombinedOutput() if err != nil { log.Println(err.Error()) } log.Println(string(out)) Use os/exec with batch expressions on Windows cmd := exec.Command("cmd", "/c", "ffmpeg -i myfile.mp4 myfile.mp3 && del myfile.mp4") out, err := cmd.CombinedOutput() if err != nil { log.Println(err.Error()) } log.Println(string(out)) Fix local language issues in Windows CMD with os/exec cmd := exec.Command("cmd", "/c", "chcp 65001 && netsh WLAN show drivers") out, err := cmd.CombinedOutput() if err != nil { log.Println(err.Error()) } log.Println(string(out)) When using this method, remember to remove the first line of output (the chcp response) before further processing.

Friday, November 12, 2021 Read
Hero Image
Uncle Joe teaches Elastic - 30 - Elasticsearch optimization tips (4/4) - Shard optimization management

Uncle Joe teaches Elastic - 30 - Elasticsearch optimization tips (4/4) - Shard optimization management Planning considerations During search, each shard uses one thread. Too many shards can slow things down. A node has multiple thread pools by function. The search (count/search/suggest) pool size is int((# of allocated processors * 3) / 2) + 1 and the default queue_size is 1000. The write (index/delete/update/bulk) pool size is # of allocated processors and the default queue_size is also 1000. (Official docs - Thread pools)

Wednesday, November 10, 2021 Read
Hero Image
Regex for Markdown Syntax

Regex for Markdown Syntax basic-syntax # Headings headerOne = `(#{1}\s)(.*)` headeTwo = `(#{2}\s)(.*)` headerThree = `(#{3}\s)(.*)` headerFour = `(#{4}\s)(.*)` headerFive = `(#{5}\s)(.*)` headerSix = `(#{6}\s)(.*)` # Bold and Italic Text boldItalicText = `(\*|\_)+(\S+)(\*|\_)+` # Links linkText = `(\[.*\])(\((http)(?:s)?(\:\/\/).*\))` # Images imageFile = `(\!)(\[(?:.*)?\])(\(.*(\.(jpg|png|gif|tiff|bmp))(?:(\s\"|\')(\w|\W|\d)+(\"|\'))?\)` # Unordered List listText = `(^(\W{1})(\s)(.*)(?:$)?)+` # Numbered Text numberedListText = `(^(\d+\.)(\s)(.*)(?:$)?)+` # Block Quotes blockQuote = `((^(\>{1})(\s)(.*)(?:$)?)+` # Inline Code inlineCode = "(\\`{1})(.*)(\\`{1})" # Code Block codeBlock = "(\\`{3}\\n+)(.*)(\\n+\\`{3})" # Horizontal Line horizontalLine = `(\=|\-|\*){3}` # Email Text emailText = `(\<{1})(\S+@\S+)(\>{1})` # TABLES tableText = `(((\|)([a-zA-Z\d+\s#!@'"():;\\\/.\[\]\^<={$}>?(?!-))]+))+(\|))(?:\n)?((\|)(-+))+(\|)(\n)((\|)(\W+|\w+|\S+))+(\|$)`

Friday, October 29, 2021 Read
Hero Image
Google Cloud Platform(GCP): Access Linux Server using GUI running in GCP instance using Windows Remote Desktop Connection.

Google Cloud Platform(GCP): Access Linux Server using GUI running in GCP instance using Windows Remote Desktop Connection. # This will install GUI and make it as a default startup option and then restart the machine. $ sudo yum install xrdp tigervnc-server ~# sudo su ~# passwd ~# systemctl enable --now xrdp ~# netstat -antup | grep xrdp tcp 0 0 0.0.0.0:3389 0.0.0.0:* LISTEN 10202/xrdp tcp 0 0 127.0.0.1:3350 0.0.0.0:* LISTEN 10201/xrdp-sesman

Wednesday, October 20, 2021 Read
Hero Image
[Notes] Build an Ubuntu remote desktop environment on GCP

[Notes] Build an Ubuntu remote desktop environment on GCP Install # dependency sudo apt-get install ubuntu-desktop gnome-panel gnome-settings-daemon metacity nautilus gnome-terminal # VNC Server sudo apt-get install vnc4server # After install, run vncserver and set the password when prompted vncserver Modify ~/.vnc/xstartup #!/bin/sh # Uncomment the following two lines for normal desktop: # unset SESSION_MANAGER # exec /etc/X11/xinit/xinitrc [ -x /etc/vnc/xstartup ] && exec /etc/vnc/xstartup [ -r $HOME/.Xresources ] && xrdb $HOME/.Xresources xsetroot -solid grey vncconfig -iconic & x-terminal-emulator -geometry 80x24+10+10 -ls -title "$VNCDESKTOP Desktop" & x-window-manager & gnome-panel & gnome-settings-daemon & metacity & nautilus & Execute # Kill the current vncserver and restart vncserver -kill :1 # vncserver defaults to port 5900; :1 is 5901, and so on vncserver :1 Start vncserver on reboot @reboot /usr/bin/vncserver :1

Wednesday, October 20, 2021 Read
Hero Image
Understand the crypto trio: Encode, Encrypt, and Hash

Understand the crypto trio: Encode, Encrypt, and Hash Encoding It does not change the data and has no encryption effect. It simply represents the data in another form. A classic example is Morse code. JavaScript has two useful functions: encodeURI and decodeURI, which encode special characters in URLs (spaces, punctuation, etc.) into URL-safe formats. Base64 encodes binary data into ASCII characters. Huffman Coding is a lossless compression encoding algorithm. In short, it compresses by abbreviating frequently used symbols. Encryption Encryption and decryption require a key. In the simplest Caesar cipher, each letter is shifted by an offset, and that offset is the key for encryption/decryption.

Monday, October 18, 2021 Read
Hero Image
Fix Nginx 500 errors (too many open files, connection)

Fix Nginx 500 errors (too many open files, connection) Nginx 500 errors only show up in logs. Two common cases: socket() failed (24: Too many open files) while connecting to upstream $ sudo su - www-data $ ulimit -n # check current limit (ulimit -a shows all params) 1024 # vim /etc/security/limits.conf # set nofile (max number of open files) # add/modify the following two lines * soft nofile 655360 * hard nofile 655360 ulimit -n # log out and log back in to see the new value 655360 # If ulimit -n is not 655360, run ulimit -n 655360 to force set it # Then verify with ulimit -n or ulimit -Sn (soft) and ulimit -Hn (hard) (or ulimit -a). # Calculate and set from system level lsof | wc -l # count open files sudo vim /etc/sysctl.conf fs.file-max = 3268890 sudo sysctl -p 512 worker_connections are not enough while connecting to upstream # /etc/nginx/nginx.conf worker_connections 10240; # Refer to Nginx CoreModule # worker_processes 2; # worker_rlimit_nofile 10240; # events { # # worker_connections 10240; # } # Increasing Nginx connections can slow down overall speed because php-cgi is not enough. # Adjust as follows. # php-cgi was started with phpfcgid_children="10" and phpfcgid_requests="500" # ab was run on another server, connect via a switch using GBit ethernet # http://till.klampaeckel.de/blog/archives/30-PHP-performance-III-Running-nginx.html # vim /etc/nginx/nginx.conf worker_connections 10240; worker_rlimit_nofile # vim /etc/init.d/php-fcgi PHP_FCGI_CHILDREN=15 PHP_FCGI_MAX_REQUESTS=1000 change to PHP_FCGI_CHILDREN=512 # or 150 and increase gradually, watch MySQL connections PHP_FCGI_MAX_REQUESTS=10240 # The article's phpfcgid_stop() function is good and can be used if needed. # phpfcgid_stop() { # echo "Stopping $name." # pids=`pgrep php-cgi` # pkill php-cgi # wait_for_pids $pids # }

Saturday, October 9, 2021 Read
Hero Image
Super easy! Build a beautiful and handy zsh terminal

Super easy! Build a beautiful and handy zsh terminal Make your terminal stylish and fast with zsh + zim + powerlevel10k Build a 10X Engineer shell environment

Thursday, October 7, 2021 Read
Hero Image
Sign git commits with GPG

Sign git commits with GPG

Tuesday, October 5, 2021 Read