Links

Is there a regular expression to detect a valid regular expression?

Is there a regular expression to detect a valid regular expression? This is a recursive regex, and is not supported by many regex engines. PCRE based ones should support it. / ^ # start of string ( # first group start (?: (?:[^?+*{}()[\]\\|]+ # literals and ^, $ | \\. # escaped characters | \[ (?: \^?\\. | \^[^\\] | [^\\^] ) # character classes (?: [^\]\\]+ | \\. )* \] | $ (?:\?[:=!]|\?<[=!]|\?>)? (?1)?? $ # parenthesis, with recursive content | $\? (?:R|[+-]?\d+) $ # recursive matching ) (?: (?:[?+*]|\{\d+(?:,\d*)?\}) [?+]? )? # quantifiers | \| # alternative )* # repeat content ) # end first group $ # end of string / Without whitespace and comments

Friday, October 4, 2019 Read

Force file download with Nginx

Force file download with Nginx add_header Content-Disposition 'attachment;'; server { listen 80; server_name my.domain.com; location ~ ^.*/(?P<request_basename>[^/]+\.(mp3))$ { root /path/to/mp3/ add_header Content-Disposition 'attachment; filename="$request_basename"'; } } { listen 80; server_name backup.baifu-tech.net; root /data/backup/rechargecent-mago; location / { auth_basic "baifu backup center"; auth_basic_user_file /etc/nginx/ssl/htpasswd; autoindex on; autoindex_exact_size off; autoindex_localtime on; } }

Monday, August 19, 2019 Read

Shell Script Study Notes

Shell Script Study Notes Arithmetic operations val=`expr $a + $b` Operators Symbol Description Example ! NOT [ ! false ] -o OR [ $a -lt 20 -o $b -gt 20 ] -a AND [ $a -lt 20 -a $b -gt 20 ] = equality check [ $a = $b ] != inequality check [ $a != $b ] -z string length is 0, returns true if 0 [ -z $a ] -n string length is not 0, returns true if not 0 [ -n $a ] str check whether string is empty, true if not [ $a ] -b check whether file is a block device [ -b $file ] -c check whether file is a character device .. -d check whether file is a directory [ -d $file ] -f check whether file is a regular file [ -f $file ] -r check whether file is readable .. -w check whether file is writable .. -x check whether file is executable .. -s check whether file is empty .. -e check whether file exists .. Special variables Variable Meaning $0 file name of the current script $n arguments passed to a script or function; n is the position $# number of arguments passed to a script or function $* all arguments as a single word, e.g., “1 2 3” $@ all arguments as separate words, e.g., “1” “2” “3” $? exit status of the last command or return value of a function $$ current shell process ID; for scripts, the PID of the script process POSIX program exit statuses Code Meaning 0 command exited successfully > 0 failure during redirection or word expansion (~, variables, commands, arithmetic, and word splitting) 1 - 125 command exited unsuccessfully; specific meanings are command-defined 126 command found but file is not executable 127 command not found > 128 command died from a signal Input/output redirection Command Description command > file redirect output to file command > file redirect output to file by appending n > file redirect file descriptor n to file n » file redirect file descriptor n to file by appending n >& m merge output file m and n n <& m merge input file m and n « tag use content between start tag and end tag as input File include Use . or source to include files.

Wednesday, August 7, 2019 Read

Excessive Errors from `sentry.lang.javascript.processor: SoftTimeLimitExceeded()`

Excessive Errors from sentry.lang.javascript.processor: SoftTimeLimitExceeded() Counters-0 queue is rising continuously Excessive Errors from sentry.lang.javascript.processor: SoftTimeLimitExceeded() mattrobenolt: Was just going to propose this. :) Unfortunately, this is just how this feature works. If it’s not useful and is just wasting resources, it’s easy to disable globally by setting SENTRY_SCRAPE_JAVASCRIPT_CONTEXT = False in your sentry.conf.py or per project in the UI. Other than that, there’s not much else we can do here since the feature is effectively working as intended.

Thursday, July 25, 2019 Read

Python Telegram Bot

Part 1: Build a Telegram Bot Step by Step Part 2: Add NLP to the Chatbot Part 3: Add New Skills to the Chatbot Python Telegram Bot Tutorial Write a Simple Telegram Bot with Python Code snippets

Thursday, July 11, 2019 Read

Fixing Disk Space Not Freed on Linux

Fixing Disk Space Not Freed on Linux Use df -ah and du -h --max-depth=1 The total from du is far smaller than the total reported by df. When a process deletes files but keeps running, the files are not actually removed, so disk space is not freed, and those files are not counted. lsof |grep delete

Wednesday, July 10, 2019 Read

Sentry Source Code Development Notes

Sentry Source Code Development Notes Formatting uWSGI requests logs PostgreSQL: Documentation: 9.3: The Password File

Wednesday, July 3, 2019 Read

OpenResty + Redis: Block High-Frequency IPs

OpenResty + Redis: Block High-Frequency IPs init_by_lua_block { redis = require "redis" client = redis.connect('127.0.0.1', 6379) } server { listen 8080; location / { access_by_lua_file /usr/local/nginx/conf/lua/block.lua; proxy_pass http://192.168.1.102:8000; } } -- Redis-based IP rate limiting / blocking for OpenResty (ngx_lua) -- NOTE: -- This script assumes a global `client` variable is used/stored. -- Make sure `redis` module is available and `client` is initialized somewhere. local function isConnected() return client:ping() end local function createRedisConnection() return redis.connect("127.0.0.1", 6379) end -- If the Redis connection fails, stop blocking (allow traffic) if pcall(isConnected) then -- already connected (or ping succeeded) else -- not connected; try reconnect if pcall(createRedisConnection) then -- Reconnect: this will reconnect Redis on every request -- For high traffic, consider disabling reconnect (skip pcall), and allow/terminate directly via ngx.exit client = createRedisConnection() else ngx.exit(ngx.OK) end end local ttl = 60 -- sampling window (seconds) local bktimes = 30 -- requests within window to trigger block local block_ttl = 600 -- block duration after trigger (seconds) local ip = ngx.var.remote_addr local ipvtimes = client:get(ip) if ipvtimes then if ipvtimes == "-1" then -- blocked return ngx.exit(403) else local last_ttl = client:ttl(ip) -- ngx.say("key exist.ttl is ", last_ttl) if last_ttl == -1 then client:set(ip, 0) client:expire(ip, ttl) -- ngx.say("ttl & vtimes recount") return ngx.exit(ngx.OK) end local vtimes = tonumber(client:get(ip)) + 1 if vtimes < bktimes then client:set(ip, vtimes) client:expire(ip, last_ttl) -- ngx.say(ip, " view ", vtimes, " times") return ngx.exit(ngx.OK) else -- ngx.say(ip, " will be block next time.") client:set(ip, -1) client:expire(ip, block_ttl) return ngx.exit(ngx.OK) end end else -- key does not exist client:set(ip, 1) -- ngx.say(ip, " view 1 times") client:expire(ip, ttl) return ngx.exit(ngx.OK) end

Thursday, June 27, 2019 Read

Terraform Getting Started Notes

Wednesday, June 26, 2019 Read

設定 Haproxy 以防止 DDOS 攻擊

設定 Haproxy 以防止 DDOS 攻擊 TCP syn flood attacks vi /etc/sysctl.conf # Protection from SYN flood net.ipv4.tcp_syncookies = 1 net.ipv4.conf.all.rp_filter = 1 net.ipv4.tcp_max_syn_backlog = 1024 Slowloris like attacks defaults option http-server-close timeout http-request 5s timeout connect 5s timeout client 30s timeout server 10s timeout tunnel 1h 限制每一個 user 的連線數量普通用戶瀏覽網站的網頁，或是從網站下載東西時，瀏覽器一般會建立 5-7 個 TCP 鏈接。當一個惡意 client 打開了大量 TCP 連線時，耗費大量資源，因此我們必須要限制同一個用戶的連線數量。但如果有很多使用者，是從某一個私有網段，透過 NAT 的方式連線到 Server 時，且實際上我們也不知道，到底哪一個會是 NAT 的轉址後的 IP，不知道該將哪個 IP 設定為白名單，這樣的限制就會造成問題，因此我們認為實際的環境，這樣的設定應該要保留不處理。以下是一個設定的範例，最重要的地方是在 frontend ft_web 區塊的設定。 global stats socket ./haproxy.stats level admin defaults option http-server-close mode http timeout http-request 5s timeout connect 5s timeout server 10s timeout client 30s listen stats bind 0.0.0.0:8880 stats enable stats hide-version stats uri / stats realm HAProxy Statistics stats auth admin:admin frontend ft_web bind 0.0.0.0:8080 # Table definition stick-table type ip size 100k expire 30s store conn_cur # Allow clean known IPs to bypass the filter tcp-request connection accept if { src -f /etc/haproxy/whitelist.lst } # Shut the new connection as long as the client has already 10 opened tcp-request connection reject if { src_conn_cur ge 10 } tcp-request connection track-sc1 src # Split static and dynamic traffic since these requests have different impacts on the servers use_backend bk_web_static if { path_end .jpg .png .gif .css .js } default_backend bk_web # Dynamic part of the application backend bk_web balance roundrobin cookie MYSRV insert indirect nocache server srv1 192.168.1.2:80 check cookie srv1 maxconn 100 server srv2 192.168.1.3:80 check cookie srv2 maxconn 100 # Static objects backend bk_web_static balance roundrobin server srv1 192.168.1.2:80 check maxconn 1000 server srv2 192.168.1.3:80 check maxconn 1000 限制每個 user 產生新連線的速率 Limiting the connection rate per user 惡意的使用者會在短時間內建立很多連線，但如果產生新連線的速度太高，就會消耗掉過多的資源服務一個使用者。

Monday, June 17, 2019 Read

Kubernetes Runtime Explained in Plain Language

Thursday, June 13, 2019 Read

Do you understand the Nginx request processing flow?

Do you understand the Nginx request processing flow? 11 processing phases NGX_HTTP_POST_READ_PHASE: A phase after receiving the full HTTP headers. It is before URI rewrite. Very few modules register in this phase, and it is skipped by default. NGX_HTTP_SERVER_REWRITE_PHASE: The phase that modifies the URI before matching the location, used for redirects. This is where rewrite directives in the server block but outside location are executed. While reading request headers, nginx selects the virtual host by host and port.

Thursday, March 7, 2019 Read