Hero Image
What the f*ck Python! 🐍

What the f*ck Python! 🐍

Friday, October 11, 2019 Read
Hero Image
Fighting ISP Cache Hijacking Again with iptables

Fighting ISP Cache Hijacking Again with iptables Cause The fight against the carrier cache problem started two years ago. The carrier even cached cnpm data. Worse, their cache servers were not only slow like a turtle in a marathon, they also crashed frequently, so I just wanted to write code but had to face a wall of red errors. Fix iptables -I FORWARD -p tcp -m tcp -m ttl --ttl-gt 20 -m ttl --ttl-lt 30 -j DROP

Monday, October 7, 2019 Read
Hero Image
Use Nginx and mod_pagespeed to Convert Images to WebP on the Fly

Use Nginx and mod_pagespeed to Convert Images to WebP on the Fly Compile ngx_pagespeed First make sure Nginx is built with --with-compat, so we do not need to rebuild Nginx from scratch. incubator: https://github.com/apache/incubator-pagespeed-ngx.git # Switch to the nginx source directory and configure the build ./configure --with-compat --add-dynamic-module=../incubator-pagespeed-ngx # Build modules make modules # Copy the built module into the nginx modules directory sudo cp objs/ngx_pagespeed.so /etc/nginx/modules/ # Create the cache directory for converted images sudo mkdir -p /var/ngx_pagespeed_cache sudo chown -R www-data:www-data /var/ngx_pagespeed_cache load_module modules/ngx_pagespeed.so; # enable pagespeed module on this server block pagespeed on; # Needs to exist and be writable by nginx. Use tmpfs for best performance. pagespeed FileCachePath /var/ngx_pagespeed_cache; # Ensure requests for pagespeed optimized resources go to the pagespeed handler # and no extraneous headers get set. location ~ "\.pagespeed\.([a-z]\.)?[a-z]{2}\.[^.]{10}\.[^.]+" { add_header "" ""; } location ~ "^/pagespeed_static/" { } location ~ "^/ngx_pagespeed_beacon$" { } pagespeed RewriteLevel CoreFilters; The last line (pagespeed RewriteLevel CoreFilters;) specifies the enabled optimizations. It includes basic filters such as:

Monday, October 7, 2019 Read
Hero Image
Is there a regular expression to detect a valid regular expression?

Is there a regular expression to detect a valid regular expression? This is a recursive regex, and is not supported by many regex engines. PCRE based ones should support it. / ^ # start of string ( # first group start (?: (?:[^?+*{}()[\]\\|]+ # literals and ^, $ | \\. # escaped characters | \[ (?: \^?\\. | \^[^\\] | [^\\^] ) # character classes (?: [^\]\\]+ | \\. )* \] | \( (?:\?[:=!]|\?<[=!]|\?>)? (?1)?? \) # parenthesis, with recursive content | \(\? (?:R|[+-]?\d+) \) # recursive matching ) (?: (?:[?+*]|\{\d+(?:,\d*)?\}) [?+]? )? # quantifiers | \| # alternative )* # repeat content ) # end first group $ # end of string / Without whitespace and comments

Friday, October 4, 2019 Read
Hero Image
Force file download with Nginx

Force file download with Nginx add_header Content-Disposition 'attachment;'; server { listen 80; server_name my.domain.com; location ~ ^.*/(?P<request_basename>[^/]+\.(mp3))$ { root /path/to/mp3/ add_header Content-Disposition 'attachment; filename="$request_basename"'; } } { listen 80; server_name backup.baifu-tech.net; root /data/backup/rechargecent-mago; location / { auth_basic "baifu backup center"; auth_basic_user_file /etc/nginx/ssl/htpasswd; autoindex on; autoindex_exact_size off; autoindex_localtime on; } }

Monday, August 19, 2019 Read
Hero Image
Shell Script Study Notes

Shell Script Study Notes Arithmetic operations val=`expr $a + $b` Operators Symbol Description Example ! NOT [ ! false ] -o OR [ $a -lt 20 -o $b -gt 20 ] -a AND [ $a -lt 20 -a $b -gt 20 ] = equality check [ $a = $b ] != inequality check [ $a != $b ] -z string length is 0, returns true if 0 [ -z $a ] -n string length is not 0, returns true if not 0 [ -n $a ] str check whether string is empty, true if not [ $a ] -b check whether file is a block device [ -b $file ] -c check whether file is a character device .. -d check whether file is a directory [ -d $file ] -f check whether file is a regular file [ -f $file ] -r check whether file is readable .. -w check whether file is writable .. -x check whether file is executable .. -s check whether file is empty .. -e check whether file exists .. Special variables Variable Meaning $0 file name of the current script $n arguments passed to a script or function; n is the position $# number of arguments passed to a script or function $* all arguments as a single word, e.g., “1 2 3” $@ all arguments as separate words, e.g., “1” “2” “3” $? exit status of the last command or return value of a function $$ current shell process ID; for scripts, the PID of the script process POSIX program exit statuses Code Meaning 0 command exited successfully > 0 failure during redirection or word expansion (~, variables, commands, arithmetic, and word splitting) 1 - 125 command exited unsuccessfully; specific meanings are command-defined 126 command found but file is not executable 127 command not found > 128 command died from a signal Input/output redirection Command Description command > file redirect output to file command > file redirect output to file by appending n > file redirect file descriptor n to file n » file redirect file descriptor n to file by appending n >& m merge output file m and n n <& m merge input file m and n « tag use content between start tag and end tag as input File include Use . or source to include files.

Wednesday, August 7, 2019 Read
Hero Image
Excessive Errors from `sentry.lang.javascript.processor: SoftTimeLimitExceeded()`

Excessive Errors from sentry.lang.javascript.processor: SoftTimeLimitExceeded() Counters-0 queue is rising continuously Excessive Errors from sentry.lang.javascript.processor: SoftTimeLimitExceeded() mattrobenolt: Was just going to propose this. :) Unfortunately, this is just how this feature works. If it’s not useful and is just wasting resources, it’s easy to disable globally by setting SENTRY_SCRAPE_JAVASCRIPT_CONTEXT = False in your sentry.conf.py or per project in the UI. Other than that, there’s not much else we can do here since the feature is effectively working as intended.

Thursday, July 25, 2019 Read
Hero Image
Python Telegram Bot

Part 1: Build a Telegram Bot Step by Step Part 2: Add NLP to the Chatbot Part 3: Add New Skills to the Chatbot Python Telegram Bot Tutorial Write a Simple Telegram Bot with Python Code snippets

Thursday, July 11, 2019 Read
Hero Image
Fixing Disk Space Not Freed on Linux

Fixing Disk Space Not Freed on Linux Use df -ah and du -h --max-depth=1 The total from du is far smaller than the total reported by df. When a process deletes files but keeps running, the files are not actually removed, so disk space is not freed, and those files are not counted. lsof |grep delete

Wednesday, July 10, 2019 Read
Hero Image
Sentry Source Code Development Notes

Sentry Source Code Development Notes Formatting uWSGI requests logs PostgreSQL: Documentation: 9.3: The Password File

Wednesday, July 3, 2019 Read
Hero Image
OpenResty + Redis: Block High-Frequency IPs

OpenResty + Redis: Block High-Frequency IPs init_by_lua_block { redis = require "redis" client = redis.connect('127.0.0.1', 6379) } server { listen 8080; location / { access_by_lua_file /usr/local/nginx/conf/lua/block.lua; proxy_pass http://192.168.1.102:8000; } } -- Redis-based IP rate limiting / blocking for OpenResty (ngx_lua) -- NOTE: -- This script assumes a global `client` variable is used/stored. -- Make sure `redis` module is available and `client` is initialized somewhere. local function isConnected() return client:ping() end local function createRedisConnection() return redis.connect("127.0.0.1", 6379) end -- If the Redis connection fails, stop blocking (allow traffic) if pcall(isConnected) then -- already connected (or ping succeeded) else -- not connected; try reconnect if pcall(createRedisConnection) then -- Reconnect: this will reconnect Redis on every request -- For high traffic, consider disabling reconnect (skip pcall), and allow/terminate directly via ngx.exit client = createRedisConnection() else ngx.exit(ngx.OK) end end local ttl = 60 -- sampling window (seconds) local bktimes = 30 -- requests within window to trigger block local block_ttl = 600 -- block duration after trigger (seconds) local ip = ngx.var.remote_addr local ipvtimes = client:get(ip) if ipvtimes then if ipvtimes == "-1" then -- blocked return ngx.exit(403) else local last_ttl = client:ttl(ip) -- ngx.say("key exist.ttl is ", last_ttl) if last_ttl == -1 then client:set(ip, 0) client:expire(ip, ttl) -- ngx.say("ttl & vtimes recount") return ngx.exit(ngx.OK) end local vtimes = tonumber(client:get(ip)) + 1 if vtimes < bktimes then client:set(ip, vtimes) client:expire(ip, last_ttl) -- ngx.say(ip, " view ", vtimes, " times") return ngx.exit(ngx.OK) else -- ngx.say(ip, " will be block next time.") client:set(ip, -1) client:expire(ip, block_ttl) return ngx.exit(ngx.OK) end end else -- key does not exist client:set(ip, 1) -- ngx.say(ip, " view 1 times") client:expire(ip, ttl) return ngx.exit(ngx.OK) end

Thursday, June 27, 2019 Read
Hero Image
Terraform Getting Started Notes

Terraform Getting Started Notes

Wednesday, June 26, 2019 Read