Bash production guide

Bash for production systems
Survival guide for quality scripts
Adrien Mahieux - Sysadmin++ (Performance Engineer)
gh: github.com/Saruspete
tw: @Saruspete
gm: adrien.mahieux@gmail.com

$(whoami)
Adrien Mahieux - @Saruspete
French pun, because I constantly grumble...
Work in highly critical environment…
Ok, maybe not that critical. No life will be lost if we reboot the
wrong server.
My prod is not your prod
If you think you can handle it, we hire !
Time^W Latency is money
You don’t have time to rewrite everything ? You certainly
have even less time when it’s broken at the worst moment.
Why this guide ?
- Because I always forgot how to do some snippets
- Sharing is caring
- Snippets and samples are easy to read / understand / copy
- Understand use-cases before using them
What to do with it ?
- It’s not a tutorial. More like examples of safe ways to write
usual patterns.
- help you do enlightened choices. If you have all the cards in
hand, you’re the best to know what choice to make
- share with your friends & coworkers.
- stop whining "shell scripts are unreadable", and rewrite
them cleanly.
- If your script is unreadable, you don’t know bash enough (or
you shouldn’t have used it at first).
2

Different shells, different features
3

Why and When to use bash ?
Define your use-case
- Commercial Support ?
Accountability and robustness
- Supported systems ?
Wide or short list, tests, versions,
duration of support...
- Why shell script ?
Other languages available
- bash != GNU != POSIX
Check support with your target
List the features you’ll need
- Required complexity ?
Shell is bad at handling complex
parsing and long texts
- Interactions with other binaries ?
Shell is the glue between processes
but parsing is tricky.
- Are shell features enough ?
May require a lot of external tools to
process content as required
4

Others shells available
5
Shell Full Name Advantages Inconvenients
bash Bourne Again Shell Full featured Not standard on Unix
ksh Korn Shell Full featured
Fast
Not standard on Linux
ash Almquist Shell Portable
Very fast
Features missing
dash Debian Almquist Shell Portable
Very fast
More features added
Features missing
sh Bourne Shell Portable Features missing

What’s the target OS ?
OS Name /bin/sh real name Bash Location Bash Version
Linux Redhat 7 bash /bin/bash 4.2
Linux Debian 9 dash /bin/bash 4.4
Busybox ash → /bin/busybox N/A N/A
OpenBSD 6.2 ksh (pdksh) /usr/local/bin/bash 4.4
FreeBSD 12 sh (ash) /usr/local/bin/bash 4.4
OSX 10.13 bash /bin/bash 3.2
Illumos ksh93 (may vary) /bin/bash 4.3
Solaris 11 sh (not posix) (/usr/bin/xpg4) /usr/bin/bash 4.3
AIX 7.2 ksh /usr/bin/bash 4.4
"/usr/bin/env bash" to the rescue (looks in $PATH).
6

A word on the shebang...
7
Bash is only one of the many "shell command interpreter". But it’s the default one
on GNU/Linux, which is the most known and deployed *nix distribution.
Consider POSIX standard if your script is simple enough. This will ease portability.
The choice of the correct shebang is depending on target usage:
#!/bin/bash
- If your script is not meant to be
re-used in an untested distribution
(eg, part of commercial product
with strict list of supported OS)
- If your script is tightly tied to the
target system, and you don’t want
to rely on the $PATH value.
#!/usr/bin/env bash
- If your script will be open-source
in some way, you should ease the
work of all contributors, whatever
their OS.
- If your script is more for users
than system, it will help

POSIX or bash ?
Features & keywords not in POSIX. Mostly coming from KSH
- Arrays and associative arrays (bash-4)
- [[ extended operator
- Replacement ${var//search/replace} or substring ${var:offset:length}
- function f { }
- declare / typeset / type
- Arithmetic for loop "for ((i=0; i<10; i++)); do … ; done"
- extended glob
- $RANDOM $BASH_*
- /dev/tcp/$host/$port
- Named pipes
- source
- Mapfile / readarray 8

Differences
[ is a binary & a built-in.
[[ is a keyword.
function f {}
f() { }
function f() {}
$0
$FUNCNAME
$LINENO
${PIPESTATUS[@]}
[ is using arguments. No smart parsing
[[ is more lisible, faster and smarter
function does not exists in POSIX
"function f ()" only works in bash
In ksh, $0 behaves differently within
“function f” and “f()”
In bash, use $FUNCNAME
The pipestatus is an array with the return
code of all commands in a pipeline. So
"$?" Is equivalent to "${PIPESTATUS[-1]} "
9

Differences - built in VS binary
$ time ps
PID TTY TIME CMD
4345 pts/16 00:00:00 ps
30198 pts/16 00:00:00 bash
real 0m0.009s
user 0m0.004s
sys 0m0.004s
$ /usr/bin/time ps
PID TTY TIME CMD
4349 pts/16 00:00:00 time
4350 pts/16 00:00:00 ps
30198 pts/16 00:00:00 bash
0.00user 0.00system 0:00.00elapsed
85%CPU (0avgtext+0avgdata
2140maxresident)k
0inputs+0outputs
(0major+110minor)pagefaults 0swaps
10

Differences - Variable visibility
11
v1="global"
v2="global"
f1 () {
v1="func"
typeset v2="func" v3="func"
} ; f1
echo "v1=$v1 v2=$v2 v3=$v3"
v1=func v2=global v3=
recurse() (
[[ $1 -le 0 ]] && return
v2="$1 $(recurse $(($1 -1))"
echo "$v2"
)
recurse 3 ; echo "$v2"
3 2 1
global
POSIX: all vars are global (even those defined
inside a function)
bash: With keyword, kept local inside a function,
even if overlap a global var.
ksh: differs between “function f” and “f()” :
- function f : local, like bash
- f() : all global, like posix
Recursion ? use function parameters, or use ( )
instead of { } to declare the function body

The magic header
Shebang (change according to your use)
All vars must be declared
All erasing stream must be explicit
Reset the lang to avoid localization
Reset the path to a secure one
Always work with absolute dirs ! Your
script can be called from anywhere.
If GNU readlink is not available, use :
"MYPATH="$(cd -- "$(dirname
-- "$0")" && pwd -P)"
Prefer “typeset” over “declare” for KSH
None will work on sh / ash / dash
#!/usr/bin/env bash
set -o nounset # or "set -u"
set -o noclobber # or "set -C"
export LC_ALL=C
export PATH="/bin:/sbin:/usr/bin:/usr/sbin:$PATH"
readonly MYSELF="$(readlink -f $0)"
readonly MYPATH="${MYSELF%/*}"
readonly MYTEMP="$MYPATH/temp/run.$$"
mkdir -p "$MYTEMP"
# If needed to cleanup
# trap '(rm -fr $MYTEMP)' TERM QUIT INT
13

The magic header - implications
All variables accessed must be defined, even
through tests. So you must provide a default
value if you want to test an unknown variable:
${varname:-defaultvalue}
With noclobber, bash will refuse to truncate a file
if you don’t explicitly ask so with >|
set -o nounset
[[ -n "${1:-}" ]]
set -o noclobber
echo "Start instance" >|
file.log
14

Variable assignment - usual rules
typeset VAR1="Can by used anywhere"
ftest () {
typeset tmpval="only used localy"
echo "$VAR1 != $tmpval"
}
UPPERCASE variables are global to the
whole script
lowercase variables are local to a
function.
This won’t change the variable visibility,
but you’ll have a hint on whether you
should use it or not.
This is a commonly accepted rule, but not
a language requirement. You should
adhere to it anyway.
15

Variable assignment - String
hostn="$(uname -n)"
strdyn="value with spaces on $hostn"
strraw='value with $special chars
/tmp/*'
strcat='The quick brown fox'
strcat+="Jumps over the lazy dog"
Always enclose interpretable strings with
double quotes !
Always enclose non-interpretable (literal)
strings with single quotes
Concatenation is also way easier with
bash "+=" syntax.
16

Variable assignment - String Quoting
hostn="$(uname -n)"
strdyn="value with spaces on $hostn"
strraw='value with $special chars
/tmp/*'
echo $strdyn
value with spaces on odin
echo $strraw
value with $special chars
/tmp/pulse-PKdhtXMmr1 /tmp/runtime-adrien
/tmp/ssh-OwoBLZD6leEN
echo "$strraw"
value with $special chars /tmp/*
Let’s take our previous strings and focus
on the quoting...
If you don’t put double quotes between
echo, it may not change the output…
But if your string has any special char, it
will hurt you… badly… because they will
get interpreted, then passed on to echo
So, ALWAYS double quotes your
variables. Especially if they are strings.
17

Variable assignment - Integer
typeset -i i=5
i="hello"
echo $i
i=0
i+=2
If you declare a variable as an int, it will
save you from mis-usage or bad values
And also ease your life with += operator
18

Variable assignment - Integer bases
typeset -i i=5
i=080
bash: 080: value too great for base
echo $((16#A))
10
echo $((36#helloworld))
1767707668033969
echo $((16#deadbeef))
3735928559
printf "%x" 3735928559
deadbeef
But beware of 0 as octal prefix... (may
happen if using “date +%j”)
You can also convert any base to base 10
As expected works back and forth
19

Variable assignment - nameref
nameref are kind of a pointer. It will avoid
you a lot of troubles with “eval”. The
nameref is assigned during the typeset.
Then, you can manipulate directly the
targeted variable with a common name.
If you modify the nameref, you’ll modify
the referenced variable (not the nameref)
If you want to change which variable is
referenced, you have to use typeset.
Same for unset: to unset the nameref (and
not the referenced var) use “-n”.
typeset foo="bar"
typeset -i num=2
typeset foo2="barbaz"
typeset -n ref="foo"
echo "The referenced var = $ref"
The referenced var = bar
ref="foo$num"
echo "And now, foo=$foo"
And now, foo=foo2
typeset -n ref="foo$num"
echo "This time, foo$num=$ref"
This time, foo2=barbaz
unset -n ref
20

String expansion
rm /tmp/$prefix.$$.*
shopt -s globstar
ls /proc/**
bash: /bin/ls: Argument list too long
sudo ls /root/.ssh/id_*
ls: cannot access /root/.ssh/id_*: No
such file or directory
Expansion (var, list, star, completion…) is
done by the shell, not the application.
If too many files matches a pattern, you’ll
have error “too many arguments”
If your shell doesn’t have access to read
files and do the expansion, you’ll have an
error, and the pattern will be provided as
is (unless you set globfail or nullglob
option)
21

String Heredoc
Heredoc is very useful to put large block
of text inside a script, like a configuration
template, a help section… It’s a stream, so
uses stdin/out, not arguments.
Why use tabs instead of space for
indenting (aside the fact it’s their primary
usage) ? Because bash can strip the tabs
on this heredoc, not spaces.
As usual, if you don’t want any expansion
to happen, use single quotes between
your separator keyword
You may also use heredoc as a cheap
environment variable and command
processing inside a configuration file for
your application (but shouldn’t)
$ cat >/tmp/toto <<EOT
Usage: $0 <required> [optional]
This is a multiline text
EOT
$ cat <<-EOT
Using "<<-", the tabs at start of this text
are stripped, so you can keep indent
EOT
$ cat << 'EOT'
This text won't be taken as a litteral.
No $variable or $(cat /etc/passwd) will expand
EOT
$ (echo 'cat << _EOT'; cat file.cfg; echo
'_EOT')|/bin/sh
22

Process - Execution
cat /proc/self/mounts
usrs="`getent passwd|awk -F: '{print $1}'`"
grps="$(getent group|awk -F: '{print $1}')"
alias ls="ls --color"
ls
cd() {
builtin cd "$@" && echo "$PWD"
}
23
To execute a cmd, just call it…
To capture its output, you can use
`cmd` or $(cmd). But for sake of
clarity, always use $(cmd) format.
Even when overridden, you can
still call the binary by prefixing
But if you want a builtin, just prefix
the command by "builtin"

Process - Background processes
tail -f /var/log/messages >/dev/null
&
disown
for i in {1..10}; do
sleep $i &
done
wait $!
wait
24
You can run cmds in background...
and forget about them (daemon like)
Or launch multiple applications in
background…
And only wait for the last one you launched
Or wait for all of them

Process - Job specification
for i in {1..9}; do
sleep 10$i &
done
jobs
[1] Running sleep 10$i &
[ . . . ]
[8]- Running sleep 10$i &
[9]+ Running sleep 10$i &
kill %5
fg %-
25
Lets start a few commands in
background.
And you can see their jobspec ID and
status using “jobs”
%X : where X is the jobspec ID
%+ : (current job) the command you act
on if you don’t specify a jobspec ID.
%- : one-before last command.
Builtin like wait, disown, kill, fg, bg…
understand the % as a jobspec id.

Process - Coprocess
Don’t.
Just. Fuckin’. Don’t.
If you need coprocess, you need another
scripting language, like python or perl.
EXCEPTION
if you are Clifford Williams level and can
produce scripts like the one pasted here.
But in that case, you won’t learn anything
new within this document
26
#!/usr/bin/env ksh93
#Author: G. Clifford Williams
#email: gcw-ksh93@notadiscussion.com
#Purpose: This is a simple producer/consumer example written in KSH93 to
# demonstrate parallel execution with discreet I/O.
#
#Note: This example script requires KSH93. It will not work with BASH, PDKSH,
# MKSH, ZSH, KSH88, etc. Further it incorporates features foun in version r+
# and later. KSH93 u+ has been around for 5 years and is the version I used
# but your mileage may vary.
typeset -a inputs=()
typeset -a outputs=()
set +o bgnice
for counter in {A..F}{0..9} ; do
#create a background job that has output on it's STDOUT at a random interval
{ while true ; do
#print -n "pump ${counter}: $(date)" >> ${my_file}
integer sleep_time=$(( $(( $RANDOM + 1 )) / 32768.0 * 5 ))
printf "pump %s/%i: %(%H:%M:%S)Tn" ${counter} ${sleep_time} now
sleep ${sleep_time}
done
}|&
# The '|&' above is how ksh launches a co-process in the background with
# bi-directional I/O. We wrap everything in a command-list ({;}) for clarity
#The bit below can seem confusing. The co-process produces output which is
#input to the consumer. We aren't sending any messages to the co-process in
#this case but if we were our output would be the input to the co-process.
#The [in,out](put) variables are named from the perspective of the parent.
exec {in}<&p #store the output fdesc in $in
inputs+=( ${in} ) #append the value in $in to the $inputs list
exec {out}>&p #store the input fdesc in $out
outputs+=( ${out} ) #append the value in $out to the $outputs list
done
while true; do #perpetual loop
for counter in ${inputs[*]}; do #iterate through the list of input fdescs
unset line_in #clear our holder variable
read -t0.1 -u${counter} line_in #timeout after .1 seconds of no input
if [[ -z "${line_in}" ]] ; then
print "${counter}: timed out"
else
print "${counter}: ${line_in}"
fi
done
done

Process - Signal handling
readonly TMPDIR="/tmp/data.$$"
mkdir -p "$TMPDIR"
trap INT QUIT TERM cleanup
trap USR1 status
cleanup () {
rm -rf "$TMPDIR"
}
status () {
echo "Current TMPDIR: $TMPDIR"
}
Use “trap” to catch signals like Ctrl+C or
“kill”.
You can list available signals with “trap
-l”, but don’t trap them if you don’t know
what they’re doing !
27

Tests - “if test1 / else” VS “test1 && { } || { }”
# if/elif/else/fi are real test keywords
# the result of the test in them are the only
# way to act on a branch. The result inside
# their block doesn’t change the validity of
# the differents branches.
if grep root /etc/passwd > /dev/null; then
[[ -e /nonexistantfile ]]
else
echo "This should not be printed"
fi
# Nothing is displayed
# { ... } is a ‘group command’.
# its last command will be the return code
# of the whole group.
# By using a || or && after, bash will act
# on this final return code, not just the
# test at the beginning.
grep root /etc/passwd > /dev/null && {
[[ -e /nonexistantfile ]]
} || {
echo "This should not be printed"
}
# Display "This should not be printed"
28

Variable tests - [ or [[
29
# In pure bash, always use [[
typeset -i foo=15
[[ $foo -gt 7 ]]
[[ $foo > 7 ]]
[[ $foo = 15 && $foo < 30 ]]
# Finding a string
haystack='spaces and * $$ >'
needle='and'
[[ "$haystack" = *$needle* ]]
# same, but case insensitive
shopt -s nocasematch
[[ "$haystack" = *$needle* ]]
# Use [ for POSIX compatibility
foo=15
[ "$foo" -gt 7 ]
[ $(($foo > 7)) -ne 0 ]
[ "$foo" = 7 ] && [ "$foo" -lt 30 ]
# You’ll need to use grep or a switch
case $haystack in
*and*) echo 'found !' ;;
esac
# Could also be done with "grep -i"
case $haystack in
*[aA][nN][dD]*) echo 'found !' ;;
esac
# Beware of [ and keywords, like ‘>’
[ "$foo" > 7 ] # will create file '7'

Variable tests
30
Test if $v ... Do Don’t
is empty [[ -z "$v" ]] [ x$v = x ]
is equal to $x [[ "$v" = "$x" ]] [ $v = $x ]
is in a list of words [[ "$v" =~ ^(foo|bar)$ ]] echo "$v"|egrep '^(foo|bar)$'
contains a pattern [[ "$v" = *"foo"* ]] echo "$v"|grep -i 'foo'

Variable default value
31
Action Format
Expands to "def" if $v is unset. Else expands to value of $v ${v-def}
Expands to "def" if $v is unset or empty. Else expands to value of $v ${v:-def}
Expands to "def" if $v is set. Else expands to nothing ${v+def}
Expands to "def" if $v is set and not empty. Else expands to nothing ${v:+def}
Set value of $v to "def" if $v is unset. Expands to the final value of $v ${v=def}
Set value of $v to "def" if $v is unset or empty. Expands to the final value of $v ${v:=def}
# You can use :+ for options mapping between options and external tools
ratio="500"
crop=""
myimagetool ${ratio:+--ratio=$ratio} ${crop:+--crop=$crop} file.png

Variable modification
32
Action Format Result
Remove ending-pattern ${v%.txt} /path/to/prefix_file
Remove ending-pattern (ungreedy) ${v%/*} /path/to
Remove starting-pattern (ungreedy) ${v#*/} path/to/prefix_file.txt
Remove starting-pattern (greedy) ${v##*/} prefix_file.txt
Replace pattern with another ${v//p??/foo} /fooh/to/foofix_file.txt
Replace starting pattern with another (greedy) ${v/#*prefix/new
}
new_file.txt
Replace ending nattern with another (greedy) ${v/%.*/.pdf} /path/to/prefix_file.pdf
Substring (starting at pos 6, for 4 chars) ${v:6:4} to/p
Substring (last 5 chars) ${v:(-5)} e.txt
v="/path/to/prefix_file.txt"

Variable modification
33
s="hello World fooBAR"
Action Format Result
Uppercase the first occurence of “o” ${s^o} hellO World fooBAR
Uppercase all chars ${s^^} HELLO WORLD FOOBAR
Uppercase all occurences of “o” ${s^^o} hellO WOrld fOOBAR
Lowercase the full string ${s,,} Hello world foobar
Invert the case of first char ${s~} Hello World fooBAR
Invert the case of all chars ${s~~} HELLO wORLD FOObar
Invert the case of letters o ${s~~o} hellO WOrld fOOBAR

Variable modification - arrays
typeset -a ifaces=( $(find /sys/class/net/ -type l)
)
echo ${ifaces[@]}
/sys/class/net/lo /sys/class/net/enp0s31f6
/sys/class/net/wwp0s20f0u6 /sys/class/net/wlp4s0
/sys/class/net/virbr0
echo ${ifaces[@]##*/}
lo enp0s31f6 wwp0s20f0u6 wlp4s0 virbr0
typeset -a opts=("hello" "world")
echo cmd ${opts[@]/#/--text=}
cmd --text=hello --text=world
34
You can act on all values
in an array too.
Decomposing it:
ifaces[@] : all values
## : Remove all matches
*/ : Pattern to remove
For all values of array
opts, replace from
begining ("/#") all empty
values (no pattern given)
by "--text=" (text after the
2nd "/")

Variable expansion - Position parameters
$@ $* "$@" "$*"
(A1)
(A2)
(A3.1)
(A3.2)
(A1)
(A2)
(A3.1)
(A3.2)
(A1)
(A2)
(A3.1 A3.2)
(A1)
(A2)
(A3.1)
(A3.2)
./test.sh "A1" "A2" "A3.1 A3.2"
for v in $@; do
echo "($v)"
done
99% of the time, you’ll want to use “$@” to
keep intact your elements with spaces
within.
Without quotes, $@ and $* have same
behaviour.
The only caveat comes with "$*" : It use the
first char of $IFS as separator (which
happens to be space by default).
35
poorManCsv() {
IFS=';'
echo "$*"
}
poorManCsv hello world "1 2 3"
Hello;world;1 2 3

Variable expansion - On array
usesOnlyArgs2and3 () {
for v in "${@:2:2}"; do
echo "($v)"
done
}
usesOnlyArgs2and3 v1 v2 "v3.1 v3.2" v4
(v2)
(v3.1 v3.2)
If you apply the substring format to
an array (${@:x:y} or ${t[@]:x:y}),
instead of doing a substring, it will
select Y elements of the array
starting from element number X
Beware: when using an array,
elements IDs starts at 0.
When using $@, $@[0] is equivalent
to $0 which is the name of the
script. So elements ID are shifted by
1.
36

Brace expansion
$ echo prefix-{x,y,z}-suffix
prefix-x-suffix prefix-y-suffix prefix-z-suffix
$ echo mv /path/to/file{,.bak}
mv /path/to/file /path/to/file.bak
$ echo mkdir -p "/home/www/"{bin,cron,html,{conf,logs,priv}/{nginx,phpfpm}}
mkdir -p /home/www/bin /home/www/cron /home/www/html
/home/www/conf/nginx /home/www/conf/phpfpm
/home/www/logs/nginx /home/www/logs/phpfpm
/home/www/priv/nginx /home/www/priv/phpfpm
# But beware, chars are expanded from their ASCII code...
$ echo {d..Z}
d c b a ` _ ^ ] [ Z Y X
37

Array - Indexed
typeset -a t
# Push (ksh way):
t[${#t[@]}]="don't"
t[${#t[@]}]="use this"
t[${#t[@]}]="syntax"
# Push (bash style)
t+=("Prefer this")
t+=("Much cleaner")
t+=("Bash syntax")
# Count:
${#t[@]}
# Iterate over values
for value in "${t[@]}"; do :; done
# Iterate over values using index
for key in "${!t[@]}"; do
typeset value="${t[$key]}"
done
# Replace over an array
t[0]="hello world"
t[1]="hi there"
for i in "${t[@]//h/z}"; do echo "$i";
done
zello zorld
hi zhere
38

Array - Associative
Same as indexed arrays but declared
using "typeset -A"
Available on bash4 and ksh88+
Automatically ordered on ksh
Can expand and filter keys with
${!pattern}
39
typeset -A t
# Like a foreach
for key in ${!t[@]}; do
val="${t[$key]}"
done
# Can also autocomplete indexes
echo "${!BASH@}"
BASH BASHOPTS BASHPID BASH_ALIASES
BASH_ARGC BASH_ARGV BASH_CMDS
BASH_COMMAND
BASH_COMPLETION_VERSINFO
BASH_LINENO BASH_SOURCE
BASH_SUBSHELL BASH_VERSINFO
BASH_VERSION
unset ${!PYTHON@}

Array - mapfile / readarray
# Read file.txt and put lines in ARR
readarray ARR < file.txt
# read only 5 lines after the 2nd line
readarray -s 2 -n 5 ARR < /etc/passwd
# There is also a callback, to be called
# every X quantum
cb () {
num="$1"
str="${2%?}"
echo "$num = $str"
}
mapfile -C 'cb' -c 1 ARR < /etc/passwd
40
Easily map a stream into an
array
It’s also easier than play with tail
+ head.
As the line ($2) contains also
the separator (n here), I’m
removing it with "${2%?}" as it’s
the last char.

Loops - iterate over elements
words="hello world foo bar"
for w in $words; do
echo "($w)"
done
typeset -a words=("hello world" "foo bar")
for w in "${words[@]}"; do
echo "($w)"
done
for file in /var/tmp/*; do
echo "==== $file: $(< $file)"
done
for i in {1..10}; do
echo "Im the $i"
done
41
For loop are used to iterate over a
list of words (based on $IFS).
If you have a word that has an $IFS
inside, it’ll be split in two. Double
quotes won’t work: “$word” will be
seen as a single element: use arrays
If you want to list files, don’t call for
“ls”, simply use patterns

Loops - while read
while IFS=: read login pass uid gid gecos home shell _canary; do
[[ -n "$_canary" ]] && {
echo "Line for user $login is malformated (canary: $_canary)"
continue
}
echo "$login - $uid - $home"
done < /etc/passwd
- The last var in the list will contain all remaining words (useful for reading the whole line with “while
read line”).
- If you need to change IFS, put it just after the "while", like an environment variable set only to “read”,
so it will not "contaminate" the loop, nor after.
- If you have a fixed number of elements, add a “canary” (_junk) at the end. If it not empty, that means
you had more fields than awaited, and some vars are potentially incorrect.
42

Loops - while read + var assignation inside
typeset -A users
awk -F: '$3>100{print $1,$5}' /etc/passwd | while read login gecos; do
users[$login]="$gecos"
done
for login in ${!users[@]}; do
echo "$login = ${users[$login]}"
done
Solution 1 : Bashism
while read login gecos; do
users[$login]="$gecos"
done < <(awk -F: '$3>100{print $1,$5}' /etc/passwd)
Solution 2 : Set hack
set +m
shopt -o lastpipe
Won’t work : pipe will execute left expression in a subshell, not propagating variables to parent shell.
43

Streams - redirection
cat file1 | sort > /tmp/$$.f1
cat file2 | sort > /tmp/$$.f2
vimdiff /tmp/$$.f1 /tmp/$$.f2
rm /tmp/$$.f1 /tmp/$$.f2
# Is equivalent to :
vimdiff <(cat file1 | sort) <(cat file2
| sort)
When using tools that requires files to
work on (eg, diff 2 content), but the
output is from commands, you need to
use temporary files.
With bash, no need to use temporary
files, it’ll provide the stdout’ fd to other
commands
44

Streams - redirection
# Prefix the stderr redirection
echo >&2 "This is an error message"
# Redirect stderr to stdout
exec 2>&1
# Or close them all before daemonizing
exec 0>&- 1>&- 2>&-
# You can create new out FD
exec 99>/tmp/debug.log
# But beware of the order!
exec 2>&1 1>/dev/null
# Is different from
Exec 1>/dev/null 2>&1
Outputs the message to stderr. Useful
for logging
You can also redirect streams while in
the script. Close them with "&-"
If you try to write to a closed fd, you’ll
have the error “Bad file descriptor”
Redirections are using dup() to duplicate
a stream, so calling order is critical !
If you send a stream to a target(2>&1)
then close the target (1>/dev/null) 2 will
still have a fd opened to original target.
45

Streams - Embedded TCP client
# No telnet ? bash can act like one...
# Open the socket as a stream
exec 5<> /dev/tcp/$server/$port
# Write something to it
echo -e "HEAD / HTTP/1.0nHost: ${server}nn"
>&5
# and read from it
cat 0<&5
# When finished, close it
exec 5>&-
Virtual folder /dev/tcp is
handled by bash.
You can use it as a standard
stream.
Then, a lot of tools can be
rewritten as pure bash, given
you implement the protocol :)
46

Debugging
# Combine the topics we saw
# A simple log to stderr
echo >&2 "Starting debugging"
# Change the debug prefix to a more useful one
export
PS4='${BASH_SOURCE}::${FUNCNAME[0]}::$LINENO)'
# Push the debug log in a dedicated file
exec 99>$0.dbg
BASH_XTRACEFD=99
# And activate debug
set -x
47

Flags to change bash behavior
4848

Shell Options - stricture
set -o nounset # set -u
set -o noclobber # set
-C
set -o noexec # set -n
set -o errexit # set -e
set -o monitor # set -m
set -o pipefail
All variables used must be defined first
Do not truncate an existing & non empty file
Do not execute any command (check syntax)
Exit on first return code != 0 (beware!)
Monitor processes
Return code will be the one of the first rightmost
failing (non-0) program in the pipeline, instead of
only the last (leftmost) called in pipeline.
49

Shell Options - debug
set -o monitor # set -m
set -o verbose # set -v
set -o xtrace # set -x
Monitor processes
Show content of code being processed
Enable cross-calls debug (execution and tests)
50

Shell Options - nullglob
$ for f in /etc/toto*; do
echo "== $f"
done
== /etc/toto*
$
$ shopt -s nullglob
$ for f in /etc/toto*; do
echo “== $f”
done
$
By default, if a pattern doesn’t match any file,
the shell will use the pattern itself as a literal
string. Then, the for loop will use this literal
string as if it matched a filed called “/etc/toto*”,
which does not exists.
By enabling nullglob, if a pattern doesn’t match
any file, it will expand to null (empty), so the
loop is never entered.
51

Shell Options - extglob / dotglob / globstar
$ shopt -s extglob
+( ) @( ) !( ) *( )
$ ls -d *config
ls: cannot access *config:
No such file or directory
$ shopt -s dotglob
$ ls -d *config
.config
.gitconfig
$ ls /proc/$$/**
/proc/
$ shopt -s globstar
$ ls /proc/$$/**
52

Shell Options - nice for interactive shell
shopt -s autocd
shopt -s dirspell
shopt -s direxpand
shopt -s histappend
set -o notify # set -b
PROMPT_COMMAND="history -a;history
-r"
53
Report the status of terminated
background job immediately, rather than
waiting for next prompt.
This will sync the history between
multiple instances (zsh-like)

Special Variables
$SECONDS
$RANDOM
$FUNCNAME
$PIPESTATUS
$PROMPT_COMMAND
54

Bonus - grep and awk useful snippets
5555

grep useful snippets
# Avoid grep in process list
ps aux | grep process | grep -v grep
# Escape some regex-related chars
echo "Hello+.+" | grep '.+'
# Grep in shell scripts
find -name '*.sh' -exec grep DOH {}
;
# What about "Useless Use Of cat" ?
cat bar.txt | grep foo
# And what about couting ?
grep ^processor /proc/cpuinfo | wc -l
56
# This one self-excludes
ps aux | grep [p]rocess
# Use -F to avoid any regex
echo "Hello+.+" | grep -F '.+'
# Grep can do recursion itself
grep -r --include '*.sh' DOH
# Just call grep
grep foo bar.txt
# Same thing : only grep
grep -c ^processor /proc/cpuinfo

awk useful snippets
# Awk is often reduced to "print
$1"..
grep foo bar.txt | awk '{print $1}'
# Instead of multiple programs
cut -d: -f1 /etc/passwd |grep -v
^root
# Operations can be complex...
typeset -i s=0
for i in $(cut -d: -f3
/etc/passwd);do
s+=i
done
echo $s
57
# But awk is self-sufficient for grep
awk '/foo/{ print $1 }' bar.txt
# simply change the separator
awk -F: '$1!="root"{print $1}'
/etc/passwd
# while summing in awk is easy too !
awk -F: '{s+=$3} END{print s}'
/etc/passwd

Tools
Shell-Lab : Code Snippets and tests against different shell interpreters
https://github.com/saruspete/shell-lab/ please contribute :-)
ShellCheck : static analysis (linter) for shell code.
https://www.shellcheck.net/ / https://github.com/koalaman/shellcheck
Checkbashims : Check your script for elements not available in posix flavour.
https://packages.debian.org/devscripts
59

Bibliography
POSIX Shell: http://pubs.opengroup.org/onlinepubs/009695399/utilities/xcu_chap02.html
POSIX.1-2017: http://pubs.opengroup.org/onlinepubs/9699919799/
Bash POSIX mode: https://www.gnu.org/software/bash/manual/html_node/Bash-POSIX-Mode.html
Bashims: http://mywiki.wooledge.org/Bashism
Bash reference sheet: http://mywiki.wooledge.org/BashSheet
POSIX Tricks: http://www.etalabs.net/sh_tricks.html
Operators: https://unix.stackexchange.com/questions/306111/what-is-the-difference-between-the-bash-operators-vs-vs-vs
ksh/bash differences: https://www.dartmouth.edu/~rc/classes/ksh/print_pages.shtml
Bash pitfalls: http://mywiki.wooledge.org/BashPitfalls
Options parsing: https://stackoverflow.com/questions/192249/how-do-i-parse-command-line-arguments-in-bash
60

Many thanks for their contribution
Benjamin Riou
Aurelien Rougement - Beorn
Graham Christensen - grhmc
G. Clifford Williams - gcw@notadiscussion.com
61

Bash production guide

More Related Content

What's hot (20)

Similar to Bash production guide (20)

Recently uploaded (20)

Bash production guide