What is systemd? Why use it? how does it work? - breizhcamp

What is systemd?
Why use it?
How does it work?
By @waxzce & @clementd
At BREIZHCAMP 2017

Manage process lifecycle
● Start/Stop
● Reap Orphan processes
● Launch Order
● Monitoring
● ….

Example: gitlab initd bash script (~500 lines)
#! /bin/sh
# GITLAB
# Maintainer: @randx
# Authors: rovanion.luckey@gmail.com, @randx
### BEGIN INIT INFO
# Provides: gitlab
# Required-Start: $local_fs $remote_fs $network $syslog redis-server
# Required-Stop: $local_fs $remote_fs $network $syslog
# Default-Start: 2 3 4 5
# Default-Stop: 0 1 6
# Short-Description: GitLab git repository management
# Description: GitLab git repository management
# chkconfig: - 85 14
### END INIT INFO
###
# DO NOT EDIT THIS FILE!
# This file will be overwritten on update.
# Instead add/change your variables in /etc/default/gitlab
# An example defaults file can be found in lib/support/init.d/gitlab.default.example
###
### Environment variables
RAILS_ENV="production"
# Script variable names should be lower-case not to conflict with
# internal /bin/sh variables such as PATH, EDITOR or SHELL.
app_user="git"
app_root="/home/$app_user/gitlab"
pid_path="$app_root/tmp/pids"
socket_path="$app_root/tmp/sockets"
rails_socket="$socket_path/gitlab.socket"
web_server_pid_path="$pid_path/unicorn.pid"

sidekiq_pid_path="$pid_path/sidekiq.pid"
mail_room_enabled=false
mail_room_pid_path="$pid_path/mail_room.pid"
gitlab_workhorse_dir=$(cd $app_root/../gitlab-workhorse 2> /dev/null && pwd)
gitlab_workhorse_pid_path="$pid_path/gitlab-workhorse.pid"
gitlab_workhorse_options="-listenUmask 0 -listenNetwork unix -listenAddr $socket_path/gitlab-workhorse.socket -authBackend http://127.0.0.1:8080 -authSocket $rails_socket
-documentRoot $app_root/public"
gitlab_workhorse_log="$app_root/log/gitlab-workhorse.log"
gitlab_pages_enabled=false
gitlab_pages_dir=$(cd $app_root/../gitlab-pages 2> /dev/null && pwd)
gitlab_pages_pid_path="$pid_path/gitlab-pages.pid"
gitlab_pages_options="-pages-domain example.com -pages-root $app_root/shared/pages -listen-proxy 127.0.0.1:8090"
gitlab_pages_log="$app_root/log/gitlab-pages.log"
shell_path="/bin/bash"
gitaly_enabled=false
gitaly_dir=$(cd $app_root/../gitaly 2> /dev/null && pwd)
gitaly_pid_path="$pid_path/gitaly.pid"
gitaly_log="$app_root/log/gitaly.log"
# Read configuration variable file if it is present
test -f /etc/default/gitlab && . /etc/default/gitlab
# Switch to the app_user if it is not he/she who is running the script.
if [ `whoami` != "$app_user" ]; then
eval su - "$app_user" -c $(echo ")$shell_path -l -c '$0 "$@"'$(echo "); exit;
fi
# Switch to the gitlab path, exit on failure.
if ! cd "$app_root" ; then
echo "Failed to cd into $app_root, exiting!"; exit 1
fi
### Init Script functions
## Gets the pids from the files
check_pids(){

if ! mkdir -p "$pid_path"; then
echo "Could not create the path $pid_path needed to store the pids."
exit 1
fi
# If there exists a file which should hold the value of the Unicorn pid: read it.
if [ -f "$web_server_pid_path" ]; then
wpid=$(cat "$web_server_pid_path")
else
wpid=0
fi
if [ -f "$sidekiq_pid_path" ]; then
spid=$(cat "$sidekiq_pid_path")
else
spid=0
fi
if [ -f "$gitlab_workhorse_pid_path" ]; then
hpid=$(cat "$gitlab_workhorse_pid_path")
else
hpid=0
fi
if [ "$mail_room_enabled" = true ]; then
if [ -f "$mail_room_pid_path" ]; then
mpid=$(cat "$mail_room_pid_path")
else
mpid=0
fi
fi
if [ "$gitlab_pages_enabled" = true ]; then
if [ -f "$gitlab_pages_pid_path" ]; then
gppid=$(cat "$gitlab_pages_pid_path")
else
gppid=0
fi
fi
if [ "$gitaly_enabled" = true ]; then
if [ -f "$gitaly_pid_path" ]; then
gapid=$(cat "$gitaly_pid_path")

else
gapid=0
fi
fi
}
## Called when we have started the two processes and are waiting for their pid files.
wait_for_pids(){
# We are sleeping a bit here mostly because sidekiq is slow at writing its pid
i=0;
while [ ! -f $web_server_pid_path ] || [ ! -f $sidekiq_pid_path ] || [ ! -f $gitlab_workhorse_pid_path ] || { [ "$mail_room_enabled" = true ] && [ ! -f $mail_room_pid_path ]; } ||
{ [ "$gitlab_pages_enabled" = true ] && [ ! -f $gitlab_pages_pid_path ]; } || { [ "$gitaly_enabled" = true ] && [ ! -f $gitaly_pid_path ]; }; do
sleep 0.1;
i=$((i+1))
if [ $((i%10)) = 0 ]; then
echo -n "."
elif [ $((i)) = 301 ]; then
echo "Waited 30s for the processes to write their pids, something probably went wrong."
exit 1;
fi
done
echo
}
# We use the pids in so many parts of the script it makes sense to always check them.
# Only after start() is run should the pids change. Sidekiq sets its own pid.
check_pids
## Checks whether the different parts of the service are already running or not.
check_status(){
check_pids
# If the web server is running kill -0 $wpid returns true, or rather 0.
# Checks of *_status should only check for == 0 or != 0, never anything else.
if [ $wpid -ne 0 ]; then
kill -0 "$wpid" 2>/dev/null
web_status="$?"

else
web_status="-1"
fi
if [ $spid -ne 0 ]; then
kill -0 "$spid" 2>/dev/null
sidekiq_status="$?"
else
sidekiq_status="-1"
fi
if [ $hpid -ne 0 ]; then
kill -0 "$hpid" 2>/dev/null
gitlab_workhorse_status="$?"
else
gitlab_workhorse_status="-1"
fi
if [ $mpid -ne 0 ]; then
kill -0 "$mpid" 2>/dev/null
mail_room_status="$?"
else
mail_room_status="-1"
fi
fi
if [ $gppid -ne 0 ]; then
kill -0 "$gppid" 2>/dev/null
gitlab_pages_status="$?"
else
gitlab_pages_status="-1"
fi
fi
if [ $gapid -ne 0 ]; then
kill -0 "$gapid" 2>/dev/null
gitaly_status="$?"
else
gitaly_status="-1"

fi
fi
if [ $web_status = 0 ] && [ $sidekiq_status = 0 ] && [ $gitlab_workhorse_status = 0 ] && { [ "$mail_room_enabled" != true ] || [ $mail_room_status = 0 ]; } && { [
"$gitlab_pages_enabled" != true ] || [ $gitlab_pages_status = 0 ]; } && { [ "$gitaly_enabled" != true ] || [ $gitaly_status = 0 ]; }; then
gitlab_status=0
else
# http://refspecs.linuxbase.org/LSB_4.1.0/LSB-Core-generic/LSB-Core-generic/iniscrptact.html
# code 3 means 'program is not running'
gitlab_status=3
fi
}
## Check for stale pids and remove them if necessary.
check_stale_pids(){
check_status
# If there is a pid it is something else than 0, the service is running if
# *_status is == 0.
if [ "$wpid" != "0" ] && [ "$web_status" != "0" ]; then
echo "Removing stale Unicorn web server pid. This is most likely caused by the web server crashing the last time it ran."
if ! rm "$web_server_pid_path"; then
echo "Unable to remove stale pid, exiting."
exit 1
fi
fi
if [ "$spid" != "0" ] && [ "$sidekiq_status" != "0" ]; then
echo "Removing stale Sidekiq job dispatcher pid. This is most likely caused by Sidekiq crashing the last time it ran."
if ! rm "$sidekiq_pid_path"; then
echo "Unable to remove stale pid, exiting"
exit 1
fi
fi
if [ "$hpid" != "0" ] && [ "$gitlab_workhorse_status" != "0" ]; then
echo "Removing stale GitLab Workhorse pid. This is most likely caused by GitLab Workhorse crashing the last time it ran."
if ! rm "$gitlab_workhorse_pid_path"; then
exit 1
fi

fi
if [ "$mail_room_enabled" = true ] && [ "$mpid" != "0" ] && [ "$mail_room_status" != "0" ]; then
echo "Removing stale MailRoom job dispatcher pid. This is most likely caused by MailRoom crashing the last time it ran."
if ! rm "$mail_room_pid_path"; then
exit 1
fi
fi
if [ "$gitlab_pages_enabled" = true ] && [ "$gppid" != "0" ] && [ "$gitlab_pages_status" != "0" ]; then
echo "Removing stale GitLab Pages job dispatcher pid. This is most likely caused by GitLab Pages crashing the last time it ran."
if ! rm "$gitlab_pages_pid_path"; then
exit 1
fi
fi
if [ "$gitaly_enabled" = true ] && [ "$gapid" != "0" ] && [ "$gitaly_status" != "0" ]; then
echo "Removing stale Gitaly pid. This is most likely caused by Gitaly crashing the last time it ran."
if ! rm "$gitaly_pid_path"; then
exit 1
fi
fi
}
## If no parts of the service is running, bail out.
exit_if_not_running(){
check_stale_pids
if [ "$web_status" != "0" ] && [ "$sidekiq_status" != "0" ] && [ "$gitlab_workhorse_status" != "0" ] && { [ "$mail_room_enabled" != true ] || [ "$mail_room_status" != "0" ]; } && {
[ "$gitlab_pages_enabled" != true ] || [ "$gitlab_pages_status" != "0" ]; } && { [ "$gitaly_enabled" != true ] || [ "$gitaly_status" != "0" ]; }; then
echo "GitLab is not running."
exit
fi
}
## Starts Unicorn and Sidekiq if they're not running.
start_gitlab() {
check_stale_pids

if [ "$web_status" != "0" ]; then
echo "Starting GitLab Unicorn"
fi
if [ "$sidekiq_status" != "0" ]; then
echo "Starting GitLab Sidekiq"
fi
if [ "$gitlab_workhorse_status" != "0" ]; then
echo "Starting GitLab Workhorse"
fi
if [ "$mail_room_enabled" = true ] && [ "$mail_room_status" != "0" ]; then
echo "Starting GitLab MailRoom"
fi
if [ "$gitlab_pages_enabled" = true ] && [ "$gitlab_pages_status" != "0" ]; then
echo "Starting GitLab Pages"
fi
if [ "$gitaly_enabled" = true ] && [ "$gitaly_status" != "0" ]; then
echo "Starting Gitaly"
fi
# Then check if the service is running. If it is: don't start again.
if [ "$web_status" = "0" ]; then
echo "The Unicorn web server already running with pid $wpid, not restarting."
else
# Remove old socket if it exists
rm -f "$rails_socket" 2>/dev/null
# Start the web server
RAILS_ENV=$RAILS_ENV bin/web start
fi
# If sidekiq is already running, don't start it again.
if [ "$sidekiq_status" = "0" ]; then
echo "The Sidekiq job dispatcher is already running with pid $spid, not restarting"
else
RAILS_ENV=$RAILS_ENV bin/background_jobs start &
fi
if [ "$gitlab_workhorse_status" = "0" ]; then

echo "The GitLab Workhorse is already running with pid $spid, not restarting"
else
# No need to remove a socket, gitlab-workhorse does this itself.
# Because gitlab-workhorse has multiple executables we need to fix
# the PATH.
$app_root/bin/daemon_with_pidfile $gitlab_workhorse_pid_path
/usr/bin/env PATH=$gitlab_workhorse_dir:$PATH
gitlab-workhorse $gitlab_workhorse_options
>> $gitlab_workhorse_log 2>&1 &
fi
# If MailRoom is already running, don't start it again.
if [ "$mail_room_status" = "0" ]; then
echo "The MailRoom email processor is already running with pid $mpid, not restarting"
else
RAILS_ENV=$RAILS_ENV bin/mail_room start &
fi
fi
if [ "$gitlab_pages_status" = "0" ]; then
echo "The GitLab Pages is already running with pid $spid, not restarting"
else
$app_root/bin/daemon_with_pidfile $gitlab_pages_pid_path
$gitlab_pages_dir/gitlab-pages $gitlab_pages_options
>> $gitlab_pages_log 2>&1 &
fi
fi
if [ "$gitaly_status" = "0" ]; then
echo "Gitaly is already running with pid $gapid, not restarting"
else
$app_root/bin/daemon_with_pidfile $gitaly_pid_path
$app_root/bin/with_env $gitaly_dir/env
$gitaly_dir/gitaly >> $gitaly_log 2>&1 &

fi
fi
# Wait for the pids to be planted
wait_for_pids
# Finally check the status to tell wether or not GitLab is running
print_status
}
## Asks Unicorn, Sidekiq and MailRoom if they would be so kind as to stop, if not kills them.
stop_gitlab() {
exit_if_not_running
echo "Shutting down GitLab Unicorn"
RAILS_ENV=$RAILS_ENV bin/web stop
fi
echo "Shutting down GitLab Sidekiq"
RAILS_ENV=$RAILS_ENV bin/background_jobs stop
fi
echo "Shutting down GitLab Workhorse"
kill -- $(cat $gitlab_workhorse_pid_path)
fi
if [ "$mail_room_enabled" = true ] && [ "$mail_room_status" = "0" ]; then
echo "Shutting down GitLab MailRoom"
RAILS_ENV=$RAILS_ENV bin/mail_room stop
fi
echo "Shutting down gitlab-pages"
kill -- $(cat $gitlab_pages_pid_path)
fi
echo "Shutting down Gitaly"
kill -- $(cat $gitaly_pid_path)
fi

# If something needs to be stopped, lets wait for it to stop. Never use SIGKILL in a script.
while [ "$web_status" = "0" ] || [ "$sidekiq_status" = "0" ] || [ "$gitlab_workhorse_status" = "0" ] || { [ "$mail_room_enabled" = true ] && [ "$mail_room_status" = "0" ]; } || { [
"$gitlab_pages_enabled" = true ] && [ "$gitlab_pages_status" = "0" ]; } || { [ "$gitaly_enabled" = true ] && [ "$gitaly_status" = "0" ]; }; do
sleep 1
check_status
printf "."
if [ "$web_status" != "0" ] && [ "$sidekiq_status" != "0" ] && [ "$gitlab_workhorse_status" != "0" ] && { [ "$mail_room_enabled" != true ] || [ "$mail_room_status" != "0" ]; } &&
{ [ "$gitlab_pages_enabled" != true ] || [ "$gitlab_pages_status" != "0" ]; } && { [ "$gitaly_enabled" != true ] || [ "$gitaly_status" != "0" ]; }; then
printf "n"
break
fi
done
sleep 1
# Cleaning up unused pids
rm "$web_server_pid_path" 2>/dev/null
# rm "$sidekiq_pid_path" 2>/dev/null # Sidekiq seems to be cleaning up its own pid.
rm -f "$gitlab_workhorse_pid_path"
rm "$mail_room_pid_path" 2>/dev/null
fi
rm -f "$gitlab_pages_pid_path"
rm -f "$gitaly_pid_path"
print_status
}
## Prints the status of GitLab and its components.
print_status() {
check_status
if [ "$web_status" != "0" ] && [ "$sidekiq_status" != "0" ] && [ "$gitlab_workhorse_status" != "0" ] && { [ "$mail_room_enabled" != true ] || [ "$mail_room_status" != "0" ]; } && {
[ "$gitlab_pages_enabled" != true ] || [ "$gitlab_pages_status" != "0" ]; } && { [ "$gitaly_enabled" != true ] || [ "$gitaly_status" != "0" ]; }; then
echo "GitLab is not running."
return
fi
echo "The GitLab Unicorn web server with pid $wpid is running."

else
printf "The GitLab Unicorn web server is 033[31mnot running033[0m.n"
fi
echo "The GitLab Sidekiq job dispatcher with pid $spid is running."
else
printf "The GitLab Sidekiq job dispatcher is 033[31mnot running033[0m.n"
fi
echo "The GitLab Workhorse with pid $hpid is running."
else
printf "The GitLab Workhorse is 033[31mnot running033[0m.n"
fi
if [ "$mail_room_status" = "0" ]; then
echo "The GitLab MailRoom email processor with pid $mpid is running."
else
printf "The GitLab MailRoom email processor is 033[31mnot running033[0m.n"
fi
fi
echo "The GitLab Pages with pid $mpid is running."
else
printf "The GitLab Pages is 033[31mnot running033[0m.n"
fi
fi
echo "Gitaly with pid $gapid is running."
else
printf "Gitaly is 033[31mnot running033[0m.n"
fi
fi
if [ "$web_status" = "0" ] && [ "$sidekiq_status" = "0" ] && [ "$gitlab_workhorse_status" = "0" ] && { [ "$mail_room_enabled" != true ] || [ "$mail_room_status" = "0" ]; } && { [
"$gitlab_pages_enabled" != true ] || [ "$gitlab_pages_status" = "0" ]; } && { [ "$gitaly_enabled" != true ] || [ "$gitaly_status" = "0" ]; }; then
printf "GitLab and all its components are 033[32mup and running033[0m.n"

fi
}
## Tells unicorn to reload its config and Sidekiq to restart
reload_gitlab(){
exit_if_not_running
if [ "$wpid" = "0" ];then
echo "The GitLab Unicorn Web server is not running thus its configuration can't be reloaded."
exit 1
fi
printf "Reloading GitLab Unicorn configuration... "
RAILS_ENV=$RAILS_ENV bin/web reload
echo "Done."
echo "Restarting GitLab Sidekiq since it isn't capable of reloading its config..."
RAILS_ENV=$RAILS_ENV bin/background_jobs restart
if [ "$mail_room_enabled" != true ]; then
echo "Restarting GitLab MailRoom since it isn't capable of reloading its config..."
RAILS_ENV=$RAILS_ENV bin/mail_room restart
fi
wait_for_pids
print_status
}
## Restarts Sidekiq and Unicorn.
restart_gitlab(){
check_status
if [ "$web_status" = "0" ] || [ "$sidekiq_status" = "0" ] || [ "$gitlab_workhorse" = "0" ] || { [ "$mail_room_enabled" = true ] && [ "$mail_room_status" = "0" ]; } || { [
"$gitlab_pages_enabled" = true ] && [ "$gitlab_pages_status" = "0" ]; } || { [ "$gitaly_enabled" = true ] && [ "$gitaly_status" = "0" ]; }; then
stop_gitlab
fi
start_gitlab
}

### Finally the input handling.
case "$1" in
start)
start_gitlab
;;
stop)
stop_gitlab
;;
restart)
restart_gitlab
;;
reload|force-reload)
reload_gitlab
;;
status)
print_status
exit $gitlab_status
;;
*)
echo "Usage: service gitlab {start|stop|restart|reload|status}"
exit 1
;;
esac
exit

Gitlab needs?
4 processes:
● Unicorn (ruby webserver process)
● Sidekiq (ruby async worker process)
● Mailroom (go mail management process)
● Workhorse (go, custom reverse proxy and git http management)
Depends on:
● Web server (apache, nginx…)
● Redis
● Mail server (optional)
● Database (mySQL, postgres…)
One task: backups (rake)

Gitlab needs?
● File system ready
● Network ready
● User management
● ...
4 processes:
● Unicorn (ruby webserver process)
● Sidekiq (ruby async worker process)
● Mailroom (go mail management process)
● Workhorse (go, custom reverse proxy and git http management)
Depends on:
● Web server (apache, nginx…)
● Redis
● Mail server (optional)
● Database (mySQL, postgres…)
One task: backups

Has the process A been started?
So, do we start B?

If A crashes,
do we need to restart it?
What about B?

What about the users running the process?

Using initd, process management quality
varies from bash script to bash script, as
well as bash script quality
And changing configuration requires to
write bash code

The features we need, out of the box

gitlab-unicorn.service
[Unit]
Description=GitLab Unicorn Server
Requires=redis.service
Wants=mysqld.service postgresql.service
After=redis.service mysqld.service postgresql.service
[Service]
User=git
WorkingDirectory=/home/git/gitlab
Environment=RAILS_ENV=production
SyslogIdentifier=gitlab-unicorn
PIDFile=/home/git/gitlab/tmp/pids/unicorn.pid
ExecStart=/usr/bin/bundle exec "unicorn_rails -D -c /home/git/gitlab/config/unicorn.rb -E
production"
[Install]
WantedBy=multi-user.target

gitlab-sidekiq.service
[Unit]
Description=GitLab Sidekiq Worker
Requires=redis.service
Wants=mysqld.service postgresql.service
After=redis.service mysqld.service postgresql.service
[Service]
Type=forking
User=git
SyslogIdentifier=gitlab-sidekiq
PIDFile=/home/git/gitlab/tmp/pids/sidekiq.pid
Restart=on-failure
RestartSec=5
ExecStart=/usr/bin/bundle exec "sidekiq -C config/sidekiq_queues.yml -c 5 -e production -P tmp/pids/sidekiq.pid
-d -L log/sidekiq.log >> log/sidekiq.log 2>&1"
ExecStop=/usr/bin/bundle exec "sidekiqctl stop /home/git/gitlab/tmp/pids/sidekiq.pid >>
/home/git/gitlab/log/sidekiq.log 2>&1"
[Install]

gitlab-mailroom.service
[Unit]
Description=Gitlab mailroom Worker
[Service]
User=git
SyslogIdentifier=gitlab-mailroom
ExecStart=/usr/bin/bundle exec "mail_room -q -c /home/git/gitlab/config/mail_room.yml"
Restart=on-failure
RestartSec=5
[Install]

gitlab-workhorse.service
[Unit]
Description=Gitlab Workhorse handles slow HTTP requests for Gitlab.
Requires=gitlab-unicorn.service
After=gitlab-unicorn.service
[Service]
Type=forking
User=git
SyslogIdentifier=gitlab-workhorse
PIDFile=/home/git/gitlab/tmp/pids/gitlab-workhorse.pid
ExecStart=/home/git/gitlab/bin/daemon_with_pidfile /home/git/gitlab/tmp/pids/gitlab-workhorse.pid
/home/git/gitlab-workhorse/gitlab-workhorse -listenUmask 0 -listenNetwork unix -listenAddr
/home/git/gitlab/tmp/sockets/gitlab-workhorse.socket -authBackend http://127.0.0.1:8080
-authSocket /home/git/gitlab/tmp/sockets/gitlab.socket -documentRoot /home/git/gitlab/public >>
/home/git/gitlab/log/gitlab-workhorse.log 2>&1
[Install]

Working using systemctl
start/stop/reload
status
daemon-reload
list...

Service
application
Start / stop
Start conditions
Dependencies

Target
Logical groups of units (`network.target`, `multi-user.target`, …)

Socket, path
Starts a service when a specific socket is used (socket-based activation)
Same for path-based activation

Timer
Start a service periodically

Example, gitlab backup - .timer file
[Unit]
Description=Run gitlab request backup every day
After=network.target
[Timer]
OnCalendar=daily
Persistent=true
[Install]
WantedBy=timers.target

Example, gitlab backup - .service file
[Unit]
Description=Gitlab request backup
After=network.target
[Service]
Type=oneshot
User=git
ExecStart=/usr/bin/gitlab-backup

Device, mount, automount, swap
File system mounting
Derived from `/etc/fstab`

Snapshot
System state snapshot

Slice
Control Group descriptor for resource restrictions

systemd-nspawn
Namespaced container
Virtualized file system, process tree
No resource isolation by default
Used by rkt to start containers

Work in an ephemeral snapshot
systemd-nspawn -D / -xb
Starts a container with the current state of the system, changes are removed when
quitting the container.

Run an image in a container
systemd-nspawn -M Fedora-Cloud-Base-25-1.3.x86_64.raw

systemd-run
Start transient services, on the fly resource limitations
systemd-run -p BlockIOWeight=10 updatedb

Coupon on
clever-cloud.com :
systemdevoxx
On twitter: @clementd & @waxzce

What is systemd? Why use it? how does it work? - breizhcamp

More Related Content

What's hot (20)

Similar to What is systemd? Why use it? how does it work? - breizhcamp (20)

More from Quentin Adam (20)

Recently uploaded (20)

What is systemd? Why use it? how does it work? - breizhcamp