nix-processmgmt: An experimental Nix-based process manager-agnostic framework

nix-processmgmt: An experimental Nix-based
process manager-agnostic framework
Sander van der Burg
October 17, 2020
Sander van der Burg nix-processmgmt

Nix package manager: a powerful solution
Conveniently construct packages from source code and all its
required build-time dependencies
Build determinism.
Same hash code → (nearly) bit-identical build
Transparent binary deployments (by using substitutes)
Store multiple versions and variants safely next to each other
Thanks to the hash preﬁxes and the Nix store
Unprivileged user deployments
Multiple operating systems: Linux, macOS, FreeBSD, others

Nix: development environments
We can conveniently install and use all kinds of packages without
interfering with the host system’s packages.
nix-shell example
$ python --version
Python 2.7.15
$ node --version
node: command not found
$ nix-shell -p python3 nodejs
$ python --version
Python 3.8.5
$ which python
/nix/store/z65l1jqvxa58zzwwa3bvglb6asj4y8cv-python3-3.8.5/bin/python
$ node --version
v12.18.4
$ which node
/nix/store/2w6ilfh7zmbz9zqvphgxinmbn3wdqa1b-nodejs-12.18.4/bin/node

Nix package manager: deploying services?

Nix package manager: not a service manager
Nix does not manage application services/processes’ life-cycles.

Nix: service deployment integrations
There are sister projects that complement Nix with process manage-
ment:
NixOS. Generates systemd unit files to manage services.
Requires you to switch to a fully Nix-managed Linux
distribution.
nix-darwin. Generates launchd daemon configuration files.
Only works on macOS.

Nix: service deployment integrations
Nix can also be used to augment other process management solu-
tions:
Docker. Docker uses a package manager in Dockerfiles for
the construction of images.
Nix can be used as a replacement for conventional package
managers.
Nix can be used to fully build Docker images.
Not always not a compelling use case for Nix beginners →
they typically download prebuilt images from Docker Hub.
Docker is built around Linux technologies (e.g. namespaces)
and deploys Linux software

nix-processmgmt: A general solution complementing Nix
with process management

Driven by Nix and the Nix expression language
Based on simple conventions: function deﬁnitions and an
attribute set with function invocations
Similar to how packages are organized in Nixpkgs
Works with process dependencies as well: the framework
arranges the ordering, if needed
Process-manager agnostic: Integrates with sysvinit scripts,
supervisord, systemd, launchd, bsdrc scripts, cygrunsrv
Even with systems that are not qualiﬁed as process managers:
disnix, docker

Operating system agnostic: Works on NixOS, but it is not a
requirement
Conventional Linux distros, macOS, FreeBSD, Cygwin
Unprivileged user installations
A switch to disable creation of users, and changing user
permissions
No advanced concepts required, such as namespaces and
cgroups
The solution relies on conﬂict avoidance, rather than isolation

Example: a simple web application system

Packaging the webapp process (sysvinit, verbose)
{createSystemVInitScript, webapp, port ? 5000}:
createSystemVInitScript {
name = "webapp";
description = "Example web application";
environment.PORT = port;
activities = {
start = ’’
log_info_msg "Starting Example web application..."
loadproc ${webapp}/bin/webapp -D
evaluate_retval
’’;
stop = ’’
log_info_msg "Stopping Example web application..."
killproc ${webapp}/bin/webapp
evaluate_retval
’’;
restart = "$0 stop; sleep 1; $0 start";
status = "statusproc ${webapp}/bin/webapp";
};
runlevels = [ 3 4 5 ];
}

Packaging the webapp process (sysvinit, declarative)
{createSystemVInitScript, webapp, port ? 5000}:
createSystemVInitScript {
name = "webapp";
process = "${webapp}/bin/webapp";
args = [ "-D" ];
runlevels = [ 3 4 5 ];
}
We can also specify the daemon that we want to manage, instead of
the activity implementations. Most sysvinit activities (start, stop,
restart, reload, status) can be inferred.

Packaging the webapp process (systemd)
The following function composes a systemd unit instead of a
sysvinit script:
{createSystemdService, webapp, port ? 5000}:
createSystemdService {
name = "webapp";
Unit = {
Description = "Example web application";
};
Service = {
ExecStart = "${webapp}/bin/webapp";
Environment.PORT = port;
Type = "simple";
};
}
The framework contains many other process manager-
speciﬁc abstraction functions: createSupervisordProgram,
createLaunchdDaemon, createBSDRCScript etc.

Packaging the webapp process (agnostic)
Process manager-agnostic abstraction of the webapp service:
{createManagedProcess, webapp, port ? 5000}:
createManagedProcess {
name = "webapp";
daemonArgs = [ "-D" ]; # For process managers that prefer daemons
overrides = {
sysvinit.runlevels = [ 3 4 5 ];
};
}
Invokes the required target-speciﬁc abstraction function, e.g.
createSystemVInitScript, createSystemdService
overrides override/augment process manager-speciﬁc
parameters
You can treat foreground processes and daemons separately,
for optimal user experience

Instantiatable webapp processes
{createManagedProcess, webapp}:
{instanceSuffix ? "", instanceName ? "webapp${instanceSuffix}", port ? 5000}:
createManagedProcess {
name = instanceName;
inherit instanceName; # To ensure a unique PID file name
daemonArgs = [ "-D" ]; # For process managers that prefer daemons
overrides = {
sysvinit.runlevels = [ 3 4 5 ];
};
}
instanceName: ensures that the daemon command generates
unique PID ﬁle
Outer-function header: parameters that apply to all instances.
Inner-function header: instance parameters. A unique
combination ensures that multiple instances can co-exist.

Composing process instances
{ pkgs ? import <nixpkgs> {}
, stateDir ? "/var" , forceDisableUserChange ? false, processManager}:
let constructors = import ./constructors.nix {
inherit pkgs stateDir forceDisableUserChange processManager;
}; in
rec {
webapp1 = rec { # First webapp instance
port = 5000;
dnsName = "webapp1.local";
pkg = constructors.webapp {
inherit port; instanceSuffix = "1";
};
};
webapp2 = rec { # Second webapp instance
port = 5001;
dnsName = "webapp2.local";
pkg = constructors.webapp {
inherit port; instanceSuffix = "2";
};
};
...
}

Composing process instances
{ pkgs ? import <nixpkgs> {}
, stateDir ? "/var" , forceDisableUserChange ? false, processManager}:
let constructors = import ./constructors.nix {
inherit pkgs stateDir forceDisableUserChange processManager;
}; in
rec {
...
# Nginx with a config that redirects users to the appropriate webapp
# instance based on the virtual host header
nginx = {
pkg = constructors.nginxReverseProxyHostBased {
# Process dependencies used to set up redirections in nginx.conf
webapps = [ webapp1 webapp2 ];
port = 8080;
} {};
};
}

Demo: deploying the system and using it
Deploy the process instances as sysvinit scripts:
$ nixproc-sysvinit-switch processes.nix
Open the ﬁrst webapp instance (via the Nginx reverse proxy):
$ curl -H ’Host: webapp1.local’ http://localhost:8080
Open the second webapp instance (via the Nginx reverse proxy):
$ curl -H ’Host: webapp2.local’ http://localhost:8080

Demo: all kinds of process manager integrations
Deploy as systemd units (in a user session):
$ nixproc-systemd-switch --user processes.nix
Deploy as supervisord programs (stateless):
$ nixproc-supervisord-deploy-stateless processes.nix
Deploy on FreeBSD as BSD rc scripts:
$ nixproc-bsdrc-switch processes.nix
Deploy as Docker containers per service (with shared Nix store and
host networking):
$ nixproc-docker-switch processes.nix

Demo: screenshots

Other features of nix-processmgmt
Automatic creation of users and groups (createCredentials
function)
nixproc-id-assign: Automated assignment of unique IDs
for TCP/UDP ports, UIDs, GIDs to process instances
Using Disnix as a process orchestrator (works on all platforms
where Nix/Disnix is supported)

Combining nix-processmgmt with Disnix
We can use any process manager, do distributed deployment and
combine processes with non-process based services (e.g. Java web
applications in an Apache Tomcat container):

Example services packaged for nix-processmgmt
HTTP/application servers:
Apache HTTP server
Nginx
Apache Tomcat
Database:
PostgreSQL
MariaDB/MySQL
MongoDB
InﬂuxDB
Misc:
Docker
Supervisord
svnserve

Future work
Deploy containers with multiple processes (WIP)
Mutable service containers (WIP)
s6 / s6-rc backend
Work on a test strategy for services
Maybe write an RFC?

Availability
https://github.com/svanderburg/nix-processmgmt
The implementation is still a work in progress and relies on devel-
opment versions of dependencies!

References
Blog posts:
A Nix-based functional organization for managing processes,
https://sandervanderburg.blogspot.com/2019/11/
a-nix-based-functional-organization-for.html
A declarative process manager-agnostic deployment
framework based on Nix tooling,
a-declarative-process-manager-agnostic.html
Deploying container and application services with Disnix,
deploying-container-and-application.html

References
Blog posts:
Using Disnix as a simple and minimalistic dependency-based
process manager,
using-disnix-as-simple-and-minimalistic.html
Experimenting with Nix and the service management
properties of Docker,
experimenting-with-nix-and-service.html
Assigning unique IDs to services in Disnix deployment models,
assigning-unique-ids-to-services-in.html

References
E. Dolstra, The Purely Functional Software Deployment
Model, PhD thesis, Chapter 9,
https://edolstra.github.io/pubs/phd-thesis.pdf

Questions

nix-processmgmt: An experimental Nix-based process manager-agnostic framework

More Related Content

What's hot (20)

Similar to nix-processmgmt: An experimental Nix-based process manager-agnostic framework (20)

More from Sander van der Burg (18)

Recently uploaded (20)

nix-processmgmt: An experimental Nix-based process manager-agnostic framework