Deploying NPM packages with the Nix package manager

Deploying NPM packages with the Nix package
manager
Sander van der Burg
Feb 4, 2017
Sander van der Burg Deploying NPM packages with the Nix package manager

The Nix project
Family of declarative software deployment tools:
Nix. A purely functional package manager
NixOS. Nix based GNU/Linux distribution
Hydra. Nix based continuous build and integration server
Disnix. Nix based distributed service deployment
NixOps. NixOS-based multi-cloud deployment tool

The Nix project
Non-functional properties:
Generic. Can be used with many programming languages,
component technologies, and operating systems.
Reproducible. (Almost) no impurities – if inputs are the same,
result should be the same.
Reliable. Dependency completeness, (almost) atomic
upgrades and rollbacks.
Eﬃcient. Only the required deployment activities are
executed.

NixOS conﬁguration
/etc/nixos/configuration.nix
{pkgs, ...}:
{
boot.loader.grub.device = "/dev/sda";
fileSystems = [ { mountPoint = "/"; device = "/dev/sda2"; } ];
swapDevices = [ { device = "/dev/sda1"; } ];
services = {
openssh.enable = true;
xserver = {
enable = true;
desktopManager.kde4.enable = true;
};
};
environment.systemPackages = [ pkgs.mc pkgs.firefox ];
}

NixOS configuration
nixos-rebuild switch
Nix package manager builds a complete system configuration
Includes all packages and generates all configuration files, e.g.
OpenSSH configuration
Upgrades are (almost) atomic
Components are stored safely next to each other, due to hashes
No files are automatically removed or overwritten
Users can switch to older generations of system configurations
not garbage collected yet

NixOS bootloader

The Nix project
Basis of all tools: The Nix package manager

Nix store
Main idea: store all packages
in isolation from each other:
/nix/store/rpdqxnilb0cg...
-firefox-3.5.4
Paths contain a 160-bit
cryptographic hash of all
inputs used to build the
package:
Sources
Libraries
Compilers
Build scripts
. . .
/nix/store
l9w6773m1msy...-openssh-4.6p1
bin
ssh
sbin
sshd
smkabrbibqv7...-openssl-0.9.8e
lib
libssl.so.0.9.8
c6jbqm2mc0a7...-zlib-1.2.3
lib
libz.so.1.2.3
im276akmsrhv...-glibc-2.5
lib
libc.so.6

Nix expressions
openssh.nix
{ stdenv, fetchurl, openssl, zlib }:
stdenv.mkDerivation {
name = "openssh-4.6p1";
src = fetchurl {
url = http://.../openssh-4.6p1.tar.gz;
sha256 = "0fpjlr3bfind0y94bk442x2p...";
};
buildCommand = ’’
tar xjf $src
./configure --prefix=$out --with-openssl=${openssl}
make; make install
’’;
}

Nix expressions
all-packages.nix
openssh = import ../tools/networking/openssh {
inherit fetchurl stdenv openssl zlib;
};
openssl = import ../development/libraries/openssl {
inherit fetchurl stdenv perl;
};
stdenv = ...;
openssl = ...;
zlib = ...;
perl = ...;
nix-env -f all-packages.nix -iA openssh
Produces a /nix/store/l9w6773m1msy...-openssh-4.6p1
package in the Nix store.

User environments
Users can have
different sets of
installed applications.
PATH
/nix/.../profiles
current
42
/nix/store
pp56i0a01si5...-user-env
bin
firefox
ssh
bin
ssh
rpdqxnilb0cg...-firefox-3.5.4
bin
firefox

User environments
Users can have
different sets of
nix-env operations
create new user
environments in the
store.
PATH
/nix/.../profiles
current
42
/nix/store
bin
firefox
ssh
bin
ssh
bin
firefox
aqn3wygq9jzk...-openssh-5.2p1
bin
ssh
(nix-env -u openssh)

User environments
Users can have
different sets of
nix-env operations
create new user
environments in the
store.
PATH
/nix/.../profiles
current
42
/nix/store
bin
firefox
ssh
bin
ssh
bin
firefox
bin
ssh
i3d9vh6d8ip1...-user-env
bin
ssh
firefox

User environments
Users can have
different sets of
nix-env operations
create new user
environments in the
store.
PATH
/nix/.../profiles
current
42
43
/nix/store
bin
firefox
ssh
bin
ssh
bin
firefox
bin
ssh
bin
ssh
firefox

User environments
Users can have
different sets of
nix-env operations
create new user
environments in the
store.
We can atomically
switch between them.
PATH
/nix/.../profiles
current
42
43
/nix/store
bin
firefox
ssh
bin
ssh
bin
firefox
bin
ssh
bin
ssh
firefox

User environments
Users can have
different sets of
nix-env operations
create new user
environments in the
store.
We can atomically
These are roots of the
garbage collector.
PATH
/nix/.../profiles
current
43
/nix/store
bin
firefox
ssh
bin
ssh
bin
firefox
bin
ssh
bin
ssh
firefox
(nix-env --remove-generations old)

User environments
Users can have
different sets of
nix-env operations
create new user
environments in the
store.
We can atomically
These are roots of the
garbage collector.
PATH
/nix/.../profiles
current
43
/nix/store
bin
firefox
bin
ssh
bin
ssh
firefox
(nix-collect-garbage)

Nix expressions
openssh.nix
{ stdenv, fetchurl, openssl, zlib }:
name = "openssh-4.6p1";
src = fetchurl {
url = http://.../openssh-4.6p1.tar.gz;
sha256 = "0fpjlr3bfind0y94bk442x2p...";
};
tar xjf $src
./configure --prefix=$out --with-openssl=${openssl}
make; make install
’’;
}
Nix complements existing build tools with a pure environment
You can also run other build tools in an environment, e.g.
cmake, ant, scons

Pure environments
Precautions in service of making build results bit-identical
(regardless on what machine the package has been built):
Clearing (most) environment variables or setting them to
dummy values
Modifying search environment variables to contain Nix store
paths to only speciﬁed dependencies (e.g. PATH, PYTHONPATH,
CLASSPATH, PERL5LIB)
Using designated temp folders and output folders
Making packages immutable by making their ﬁles read-only
Resetting timestamps to 1
Chroot environments/namespaces
Restricting network access

What about Node.js/NPM projects?
NPM is Node.js’ ubiquitous deployment tool.
NPM also does dependency management in addition to build
management
NPM’s dependency management conﬂicts with Nix’s
dependency management

package.json conﬁguration
{
"name" :"nijs",
"version" :"0.0.23",
"description" :"An internal DSL for the Nix package manager in JavaScript",
"repository" :{
"type" :"git",
"url" :"https://github.com/svanderburg/nijs.git"
},
"author" :"Sander van der Burg",
"license" :"MIT",
"main" :"./lib/nijs",
"dependencies" :{
"optparse" :">= 1.0.3",
"slasp": "0.0.4"
}
}
$ npm install
$ ls node modules/
optparse slasp

Dealing with a conflicting dependency manager
Solution: substitute the conflicting dependency manager by
providing the dependencies with Nix first!

Solution: substitute NPM’s dependency management
{ stdenv, fetchgit, nodejs, optparse, slasp }:
let
nodeSources = ...
in
name = "nijs-0.0.23";
src = fetchgit {
src = https://github.com/svanderburg/nijs.git;
rev = "...";
sha256 = "...";
};
buildInputs = [ nodejs ];
# Compose node_modules/ folder from the dependencies
...
# Perform the build by running npm install
# Since a node_modules/ folder with the dependencies already exists,
# NPM will not obtain them
npm --registry http://www.example.com --nodedir=${nodeSources} install
’’;
}

let
nodeSources = ...
in
name = "nijs-0.0.23";
src = fetchgit {
rev = "...";
sha256 = "...";
};
...
’’;
}
Substituting dependencies
We may be able to automatically generate a Nix expression
from a package.json ﬁle since they capture similar proper-
ties!

let
nodeSources = ...
in
name = "nijs-0.0.23";
src = fetchgit {
rev = "...";
sha256 = "...";
};;
...
’’;
}
Substituting dependencies
Appears to be straight forward! Is it really that straight
forward?

Generating Nix expressions
It is actually much more complicated!

NPM dependency classes
Dependencies:
Run-time dependencies of a package
Development dependencies:
Build-time dependencies, e.g. transpilers
Should only be installed if they have been requested
Peer dependencies:
Check whether a shared dependency does not conﬂict.
In old versions of NPM, missing peer dependencies were also
installed.
Bundled dependencies:
Statically bundled with a package. No need to install.
Optional dependencies:
Dependencies that are allowed to break.
Typically used to bundle platform-speciﬁc code.
Permitting errors (especially non-deterministic ones)
incompatible with Nix deployment model

NPM dependency classes
Dependencies:
Run-time dependencies of a package
Development dependencies:
Build-time dependencies, e.g. transpilers
Should only be installed if they have been requested
Peer dependencies:
Check whether a shared dependency does not conﬂict.
In old versions of NPM, missing peer dependencies were also
installed.
Bundled dependencies:
Statically bundled with a package. No need to install.
Optional dependencies:
Dependencies that are allowed to break.
Typically used to bundle platform-speciﬁc code.
Permitting errors (especially non-deterministic ones)
incompatible with Nix deployment model
Interpreting dependency classes
We only need to consider dependencies and development de-
pendencies (if they are requested) and substract the bundled
dependencies

Version specifiers: NPM registry
NPM version specifiers are nominal and semantically versioned:
Precise version numbers: e.g. 1.0.0, 2.1.1
Version ranges and wildcards: e.g. 1.0.x, *, >= 1.0.1
Tags: latest, beta
The above version specifiers refer to packages obtained from the
NPM registry.

Version specifiers: external packages
Local filesystem paths, e.g. /home/sander/nijs
External URLs, e.g.
http://example.com/packages/nijs-0.0.23.tgz
Git URLs, e.g.
https://github.com/svanderburg/nijs.git
GitHub, BitBucket, GitLab, e.g. github:svanderburg/nijs
Above specifiers obtain packages from other sources than the
registry. They indirectly resolve to a package with a name and
version number.

Version specifiers: translation to Nix
Nix version specifiers are exact (any point of variation is reflected
in the hash prefix):
/nix/store/rpdqxnilb0cg...-firefox-3.5.4

Version specifiers: translation to Nix
Nix version specifiers are exact (any point of variation is reflected
in the hash prefix):
/nix/store/rpdqxnilb0cg...-firefox-3.5.4
To make a translation from NPM version specifiers to exact
version specifiers:
We must snapshot a version range and pinpoint the resolved
version (version ranges are unsupported in Nix)
We must supply the output hash to make it deterministic:
src = fetchurl {
url = "https://registry.npmjs.org/nijs/-/nijs-0.0.23.tgz";
sha1 = "dbf8f4a0acafbe3b8d9b71c24cbd1d851de6c31a";
};

Private, shared and cyclic dependencies
When including a package as a CommonJS module:
./node2nix/node modules/nijs/lib/execute/index.js
var slasp = require(’slasp’);
The module loader searches recursively upwards in node modules/
sub folders for the corresponding package:
./node2nix/nijs/node_modules/slasp
./node2nix/node_modules/slasp
./node_modules/slasp

Private dependency: a package residing in a package’s
node modules/ sub folder.
Shared dependency: a package residing in a parent directory’s
node modules/ sub folder.

When installing dependencies with NPM, NPM will not install a
dependency privately if a conforming shared dependency exists.
Some undesired implications:
A version range does not always yield the latest version that
ﬁts in a version range.
Cyclic dependencies are permitted. Bad practice, as packages
are supposed to be units of reuse.

Flat-module installations
npm < 3.x installs dependencies privately by default (unless a
conforming dependency exists in any parent node modules/
folder):
webapp/
...
package.json
node_modules/
express/
...
package.json
node_modules/
accepts/
array-flatten/
content-disposition/
...
ejs/
...
package.json

Flat-module installations
npm ≥ 3.x implements ﬂat-module installations (moving packages
as high as possible in the node modules/ hierarchy):
webapp/
...
package.json
node_modules/
accepts/
array-flatten/
content-disposition/
express/
...
package.json
ejs/
...
package.json
Because no packages conﬂict, they all appear in the top level
node modules/ folder. Implications:
Shorter path names, some degree of deduplication
The order in which packages are installed matters

Simulating ﬂat-module installations
We cannot build each NPM dependency as a Nix package, with a
separate store path:
CommonJS module resolves its own symlink location. Some
relative paths may not work:
var slasp = require(’../slasp’);
Flattening the module hierarchy is imperative.
Nix packages are made immutable after they have been built.
Solution: compose the entire dependency tree statically ahead of
time in one Nix package

Replacing impure version specifiers
Some version specifiers (e.g. tags and Git URLs), trigger NPM to
always check the remote locations for changes in each npm
install run:
Replace these version specifiers by: ’*’. Downside: it
sometimes confuses NPM, in particular flat module
installations.
Better solution: create fake processes (e.g git) giving a
deterministic result. This is still an open issue.

The solution1
: node2nix
1
Sort of, but not entirely :-)

Example: using node2nix on project-level
Generating Nix expressions from a package.json ﬁle:
$ node2nix
Building the project as a Nix package:
$ nix-build -A package
./result/bin/node2nix --help
Opening a development shell:
$ nix-shell -A shell
$ node bin/node2nix.js --help

Example: using node2nix on a package set
custom-packages.json
[
"node2nix"
,"bower"
,{"nijs": "https://github.com/svanderburg/nijs.git" }
,{"grunt-cli": "1.2.0" }
]
Generating Nix expressions:
$ node2nix -i custom-packages.json
Installing an NPM package with Nix:
$ nix-env -f default.nix -iA node2nix
$ node2nix --help

Example: simulating global dependencies
Global packages do not exist in Nix build environments. This is
problematic for certain kinds of projects, such as Grunt projects,
requiring the grunt-cli to be installed globally:
package.json
{
"name": "grunt-test",
"version": "0.0.1",
"private": "true",
"devDependencies": {
"grunt": "*",
"grunt-contrib-jshint": "*",
"grunt-contrib-watch": "*"
}
}
$ npm install
$ grunt build

We can supply global packages as extra packages:
supplement.json
[
"grunt-cli"
]
And generate the expressions as follows:
$ node2nix -d -i package.json
--supplement-input supplement.json

override.nix
{ pkgs ? import <nixpkgs> {}
, system ? builtins.currentSystem
}:
let
nodePackages = import ./default.nix {
inherit pkgs system;
};
in
nodePackages // {
package = nodePackages.package.override {
postInstall = "grunt";
};
}
$ nix-build override.nix

Conclusion
I have explained Nix and NPM’s deployment concepts
I have described node2nix generating Nix expressions from
package.json conﬁgurations, making it possible to deploy
NPM packages with the Nix package manager.
Besides the package manager, it also becomes possible to
deploy entire systems using Node.js with NixOps and Disnix.

Lessons
Naming packages: a name and version number typically does
not suﬃce!
Nix uses hash codes derived from all build inputs
.NET global assembly cache: strong names
Organization: isolate dependencies
Cyclic dependencies: disallow them – packages are supposed
to be units of reuse

Related work: other generators
npm2nix: Original generation attempt done by Shea Levy.
Composes Nix store paths for each dependency and symlinks
them.
Faster, but less accurate when dealing when shared
dependencies
Does not support ﬂat module installations or cyclic
dependencies
nixfromnpm: Implementation of npm2nix’s Nix concepts in
Haskell. Can partially regenerate expressions.

Related work: NPM alternatives
yarn: parallelization for speed improvements, version locking
ied: parallelization, content-addressable store, atomicity,
maximal sharing of dependencies (some of its concepts are
heavily inspired by Nix according to its author :-) )

References
Nix project: http://nixos.org.
Nix package manager: http://nixos.org/nix.
node2nix: https://github.com/svanderburg/node2nix.

Questions

Deploying NPM packages with the Nix package manager

More Related Content

What's hot (20)

Similar to Deploying NPM packages with the Nix package manager (20)

More from Sander van der Burg (13)

Recently uploaded (20)

Deploying NPM packages with the Nix package manager