FUSE and beyond: bridging filesystems paper by Emmanuel Dreyfus

FUSE and beyond: bridging file systems
Emmanuel Dreyfus
August 27, 2014
Abstract
This document presents the challenges porting a modern FUSE-based distributed file sys-
tem such as GlusterFS to NetBSD. This porting required implementing a FUSE interface on
NetBSD, which already had its own native file system in userland framework called PUFFS.
The first part of this paper will introduce the current situation of distributed file systems for
Unix platforms and how FUSE changed the game. In the second part, we have a presentation
of FUSE, and how it was first implemented on top of NetBSD’s PUFFS as REFUSE. The
third part focuses on the new PERFUSE implementation of FUSE on top of PUFFS. In the
last part we cover various non-FUSE related problem that were encountered when porting
GlusterFS to NetBSD.
It is assumed the reader is familiar with file systems and with Unix system design.
1 Distributed file systems beyond NFS
1.1 Introducing NFS
Distributed file systems are client-server based storage systems, where many machines can simul-
taneously access files stored on a given machine through a network protocol. The client processes
use the same programming interface for local and remote files. A more recent related term is
Network Attached Storage (NAS), which is a machine specialized in providing a distributed file
system service. It should not be confused with Storage Area Network (SAN), which exports to the
network block devices instead of file systems.
For decades now, Network File System (NFS) has been the industry-standard solution for Unix
distributed file systems. There are multiple reasons for this, but the main one being that it is
widely available and compatible among Unix systems. NFS is also easy to setup and maintain,
which helps making it an obvious choice.
We use the name NFS here for NFS up to version 3. There is also a NFSv4 protocol, which
is a complete, backward-incompatible rewrite of NFS. It is really a new different protocol, despite
the name being the same, and it is not as widely implemented as older NFS versions.
As this paper is written, NFS turned 30 years old, and it suffers from many shortcomings related
to its age:
Security NFS and security should not really be used in the same sentence. The NFS server
trusts client assertion of process User ID (UID), and file access is under control of client’s kernel.
This lack of server-side file access control means a malicious client will be able to bypass most of
the security checks. Moreover, NFS network traffic is clear text, which leaves it widely open to
eavesdropping and data alteration by an attacker. The only reasonable use of NFS with non-public
data is therefore when clients and servers are on the same physically secure local network, with all
machines under common administrative control.
There have been many attempts to improve the situation: AFS, kerberized NFS, and NFSv4
are the most notables, but none of them are easy to setup and as widely available as NFS.
1

High availability The NFS server is a Single Point of Failure (SPOF). If it goes down for
maintenance or because of system failure, or if it gets unreachable because of a network outage,
clients cannot access the files.
Some storage vendors developed solutions to this problem, where multiple NFS servers sync
with each others, and are able to perform fail-over in case of outage. The sync mechanism is
unfortunately not specified by the NFS standards, which means such solutions are proprietary and
uninteroperable.
NFSv4.1’s parallel NFS (pNFS) extension further improves the situation by introducing repli-
cation and fail-over mechanisms, but as noted earlier, it does not enjoy wide support among Unix
systems. Moreover, it requires a meta-data server which may be a SPOF, except if the implemen-
tation goes beyond the standards and makes it redundant.
Elasticity Elasticity is the ability to increase storage volume without down time. The underlying
Operating System (OS) of an NFS server can provide this feature though various mechanisms: an
exported filesystem can be resized within a given disk, or new disk can be added to a logical volume
provided though RAID or by a volume manager. In high end systems, the logical volume can be
made of remote block device exported by storage bricks within a Storage Area Network (SAN),
and more bricks can be added on the fly.
None of these elasticity mechanisms are specified by NFS. pNFS address this problem by sup-
porting an arbitrary number of storage servers the client can access, but as said earlier, pNFS lacks
widespread OS support.
1.2 Enter FUSE
In monolithic kernels such as Linux and the various Berkeley Software Distribution (BSD) derived
systems such as NetBSD, FreeBSD and OpenBSD, file systems have traditionally been implemented
within the kernel. This is a barrier, and results in the slow adoption of any new file system,
as adaptation to the differing internal programming interfaces of each target kernel is required.
Moreover, kernel programming is much more difficult than userland programming, as any mistake
can crash the whole computer instead of just the current process. Debugging tools also have many
more constraints in kernel space, which makes the code/debug cycles longer.
This situation radically changed with the introduction of File System in UserspacE (FUSE).
Once a kernel implements a FUSE interface, it is able to use file systems implemented as userland
processes. This paves the way for file systems implemented using portable code, which means a file
system can be developed once and used for many kernels, and as a result, for different operating
systems.
FUSE also makes file system development much faster, as programming errors only crash a file
system mount, leaving the operating system fully functional. Tracking bugs does not mean many
cycles of reboots as it did with file systems implemented in kernel. Moreover, debugging a userland
process is a task for which we have many efficient tools.
1.3 FUSE distributed file systems
We saw how the situation of Unix distributed file systems was stifled by a difficult development
environment. FUSE gives the developer the opportunity to innovate a lot in that area, which
somewhat reversed the situation: from a few, slow-evolving, non-portable file systems implemented
in kernels, we moved to a lot of fast-moving and portable FUSE file systems, up to the point that
we may have trouble choosing a solution to fulfill a need.
Since the different projects are fast moving targets, a study of their various merits and short-
comings will become quickly obsolete. We will therefore not attempt to do it here. We will just
note that there are currently two distinct families of distributed file systems:
2

Client/server file systems They follow the traditional NFS setup, with improvements such as
server replication, encryption, or SPOF removal.
Peer to peer (P2P) file systems Here any machine involved in the file system can store data.
Such file systems usually have no SPOF by design. They ensure data availability by replication,
but cannot guarantee anything about it: if too many peers are unreachable, access to a file may
be impossible. The decentralized nature of P2P file systems also usually makes them unable to
support locking.
In 2010 we had a look for a distributed file system with replication, no SPOF, elasticity, POSIX
semantics and locking support. Locking support ruled out interesting P2P file systems such as
Tahoe LAFS and FicusFS. Within the client/server family, we had a look at Hadoop HDFS,
Lustre, GlusterFS, XtreemFS, MooseFS, Cloudstore, Ceph, and PVFS. Most of them were ruled
out due to the presence of SPOF. A few of them were implemented as Linux kernel modules, and
were therefore not easily portable.
We chose to focus on GlusterFS, which supported all the features we were looking for, but it
is worth noting that other file systems projects must have evolved since then, and we have not
re-evaluated the different projects. We will therefore not assert here that GlusterFS is the best
choice today, it is just the target we focused on.
1.4 Porting GlusterFS to NetBSD
GlusterFS comes as a server side and a client side. GlusterFS server side heavily relies on ex-
tended attributes, an optional POSIX-specified file system feature. NetBSD’s implementation was
incomplete, and thus part of our work was improving it, as we will see in the fourth part of this
document. The biggest problem with porting the client-side was FUSE support. The second part
of this paper introduces FUSE and NetBSD’s first REFUSE implementation, and explains why it
could not support GlusterFS. The new PERFUSE implementation described in the third part of
this paper address this problem.
2 FUSE, PUFFS and REFUSE
2.1 FUSE implementation
FUSE was initially a Linux-only project, which gained enough popularity to become a de facto stan-
dard. Today it is also available in NetBSD, MacOS X, Microsoft Windows, FreeBSD, OpenBSD,
Minix, and OpenSolaris [1].
Figure 1: FUSE architecture
FUSE file systems are accessed by user processes just
like any other Unix file system: file operations are done
through system calls (open(2), read(2), write(2)...)
which send requests to the kernel. Inside the kernel, a
component known as the Virtual File System [2] (VFS)
routes the request to the appropriate in-kernel file sys-
tem. For FUSE file systems, VFS selects the FUSE kernel
stub, which is responsible for relaying requests to user-
space processes. This is done by performing remote pro-
cedure calls, in which the kernel stub includes the calling
process’ credentials so that the user-space process can
make access decisions.
FUSE remote procedure calls are implemented as
a message passing interface through the /dev/fuse
pseudo-device. File systems register themselves to the
kernel by opening /dev/fuse, and once this is done, the
kernel can send them requests to process through this
interface.
3

In userland, libfuse takes care of low level chores, so that file systems can be made simpler.
libfuse provides two sets of Application Programming Interfaces (APIs), known as the high level
FUSE API and the low level FUSE API. As usual, the former is simpler to use, while the later is
more flexible. Some file systems will prefer to take full control and bypass libfuse completely.
They directly open /dev/fuse and talk to the kernel through what we will call the FUSE kernel
API.
A FUSE file system therefore has the choice to use three different FUSE APIs, as depicted in
figure 1:
• High level API, provided by libfuse.
• Low level API, also through libfuse.
• Kernel API, through the /dev/fuse pseudo-device.
2.2 NetBSD’s PUFFS and REFUSE
NetBSD started its own file system in userland framework at a time when FUSE was not the de
facto standard. The result was the similar but incompatible Pass-to-Userspace Framework File
System (PUFFS) [3], which shares the same architectural design as FUSE: it has a kernel stub
operated by the VFS, a /dev/puffs message-passing interface, and libpuffs, the userland file
system component used to offload the low-level chores. The design is depicted in figure 2.
PUFFS and FUSE designs look very similar, hence the need to outline the differences: the
message passing protocol and the APIs are different (PUFFS has a single API, while FUSE offers
three).
PUFFS design was driven by the desire to export NetBSD’s VFS to user space as unchanged
as possible. Any additional functionality shall be provided by user-level libpuffs. The merit of
this approach is to favor faster and easier development in user space, instead of kernel hacking. As
a result, when it became obvious that FUSE reached the de facto standard status and had to be
supported by NetBSD, it was decided to implement it in userland on top of PUFFS. The fact that
PUFFS was already fully functional was another reason for this choice.
The first attempt at a NetBSD FUSE implementation on top of PUFFS was REFUSE [4], which
stands for REimplementation of FUSE. It makes the FUSE high level API available to programs
linking with librefuse, as depicted in figure 3. Unfortunately this approach is only able to support
a subset of FUSE file systems, as it fails to support both the FUSE low level and kernel APIs. This
is unfortunate since complex file systems tend to bypass libfuse and directly use the FUSE kernel
API, either by embedding their own version of libfuse, or by re-implementing it. GlusterFS is
an example of a FUSE file system using the FUSE kernel API. As a result it cannot be ported to
NetBSD using REFUSE. We therefore needed a new FUSE implementation for NetBSD.
3 A NetBSD FUSE implementation beyond REFUSE: PERFUSE
3.1 PERFUSE design
PERFUSE stands for PUFFS Enabled Relay to FUSE. It is a more complete attempt at supporting
FUSE on NetBSD. The target here is the FUSE kernel API, which means we can run libfuse on
the top of PERFUSE to provide the low level and high level FUSE APIs. We are therefore able to
offer the three FUSE APIs, and hence we gain the ability to support any FUSE file system.
It was chosen to implement PERFUSE as a userland daemon on top of PUFFS. Possible
alternatives would have been an in-kernel FUSE-to-PUFFS translation layer, or a native kernel
FUSE implementation sitting side by side with PUFFS. The pure-userland approach has the merit
of easing development a lot. On the other hand, it does not save us from dealing with kernel race
conditions (see part 3.4) and kernel resource contention (see part 4.4). Moreover, the FUSE mount
is tied with a PERFUSE daemon and will be forcibly (and possibly uncleanly) unmounted when
4

Figure 2: PUFFS architecture Figure 3: REFUSE architecture
the daemon terminates. That later point is not a real issue since any FUSE file system already
critically relies on userland daemons implementing them.
The PERFUSE daemon, called perfused(8), is both a PUFFS server linked with libpuffs,
and a FUSE client providing the FUSE kernel API through /dev/fuse, just like the Linux kernel
would.
When a process issues system call operating on a file within a userland file system, the VFS
will route it through the in-kernel PUFFS stub, which will pass it as a PUFFS request through
/dev/puffs to libpuffs and perfused(8). perfused(8) translates the PUFFS request into a
FUSE request and sends it through /dev/fuse to libfuse and the FUSE file system. The request
path is presented in figure 4. The FUSE reply will follow the same path backward to the process
that performed system call.
Figure 4: PERFUSE architecture
A first design problem quickly arose: a
userland process such as perfused(8) can-
not create a pseudo-device like /dev/fuse as
these are kernel interfaces only. The closest
thing we have to a pseudo-device is a named
pipe. syslogd(8) uses that same approach,
creating a /dev/log named pipe. But named
pipes are not kernel pseudo devices. At least
on NetBSD, open(2) cannot access them and
FUSE file systems will get EOPNOTSUPP when
calling open(2) on /dev/fuse. As a named
pipe, it must be accessed through the Unix
socket interface, using socket(2), bind(2)
and connect(2). And for similar reasons, a file system calling mount(2) on /dev/fuse will fail.
We work around that limitation by providing two drop-in replacements for open(2) and
mount(2) in libperfuse. perfuse_open(3) will properly open the /dev/fuse named pipe as
a Unix socket, and perfuse_mount(3) will send the mount request through the PUFFS inter-
face. Once those two operations are done, the FUSE file system operates on a file descriptor to
/dev/fuse and needs no additional changes related to PERFUSE: everything behaves just like if
it were talking to the FUSE in-kernel client through the file descriptor, while perfused(8) is the
real component holding the remote side of the file descriptor.
<perfuse.h> header defines open() as perfuse_open() and mount() as perfuse_mount().
Adapting a FUSE file system to PERFUSE is therefore just about linking with -lperfuse and
adding a #include <perfuse.h> in the source file where /dev/fuse is opened and mounted.
5

perfuse_open(3) calls regular open(2) when the target is not /dev/fuse, which means other
open(2) calls in the same source file will behave as expected. This change has been contributed
to and accepted by the libfuse project, which means that file systems using the FUSE low level
or high level APIs through libfuse will be able to work unmodified.
Once perfuse_open(3) had been implemented, it became quickly obvious that using the
/dev/fuse named pipe created problems when mounting multiple FUSE file systems. Therefore,
perfuse_open(3) was modified to use a pair of unnamed sockets provided by socketpair(2).
The /dev/fuse named socket only remains when using the debug feature of perfused(8), where
it is started attached to the terminal, and awaits the connection from the file system through
/dev/fuse.
Because all programs that use the FUSE kernel API (including those linked with libfuse) open
/dev/fuse, libperfuse piggybacks the creation of the perfused(8) daemon at perfuse_open(3)
call. This results in the perfused(8) daemon running with the same privileges as the file sys-
tem implementation, but that is appropriate as perfused(8) is essentially a protocol translation
library that happens to be implemented in a separate process.
3.2 VFS operation translation from PUFFS to FUSE
PUFFS (in NetBSD) and FUSE (in Linux) requests to user-space file systems are produced by a
kernel stub below the VFS interface. The VFS is similar in design on various Unix systems, but
as a kernel component, no standard covers its implementation, and therefore each system has its
own oddities.
For the VFS interface, files are seen as objects called vnodes (VFS nodes) on which methods
implemented by file systems can be called. Here are a few method examples from the NetBSD
VFS (VOP stands for VFS OPeration):
VOP_LOOKUP Operate on a directory vnode to find a child vnode by name
VOP_OPEN Open a file vnode
VOP_GETATTR Retrieve metadata (size, mode, access time...) for a vnode
VOP_READDIR Enumerate vnode directory children
VOP_RECLAIM Forget a vnode, freeing all structures associated with it.
The root directory vnode is obtained at mount time, and subsequent vnodes are found by name
using VOP_LOOKUP, or created using methods such as VOP_CREATE (for plain files), VOP_MKDIR
(directories), VOP_MKNOD (pseudo-devices), VOP_LINK (hard links), and VOP_SYMLINK (symbolic
links). Below is a pseudo-code equivalent of what happens:
mount("/gfs") => node1
node1.lookup("foo") => ENOENT
node1.lookup("bar") => node2
node2.getattr() => ino,uid,gid,mode,ctime...
node2.open() => 0
node2.read() => data
VFS method calls are kernel function calls using the in-kernel memory address of a struct vnode
as an argument. When translating this into a PUFFS request, the struct vnode address is replaced
by an opaque identifier. The same thing happens to VFS method calls translated into FUSE
requests by the Linux kernel. Most of the work perfused(8) does when translating requests is
just reading data obtained from VFS operation’s arguments (vap in code below), and creating a
FUSE packet with that data properly converted. Here is what the code looks like for file create
(from NetBSD’s src/lib/libperfuse/ops.c):
6

pm = ps->ps_new_msg(pu, opc, FUSE_CREATE, len, pcn->pcn_cred);
fci = GET_INPAYLOAD(ps, pm, fuse_create_in);
fci->flags = O_CREAT | O_TRUNC | O_RDWR;
fci->mode = vap->va_mode | VTTOIF(vap->va_type);
fci->umask = 0; /* Seems unused by libfuse */
(void)strlcpy((char*)(void *)(fci + 1), name, namelen);
Most of the time requests are straightforward to translate, but there are exceptions, such as
vnode end of life management. Here are the NetBSD (hence PUFFS) methods:
RELEASE Called when close() is called, but reads can happen afterward, for instance if the process
has other file descriptors open on the same file.
INACTIVE Called when the last kernel reference on a node is dropped, but it can be reused later.
RECLAIM Called to free the node
Linux (hence FUSE) is much simpler here:
RELEASE Called when a file is closed
FORGET Called to free the node
The differing semantics of reference counting and the vnode life cycle requires a complex imple-
mentation. This is why INACTIVE and RECLAIM methods require so much code in perfused(8)
sources, even though there is no argument to translate.
3.3 Unix socket semantics
PERFUSE’s interface with the FUSE file system is implemented as a Unix socket. The first
experiments used SOCK_STREAM sockets, but that turned to be inappropriate, as SOCK_STREAM
makes no guarantee that a packet of data sent through a single system call is received as sent: it
may be fragmented, and the last fragment may be tied with the next FUSE request.
This is very different from the Linux /dev/fuse semantics, which offers a reliable message-
passing interface. Linux /dev/fuse never produce fragmented packets. As a consequence, FUSE
file systems do not embed any logic to reassemble request fragments. We therefore attempted to
move to SOCK_DGRAM sockets, which guarantee that a sent datagram is received as-is.
Unfortunately, SOCK_DGRAM for local sockets does not offer reliable delivery on NetBSD. Perhaps
this is because of SOCK_DGRAM’s close association with UDP in IP sockets, which purpose is to
provide an unreliable service.
When a datagram is sent through a SOCK_DGRAM socket and the receiver’s buffer is full, the
datagram is dropped, rather than causing the sender to block. As a result, if the FUSE kernel stubs
sent requests faster than the user-space file system was able to process, our FUSE implementation
just dropped some operations. For calling processes, that meant random file-related system calls
that never returned.
This could be fixed by keeping track of sent requests, and retrying after a timeout, but this
could only be done for FUSE requests sent by perfused(8). The FUSE file system itself does not
expect /dev/fuse to loose its replies, and as a consequence it does not embed any logic to resend
them.
A much better approach could be to modify the NetBSD kernel to support reliable delivery
of datagrams with local SOCK_DGRAM sockets. However, the existing semantics that such sockets
do not block and instead drop datagrams under congestion are longstanding, and changing them
could have adverse consequences to other users.
7

Instead of an intrusive SOCK_DGRAM code modification, a path of least resistance was found: im-
plementing SOCK_SEQPACKET for local sockets. SOCK_SEQPACKET is a blend between SOCK_STREAM
and SOCK_DGRAM. It guarantees datagram integrity (no fragmentation, just like SOCK_DGRAM),
and reliable ordered delivery (like SOCK_STREAM). An embarrassing eight-years-old problem report
from Jesse Off [5] offered a patch to add support for SOCK_SEQPACKET. Adapting it to current
kernel code turned to be quite straightforward, and with no risk of breaking existing code, since
SOCK_SEQPACKET had never been supported by NetBSD before.
3.4 PUFFS request concurrency
Figure 5: GETATTR race condition
An easy but efficient stress test for GlusterFS
over FUSE/PERFUSE/PUFFS was building the
NetBSD tree within a replicated and distributed
GlusterFS volume. Early testing produced strange
outcomes, with a configure script complaining
about a syntax error. After some investigation, it
because clear that the problematic step was the gen-
eration of the configure script by autoconf. Com-
parison with a build in a native file system showed
that a few bytes were missing in the script generated
within a GlusterFS volume.
This was determined to be a locking problem. When in-kernel PUFFS stub receives the reply
to a GETATTR request, it will update the kernel-cached file metadata using the values sent by the
file system. If the file system sends a file size smaller that the value cached by kernel, the kernel
truncates the file and discards the data in excess.
This situation is shown in figure 5. The kernel sends a GETATTR request for a file, and then
writes data before waiting for the GETATTR reply. The write operation increases the file size in
its kernel representation, but when the GETATTR reply comes with an already outdated file size,
the kernel discards the data it just asked the file system to write. The file pointer is reset to the
original file size, and this is where our missing bytes disappeared.
This problem is PUFFS kernel bug which was exposed by perfused(8). The fix was to add a
lock on all file size operation: not only for size modification, but also for size read through GETATTR.
4 Miscellaneous challenges when porting GlusterFS
Porting GlusterFS to NetBSD meant much more than a FUSE implementation. In this part we
will have a look at NetBSD improvements and portability workarounds that have been necessary
to obtain a functional GlusterFS.
4.1 Extended attributes
GlusterFS uses extended attributes to maintain all the state about file replication and distribution.
It is not possible to run the GlusterFS server side without extended attributes.
NetBSD extended attribute support is rather weak. There is an old port from FreeBSD code
that is not enabled in the GENERIC kernel. It has a many limitations, some of which we were
able to overcome:
• It did not even build when we first looked at it (but it was easy to fix)
• It is limited to FFSv1 (which is obtained by newfs -O1)
• It required extended attribute functionality to be explicitly started using the extattrctl(8)
command. We added the UFS_EXTATTR_AUTOSTART kernel option to do that automatically.
8

• It required creating a backing store file for each attribute. The UFS_EXTATTR_AUTOCREATE
kernel option was added to get automatic backing store creation by the kernel.
• No userland program was extended-attribute aware. We modified cp(1) and mv(1) so that
they preserve extended attributes, but many programs remain to be modified (more on that
in section 4.1.2).
4.1.1 Extended attribute API
Another big problem with the extended attribute support obtained from FreeBSD is that its API
was different than the Linux API, and of course GlusterFS used the later. It could have been
possible to have a NetBSD-specific part for extended attribute in GlusterFS and use the FreeBSD
API there, but the API differences were not limited to function named and arguments. The
extended attribute namespace management was radically different.
The FreeBSD API uses an integer to denote the namespace, with reserved values for system and
user attributes. Linux uses string prefixes to attributes names, such as trusted.*, system.*,
security.*, or user.*. Obviously it is not possible to fit a random new namespace from the
Linux API to the FreeBSD API, as it would require reserving an integer value for each namespace,
a process that would quickly lead to collisions.
We therefore decided to support both APIs, and store as system attribute what we get through
the Linux API with namespaces trusted, system and security. Everything else is stored as user
attribute. We also retained the namespace part in the extended attribute name when storing an
attribute, so that we can return it as-is when the Linux API is used to retrieve it.
4.1.2 Extended attribute support in userland
Userland program awareness for extended attributes is a front where a lot of work remains to be
done. As said earlier, mv(1) and cp(1) were modified to preserve extended attributes. We have a
lot of other commands that would require similar treatment, the interesting ones being probably
backup-related tools. Implementation is however not straightforward, as file formats and protocols
may not be able to handle extended attributes.
dump(8) and restore(8) There is no standard describing the format, and no effort to provide
archive portability among different operating system. As a result, an extended attribute support
implementation just has to take care about backward compatibility.
tar(1) The original TAR format does not handle extended attributes, nor does the newer POSIX
standardized USTAR format [6]. But USTAR has room for extensions, as we will see with pax(1),
and tar(1) could therefore be enhanced to store and restore extended attributes. However, such
a change may not be desirable, as tar(1) is expected to produce standard-compliant archives.
cpio(1) POSIX documentation on CPIO [7] does not specify how extended attributes could be
stored. However, just like USTAR, the CPIO format can have extensions, and the cpio(5) man
page [8] indicates that at least Sun Solaris used them to store extended attributes. The situation
of cpio(1) is the same as tar(1). It could have extended attributes support, but it may not be
desirable as it could break expected standard compliance.
pax(1) This utility is able to read and write various variants of TAR and CPIO formats. POSIX
standard also defines the PAX interchange format [9], which is USTAR with specified usage of
some extensions. Unfortunately, extended attribute storage is not standardized, but at least the
specification of the PAX interchange format defines vendor extensions so that there should not be
portability problems: a program unaware of a given extensions should just ignore it. On NetBSD,
pax(1) is based on libarchive(3), which already has extended attribute support, which means
it should not be difficult to have pax(1) handling them.
9

rsync(1) This command is not part of NetBSD but can be installed on it. Starting with version
3.0.9, rsync(1) has a -X or –xattrs option to preserve extended attributes. The sources support
both Linux and FreeBSD extended attribute API, which means rsync(1) support for extended
attributes on NetBSD should be easy to achieve.
scp(1) As of OpenSSH 6.6, there is no extended attributes support for scp(1) and sftp(1).
MacOS X added a -E option to scp(1) in MacOS X.4 to copy extended attributes, but this is a
MacOS X specific extension. The sources are available under a BSD license [10], which means it
would not be difficult to implement a MacOS X compatible -E option on NetBSD’s scp(1). But
as Apple did not have that change merged in the upstream OpenSSH code repository maintained
by the OpenBSD team, it means importing from Apple would mean maintaining local patches for
OpenSSH.
4.1.3 Backend improvements
The extended attribute backend in FFSv1 is also an area for improvements. Storing the extended
attributes in files instead of integrating them in the file system creates a fragile situation, where
corruption can easily happen. Moreover we do not have a tool to check and repair backing store
integrity (fsck_ffs(8) is helpless here), nor do we have any way of efficiently backing up and
restoring extended attributes for a whole file system.
There are two ways of fixing the situation:
• import UFS2 support for extended attributes from FreeBSD
• integrate extended attributes in the file system just like quotas were with the QUOTA2
system.
4.2 dirname(3) thread-safety
dirname(3) and basename(3) are very useful library functions that respectively retrieve the di-
rectory part of a path, or the last (right hand) component of the path. Unfortunately their speci-
fications leave the implementation with too many choices available [11], resulting in incompatible
flavors on NetBSD and Linux:
char *basename(char *path);
char *dirname(char *path);
Since the caller cannot pass an output buffer and its associated length, three implementations
remain possible:
• basename(3) and dirname(3) could allocate the output buffer using malloc(3). This
solution is not explicitly allowed by the POSIX specification.
• dirname(3) can modify the input buffer by truncating the string, and basename(3) can
return a pointer within the input buffer. This is the Linux implementation.
• dirname(3) and basename(3) can return memory from static storage. This is the NetBSD
implementation.
The two latter approaches are explicitly allowed by the POSIX specifications, but they can hide
compatibility issues. Code written for the Linux flavor may assume dirname(3)/basename(3)
thread-safety, while the NetBSD flavor is thread unsafe because the static storage is shared by all
threads. That approach is standard compliant since POSIX.1 says basename(3) function “need
not be thread-safe”[11].
On the other hand, code written for the NetBSD flavor can assume dirname(3)/basename(3)
does not modify the input buffer, which is exactly what Linux’s dirname(3) does, while remaining
compliant with POSIX.1 as it “may modify the string pointed to by path”[11].
10

GlusterFS assumes in many places that the Linux implementation is in use, which implies
basename(3) and dirname(3) thread-safety assumption. This meant concurrency bugs in the
NetBSD port that had to be fixed. A first attempt was to introduce thread-safe basename_r(3)
and dirname_r(3) in NetBSD, but since no standard specifies those functions, moving forward in
that direction could have ended with even more compatibility issues. That led to the status quo
decision for NetBSD: basename(3) and dirname(3) will not change, and neither basename_r(3)
nor dirname_r(3) will be introduced.
The problem was solved by adding copies of GNU libc’s basename(3) and dirname(3) in
GlusterFS sources, so that Operating Systems not based on GNU libc could use it. Such a fix
makes sense: the software that assumes a specific implementation provides it when it is missing.
4.3 link(2) on symlinks
The link(2) system call is used to create hard links. That is, additional names in the file system
for the same file. Unix file systems also support soft links, or symbolic links (symlinks for short),
which are created by the symlink(2) system call. Symlinks are small files containing the path to
the target file. This differs from hard links because it is possible to distinguish the link from the
target, while hard links are really the same file with no other differences than the name.
Using link(2) on a symlink is another area where standards are too vague to avoid incompat-
ibilities. Should link(2) create a hard link on the symlink target, or on the symlink itself? In
other words, should link(2) try to to resolve the symlink before performing its duties? An earlier
POSIX standard [12] suggests symbolic link resolutions can be done, but does not mandate so. A
more recent version says that “it is implementation-defined whether link(2) follows the symbolic
link, or creates a new link to the symbolic link itself”[13].
The following test shows that Linux and BSD systems chose different behaviors there:
lrwxrwxrwx 1 manu manu 6 nov. 1 17:06 symlink -> target
$ ln symlink link
[Linux]lrwxrwxrwx 2 manu manu 6 nov. 1 17:06 link -> target
[*BSD] lrwxrwxrwx 2 manu manu 6 nov. 1 17:06 link -> symlink
Both behaviors are standard compliant, and since GlusterFS relied on linking symlinks for the
RENAME operation, that led to a bug in the NetBSD port. Fortunately a clean fix for that problem
was easily available since newer POSIX standards [13] introduce the linkat(2) system call, a
link(2) variant that features an AT_SYMLINK_FOLLOW flag argument specifying whether symbolic
links shall be resolved or not.
The bug was fixed by partially implementing linkat(2) in NetBSD 6, and by replacing all
link(2) occurrences in GlusterFS by linkat(2) with AT_SYMLINK_FOLLOW.
The implementation of linkat(2) in NetBSD 6 is only a partial one because linkat(2) has
more to offer than the AT_SYMLINK_FOLLOW flag. It is part of the new POSIX extended API
set 2, which introduces many new system calls allowing the caller to specify the file descriptor
of a directory used to resolve relative paths instead of the current working directory. NetBSD-
current (and therefore upcoming NetBSD 7) has an almost complete POSIX extended API set 2
implementation thanks to David Holland’s improvements of the namei(9) kernel path resolution
subsystem. The only missing bit left from the POSIX extended API set 2 is fexecve(2), which
was considered too dangerous security-wise.
4.4 pagedaemon has a nap
Implementing FUSE for NetBSD and running the NetBSD build inside GlusterFS as stress test
also revealed bugs in NetBSD. A good example is what happened when using a file system in
userland while available memory is low.
In such situation, a kernel thread named the pagedaemon is responsible for paging out files
loaded in memory to disk storage so that free memory can be recovered. But when using a PUFFS
11

file system, there were conditions where the pagedaemon could sleep forever in a deadlock, leading
the operating system to collapse because of memory shortage. The issue here is that paging out a
file means using the VOP_PUTPAGE VFS method, which needs to be marshaled into a message if it
is to be sent to a userland PUFFS file system. Allocating space for the message requires memory,
memory requires the pagedaemon to perform work, and here is our deadlock.
The biggest issue in such a bug is to understand the deadlock; the fix itself is quite straightfor-
ward. The kernel memory allocator takes a flag so that the caller can specify what should be done
if memory is not available right now: return a failure, or wait until memory is freed. In order to
fix the deadlock, we had to make sure PUFFS kernel code could not sleep awaiting for memory if
executed within the context of the pagedaemon. This means pagedaemon may fail to put pages
held by PUFFS vnodes to disk, but the odds are good that there are other memory reclaim oppor-
tunities in the system. Until free memory has not been recovered beyond pagedaemon’s target, it
will carry on reclaiming memory. At some point it will retry a PUFFS vnode with enough memory
available to allocate a PUFFS message, and the PUTPAGE operation on the PUFFS vnode will
succeed.
4.5 swapcontext(3) and pthreads
In order to achieve maximum performance, GlusterFS uses the concept of N tasks that map to P
threads. A task is responsible for a given operation. At some point it gets suspended because it
waits for a network reply. When the reply comes, the task should be continued, but the thread
that carried it may be busy on some other operation. Therefore there is a need to switch a thread
context to a task context that may have been started in another thread.
GlusterFS does that using the getcontext(2), makecontext(2) and swapcontext(3) system
calls, which respectively let the caller save the CPU context for the current thread, alter the
returning address in a saved CPU context, and restore a saved CPU context. This lets a thread
carry a random task, by altering the CPU context to set the appropriate return address.
As it is often the case with advanced usage, this relied on gray area of the POSIX standard
documents: when calling swapcontext(3), do some CPU registers need to be preserved over the
context switch? The standard will not tell, but the answer for GlusterFS is obviously yes: the CPU
register pointing to the thread local storage must not be altered by swapcontext(3), otherwise
we do not only alter the thread execution to run a task, but we also switch to another thread’s
context, and we do it by bypassing all the synchronization and locking mechanisms, since we end
up with two threads sharing the same context. This can mean re-entering critical code section
twice, leading to unpleasant crashes.
Indeed Linux’ swapcontext(3) preserves the CPU register devoted to thread local storage.
NetBSD’s swapcontext(3) altered all CPU registers, which was a problem, which the pseudo-
code below exhibits:
[lwp1] getcontext(&nctx);
[lwp2] makecontext(&nctx, (void *)*swapfunc, 0);
[lwp2] swapcontext(&octx, &nctx);
[lwp2] printf("lwp2 self = %pn", pthread_self());
[lwp1] printf("lwp1 self = %pn", pthread_self());
lwp2 self = 0xbfa00000
lwp1 self = 0xbfa00000
Fortunately NetBSD had a _UC_TLSBASE flag that could be set in a saved context to tell
swapcontext(3) to preserve thread context, but that flag was not implemented on all NetBSD
ports. After heated discussions and a decision by NetBSD technical “core” committee, it was
decided to implement the flag on all NetBSD ports, and make it the default when linking with
libpthread. That way threaded applications will use swapcontext(3) without wrecking the
thread context, and non-threaded applications see no CPU register protected against alteration by
swapcontext(3).
12

Conclusions
Porting GlusterFS to NetBSD involved implementing FUSE, which was an expected requirement.
However, it also uncovered many bugs, whether inside NetBSD’s PUFFS file system in userland
framework, or in areas unrelated to file systems, such as context switching in threads, and link(2)
on symlink semantics. It pushed the implementation of many new features such as SOCK_SEQPACKET
sockets, POSIX extended API set 2, and extended attributes (in the later case the code was already
available but broken to the point it was not functional anymore).
All the code improvements described in this paper were integrated in the involved projects’
source code repositories: NetBSD (using 2-clause BSD license), GlusterFS (using LGPLv3/GPLv2
license), and FUSE (using GPLv2 license).
There still are lot of areas that should be improved:
• Extended attribute awareness for userland programs such as dump(8), restore(8)
• Extended attribute storage within FFS file system, à la QUOTA2
• Extended attribute for non-FFS file systems
• Supporting FUSE kernel negative caching for nonexistent files
• Supporting FUSE kernel notifications to the file system
• CUSE (Character device in USEr space) support
NetBSD FUSE performance should also be benchmarked. The implementation as a user level
bridge between FUSE and PUFFS means each FUSE request has to cross the kernel interface twice
more than if FUSE was natively implemented in the kernel. While this may have implication for
local FUSE file systems, it is probably of little importance for network file systems like GlusterFS,
where most of the latency comes from the network. The impact still has to be measured though.
Acknowledgments
Thanks to Jean-Yves Migeon, Christos Zoulas, Greg Oster, Mateusz Kocielski, Thomas Klausner,
Greg Oster (again), Jeremy C. Reed, Greg Troxel, Hubert Feyrer, and Christoph Badura for
reviewing this paper, and to Florence Henry for her help with LATEX.
Bibliography
[1] Filesystem in Userspace – Wikipedia, the free encyclopedia
http://en.wikipedia.org/wiki/Filesystem_in_Userspace
[2] Virtual file system – Wikipedia, the free encyclopedia
http://en.wikipedia.org/wiki/Virtual_file_system
[3] puffs – Pass-to-Userspace Framework File System
http://2007.asiabsdcon.org/papers/P04-slides.pdf
[4] refuse – NetBSD Manual Pages
http://man.netbsd.org/cgi-bin/man-cgi?refuse++NetBSD-current
[5] tech-kern: SOCK_SEQPACKET UNIX domain socket patch
http://mail-index.netbsd.org/tech-kern/2003/04/14/0006.html
[6] pax / ustar Interchange Format – The Open Group Base Specifications Issue 7, IEEE Std 1003.1, 2013 Edition
http://pubs.opengroup.org/onlinepubs/9699919799/utilities/pax.html#tag_20_92_13_06
[7] pax / cpio Interchange Format – The Open Group Base Specifications Issue 7, IEEE Std 1003.1, 2013 Edition
[8] cpio – NetBSD Manual Pages
http://man.netbsd.org/cgi-bin/man-cgi?cpio+5+NetBSD-current
13

[9] pax / pax Interchange Format – The Open Group Base Specifications Issue 7, IEEE Std 1003.1, 2013 Edition
[10] scp – Apple Open Source / OpenSSH
https://www.opensource.apple.com/source/OpenSSH/OpenSSH-186/openssh/scp.c
[11] basename – The Open Group Base Specifications Issue 7 IEEE Std 1003.1, 2013 Edition
http://pubs.opengroup.org/onlinepubs/9699919799/functions/basename.html
[12] link – The Single UNIX Specification, Version 2
http://pubs.opengroup.org/onlinepubs/7908799/xsh/link.html
[13] link – The Open Group Base Specifications Issue 7 IEEE Std 1003.1, 2013 Edition
http://pubs.opengroup.org/onlinepubs/9699919799/functions/link.html
14

FUSE and beyond: bridging filesystems paper by Emmanuel Dreyfus

More Related Content

What's hot (19)

Similar to FUSE and beyond: bridging filesystems paper by Emmanuel Dreyfus (20)

More from eurobsdcon (20)

Recently uploaded (20)

FUSE and beyond: bridging filesystems paper by Emmanuel Dreyfus