ZFS Tutorial LISA 2011

ZFS: A File System for
Modern Hardware
Richard.Elling@RichardElling.com

USENIX LISA’11 Conference
USENIX LISA11 December, 2011

Agenda
• Overview
• Foundations
• Pooled Storage Layer
• Transactional Object Layer
• ZFS commands
• Sharing
• Properties
• Other goodies
• Performance
• Troubleshooting

ZFS Tutorial USENIX LISA’11 2

ZFS History
• Announced September 14, 2004
• Integration history
✦ SXCE b27 (November 2005)
✦ FreeBSD (April 2007)
✦ Mac OSX Leopard
✤ Preview shown, but removed from Snow Leopard
✤ Disappointed community reforming as the zfs-macos google group
(Oct 2009)
✦ OpenSolaris 2008.05
✦ Solaris 10 6/06 (June 2006)
✦ Linux FUSE (summer 2006)
✦ greenBytes ZFS+ (September 2008)
✦ Linux native port funded by the US DOE (2010)
• More than 45 patents, contributed to the CDDL Patents Common

ZFS Design Goals
• Figure out why storage has gotten so complicated
• Blow away 20+ years of obsolete assumptions
• Gotta replace UFS
• Design an integrated system from scratch

End the suffering


Limits

• 248 — Number of entries in any individual directory
• 256 — Number of attributes of a file [*]
• 256 — Number of files in a directory [*]
• 16 EiB (264 bytes) — Maximum size of a file system
• 16 EiB — Maximum size of a single file
• 16 EiB — Maximum size of any attribute
• 264 — Number of devices in any pool
• 264 — Number of pools in a system
• 264 — Number of file systems in a pool
• 264 — Number of snapshots of any file system
• 256 ZiB (278 bytes) — Maximum size of any pool

[*] actually constrained to 248 for the number of files in a ZFS file system


Understanding Builds
• Build is often referenced when speaking of feature/bug integration
• Short-hand notation: b###
• Distributions derived from Solaris NV (Nevada)
✦ NexentaStor
✦ Nexenta Core Platform
✦ SmartOS
✦ Solaris 11 (nee OpenSolaris)
✦ OpenIndiana
✦ StormOS
✦ BelleniX
✦ SchilliX
✦ MilaX
• OpenSolaris builds
✦ Binary builds died at b134
✦ Source releases continued through b147
• illumos stepping up to ﬁll void left by OpenSolaris’ demise

Community Links
• Community links
✦ nexenta.org
✦ nexentastor.org
✦ freebsd.org
✦ zfsonlinux.org
✦ zfs-fuse.net
✦ groups.google.com/group/zfs-macos
• ZFS Community
✦ hub.opensolaris.org/bin/view/Community+Group+zfs/
• IRC channels at irc.freenode.net
✦ #zfs


Overhead View of a Pool

Pool
File System
Configuration
Information

Volume
File System

Volume
Dataset


Hybrid Storage Pool

Adaptive Replacement Cache (ARC)

separate Main Pool
Main Pool Level 2 ARC
intent log

Write optimized HDD
HDD Read optimized
device (SSD) HDD device (SSD)

Size (GBytes) 1 - 10 GByte large big
Cost write iops/$ size/$ size/$

Use sync writes persistent storage read cache

Performance secondary
low-latency writes low-latency reads
optimization
Need more
speed? stripe more, faster devices stripe


Layer View

raw swap dump iSCSI ?? ZFS NFS CIFS ??

ZFS Volume Emulator (Zvol) ZFS POSIX Layer (ZPL) pNFS Lustre ??

Transactional Object Layer

Pooled Storage Layer

Block Device Driver

HDD SSD iSCSI ??

November 8, 2010 USENIX LISA’10 11

Source Code Structure

File system Mgmt
Device
Consumer Consumer
libzfs

Interface
ZPL ZVol /dev/zfs
Layer

Transactional ZIL ZAP Traversal
Object
Layer DMU DSL

ARC
Pooled
Storage ZIO
Layer
VDEV Configuration


Acronyms
• ARC – Adaptive Replacement Cache
• DMU – Data Management Unit
• DSL – Dataset and Snapshot Layer
• JNI – Java Native Interface
• ZPL – ZFS POSIX Layer (traditional ﬁle system interface)
• VDEV – Virtual Device
• ZAP – ZFS Attribute Processor
• ZIL – ZFS Intent Log
• ZIO – ZFS I/O layer
• Zvol – ZFS volume (raw/cooked block device interface)


NexentaStor Rosetta Stone

NexentaStor OpenSolaris/ZFS
Volume Storage pool
ZVol Volume
Folder File system


nvlists
• name=value pairs
• libnvpair(3LIB)
• Allows ZFS capabilities to change without changing the
physical on-disk format
• Data stored is XDR encoded
• A good thing, used often


Versioning
• Features can be added and identiﬁed by nvlist entries
• Change in pool or dataset versions do not change physical on-
disk format (!)
✦ does change nvlist parameters
• Older-versions can be used
✦ might see warning messages, but harmless
• Available versions and features can be easily viewed
✦ zpool upgrade -v
✦ zfs upgrade -v
• Online references (broken?)
✦ zpool: hub.opensolaris.org/bin/view/Community+Group+zfs/N
✦ zfs: hub.opensolaris.org/bin/view/Community+Group+zfs/N-1

Don't confuse zpool and zfs versions

zpool Versions

VER DESCRIPTION
--- ------------------------------------------------
1 Initial ZFS version
2 Ditto blocks (replicated metadata)
3 Hot spares and double parity RAID-Z
4 zpool history
5 Compression using the gzip algorithm
6 bootfs pool property
7 Separate intent log devices
8 Delegated administration
9 refquota and refreservation properties
10 Cache devices
11 Improved scrub performance
12 Snapshot properties
13 snapused property
14 passthrough-x aclinherit support

Continued...


More zpool Versions

VER DESCRIPTION
--- ------------------------------------------------
15 user/group space accounting
16 stmf property support
17 Triple-parity RAID-Z
18 snapshot user holds
19 Log device removal
20 Compression using zle (zero-length encoding)
21 Deduplication
22 Received properties
23 Slim ZIL
24 System attributes
25 Improved scrub stats
26 Improved snapshot deletion performance
27 Improved snapshot creation performance
28 Multiple vdev replacements

For Solaris 10, version 21 is “reserved”

zfs Versions

VER DESCRIPTION
----------------------------------------------
1 Initial ZFS filesystem version
2 Enhanced directory entries
3 Case insensitive and File system unique
identifier (FUID)
4 userquota, groupquota properties
5 System attributes


Copy on Write

1. Initial block tree 2. COW some data

3. COW metadata 4. Update Uberblocks & free


COW Notes

• COW works on blocks, not files
• ZFS reserves 32 MBytes or 1/64 of
pool size
✦ COWs need some free space to
remove files
✦ need space for ZIL
• For fixed-record size workloads
“fragmentation” and “poor
performance” can occur if the
recordsize is not matched
• Spatial distribution is good fodder for
performance speculation
✦ affects HDDs
✦ moot for SSDs

To fsck or not to fsck
• fsck was created to fix known inconsistencies in file system
metadata
✦ UFS is not transactional
✦ metadata inconsistencies must be reconciled
✦ does NOT repair data – how could it?
• ZFS doesn't need fsck, as-is
✦ all on-disk changes are transactional
✦ COW means previously existing, consistent metadata is not
overwritten
✦ ZFS can repair itself
✤ metadata is at least dual-redundant
✤ data can also be redundant
• Reality check – this does not mean that ZFS is not susceptible to
corruption
✦ nor is any other file system






Block Device Driver

HDD SSD iSCSI ??


vdevs – Virtual Devices

Logical vdevs

root vdev

top-level vdev top-level vdev
children[0] children[1]
mirror mirror

vdev vdev vdev vdev
type = disk type = disk type = disk type = disk
children[0] children[0] children[0] children[0]

Physical or leaf vdevs


vdev Labels
• vdev labels != disk labels
• Four 256 kByte labels written to every physical vdev
• Two-stage update process
✦ write label0 & label2
✦ ﬂush cache & check for errors
✦ write label1 & label3
✦ ﬂush cache & check for errors N = 256k * (size % 256k)
M = 128k / MIN(1k, sector size)
0 256k 512k 4M N-512k N-256k N

label0 label1 boot block label2 label3

...
Boot Name=Value
Blank
header Pairs M-slot Uberblock Array

0 8k 16k 128k 256k
25

ZFS Tutorial USENIX LISA’11

Observing Labels
# zdb -l /dev/rdsk/c0t0d0s0
--------------------------------------------
LABEL 0
--------------------------------------------
version=14
name='rpool'
state=0
txg=13152
pool_guid=17111649328928073943
hostid=8781271
hostname=''
top_guid=11960061581853893368
guid=11960061581853893368
vdev_tree
type='disk'
id=0
guid=11960061581853893368
path='/dev/dsk/c0t0d0s0'
devid='id1,sd@SATA_____ST3500320AS_________________9QM3FWFT/a'
phys_path='/pci@0,0/pci1458,b002@11/disk@0,0:a'
whole_disk=0
metaslab_array=24
metaslab_shift=30
ashift=9
asize=157945167872
is_log=0

Uberblocks
• Sized based on minimum device block size
• Stored in 128-entry circular queue
• Only one uberblock is active at any time
✦ highest transaction group number
✦ correct SHA-256 checksum
• Stored in machine's native format
✦ A magic number is used to determine endian format when
imported
• Contains pointer to Meta Object Set (MOS)
Device Block Size Uberblock Size Queue Entries
512 Bytes,1 KB 1 KB 128
2 KB 2 KB 64
4 KB 4 KB 32

About Sizes
• Sizes are dynamic
• LSIZE = logical size
• PSIZE = physical size after compression
• ASIZE = allocated size including:
✦ physical size
✦ raidz parity
✦ gang blocks

Old notions of size reporting confuse people

VDEV


Dynamic Striping
• RAID-0
✦ SNIA definition: fixed-length sequences of virtual disk data
addresses are mapped to sequences of member disk addresses
in a regular rotating pattern
• Dynamic Stripe
✦ Data is dynamically mapped to member disks
✦ No fixed-length sequences
✦ Allocate up to ~1 MByte/vdev before changing vdev
✦ vdevs can be different size
✦ Good combination of the concatenation feature with RAID-0
performance


Dynamic Striping

RAID-0 Column size = 128 kBytes, stripe width = 384 kBytes

384 kBytes

ZFS Dynamic Stripe recordsize = 128 kBytes

Total write size = 2816 kBytes


Mirroring
• Straightforward: put N copies of the data on N vdevs
• Unlike RAID-1
✦ No 1:1 mapping at the block level
✦ vdev labels are still at beginning and end
✦ vdevs can be of different size
✤ effective space is that of smallest vdev
• Arbitration: ZFS does not blindly trust either side of mirror
✦ Most recent, correct view of data wins
✦ Checksums validate data


Dynamic vdev Replacement

• zpool replace poolname vdev [vdev]
• Today, replacing vdev must be same size or larger
✦ NexentaStor 2 ‒ as measured by blocks
✦ NexentaStor 3 ‒ as measured by metaslabs
• Replacing all vdevs in a top-level vdev with larger vdevs results
in top-level vdev resizing
• Expansion policy controlled by:
✦ NexentaStor 2 ‒ resize on import
✦ NexentaStor 3 ‒ zpool autoexpand property

15G 10G
10G 15G 10G 20G 15G 20G
10G 15G 20G 20G 20G 20G
10G

10G Mirror 10G Mirror 15G Mirror 15G Mirror 20G Mirror


RAIDZ
• RAID-5
✦ Parity check data is distributed across the RAID array's disks
✦ Must read/modify/write when data is smaller than stripe width
• RAIDZ
✦ Dynamic data placement
✦ Parity added as needed
✦ Writes are full-stripe writes
✦ No read/modify/write (write hole)
• Arbitration: ZFS does not blindly trust any device
✦ Does not rely on disk reporting read error
✦ If checksum fails, read parity

Space used is dependent on how used

RAID-5 vs RAIDZ

DiskA DiskB DiskC DiskD DiskE
D0:0 D0:1 D0:2 D0:3 P0
RAID-5 P1 D1:0 D1:1 D1:2 D1:3
D2:3 P2 D2:0 D2:1 D2:2
D3:2 D3:3 P3 D3:0 D3:1

DiskA DiskB DiskC DiskD DiskE
P0 D0:0 D0:1 D0:2 D0:3
RAIDZ P1 D1:0 D1:1 P2:0 D2:0
D2:1 D2:2 D2:3 P2:1 D2:4
D2:5 Gap P3 D3:0


RAIDZ and Block Size

If block size >> N * sector size, space consumption is like RAID-5
If block size = sector size, space consumption is like mirroring

PSIZE=2KB
ASIZE=2.5KB DiskA DiskB DiskC DiskD DiskE
P0 D0:0 D0:1 D0:2 D0:3
P1 D1:0 D1:1 P2:0 D2:0
PSIZE=1KB D2:1 D2:2 D2:3 P2:1 D2:4
ASIZE=1.5KB D2:5 Gap P3 D3:0

PSIZE=3KB PSIZE=512 bytes
ASIZE=4KB + Gap ASIZE=1KB

Sector size = 512 bytes

Sector size can impact space savings

RAID-5 Write Hole
• Occurs when data to be written is smaller than stripe size
• Must read unallocated columns to recalculate the parity or the
parity must be read/modify/write
• Read/modify/write is risky for consistency
✦ Multiple disks
✦ Reading independently
✦ Writing independently
✦ System failure before all writes are complete to media could
result in data loss
• Effects can be hidden from host using RAID array with
nonvolatile write cache, but extra I/O cannot be hidden from
disks


RAIDZ2 and RAIDZ3
• RAIDZ2 = double parity RAIDZ
• RAIDZ3 = triple parity RAIDZ
• Sorta like RAID-6
✦ Parity 1: XOR
✦ Parity 2: another Reed-Soloman syndrome
✦ Parity 3: yet another Reed-Soloman syndrome
• Arbitration: ZFS does not blindly trust any device
✦ Does not rely on disk reporting read error
✦ If data not valid, read parity
✦ If data still not valid, read other parity

Space used is dependent on how used

Evaluating Data Retention
• MTTDL = Mean Time To Data Loss
• Note: MTBF is not constant in the real world, but keeps math
simple
• MTTDL[1] is a simple MTTDL model
• No parity (single vdev, striping, RAID-0)
✦ MTTDL[1] = MTBF / N
• Single Parity (mirror, RAIDZ, RAID-1, RAID-5)
✦ MTTDL[1] = MTBF2 / (N * (N-1) * MTTR)
• Double Parity (3-way mirror, RAIDZ2, RAID-6)
✦ MTTDL[1] = MTBF3 / (N * (N-1) * (N-2) * MTTR2)
• Triple Parity (4-way mirror, RAIDZ3)
✦ MTTDL[1] = MTBF4 / (N * (N-1) * (N-2) * (N-3) * MTTR3)


Another MTTDL Model
• MTTDL[1] model doesn't take into account unrecoverable
read
• But unrecoverable reads (UER) are becoming the dominant
failure mode
✦ UER specifed as errors per bits read
✦ More bits = higher probability of loss per vdev
• MTTDL[2] model considers UER


Why Worry about UER?
• Richard's study
✦ 3,684 hosts with 12,204 LUNs
✦ 11.5% of all LUNs reported read errors
• Bairavasundaram et.al. FAST08
www.cs.wisc.edu/adsl/Publications/corruption-fast08.pdf
✦ 1.53M LUNs over 41 months
✦ RAID reconstruction discovers 8% of checksum mismatches
✦ “For some drive models as many as 4% of drives develop
checksum mismatches during the 17 months examined”


• RAID array study


• RAID array study

Unrecoverable Disk Disappeared
Reads “disk pull”

“Disk pull” tests aren’t very useful

MTTDL[2] Model
• Probability that a reconstruction will fail
✦ Precon_fail = (N-1) * size / UER
• Model doesn't work for non-parity schemes
✦ single vdev, striping, RAID-0
• Single Parity (mirror, RAIDZ, RAID-1, RAID-5)
✦ MTTDL[2] = MTBF / (N * Precon_fail)
• Double Parity (3-way mirror, RAIDZ2, RAID-6)
✦ MTTDL[2] = MTBF2/ (N * (N-1) * MTTR * Precon_fail)
• Triple Parity (4-way mirror, RAIDZ3)
✦ MTTDL[2] = MTBF3/ (N * (N-1) * (N-2) * MTTR2 *
Precon_fail)


Practical View of MTTDL[1]


MTTDL[1] Comparison


MTTDL Models: Mirror

Spares are not always better...

MTTDL Models: RAIDZ2


Space, Dependability, and Performance


Dependability Use Case
• Customer has 15+ TB of read-mostly data
• 16-slot, 3.5” drive chassis
• 2 TB HDDs
• Option 1: one raidz2 set
✦ 24 TB available space
✤ 12 data
✤ 2 parity
✤ 2 hot spares, 48 hour disk replacement time
✦ MTTDL[1] = 1,790,000 years
• Option 2: two raidz2 sets
✦ 24 TB available space (each set)
✤ 6 data
✤ 2 parity
✤ no hot spares
✦ MTTDL[1] = 7,450,000 years

Ditto Blocks
• Recall that each blkptr_t contains 3 DVAs
• Dataset property used to indicate how many copies (aka ditto
blocks) of data is desired
✦ Write all copies
✦ Read any copy
✦ Recover corrupted read from a copy
• Not a replacement for mirroring
✦ For single disk, can handle data loss on approximately 1/8
contiguous space
• Easier to describe in pictures...
copies parameter Data copies Metadata copies
copies=1 (default) 1 2
copies=2 2 3
copies=3 3 3

Copies in Pictures


When Good Data Goes Bad

File system If it’s a metadata Or we get
does bad read block FS panics back bad
Can not tell does disk rebuild data


Checksum Veriﬁcation

ZFS veriﬁes checksums for every read
Repairs data when possible (mirror, raidz, copies>1)

Read bad data Read good data Repair bad data


ZIO - ZFS I/O Layer

56

ZIO Framework
• All physical disk I/O goes through ZIO Framework
• Translates DVAs into Logical Block Address (LBA) on leaf
vdevs
✦ Keeps free space maps (spacemap)
✦ If contiguous space is not available:
✤ Allocate smaller blocks (the gang)
✤ Allocate gang block, pointing to the gang
• Implemented as multi-stage pipeline
✦ Allows extensions to be added fairly easily
• Handles I/O errors


ZIO Write Pipeline

ZIO State Compression Checksum DVA vdev I/O

open
compress if
savings >
12.5%
generate
allocate

start
start
start
done
done
done
assess
assess
assess

done

Gang and deduplicaiton activity elided, for clarity

ZIO Read Pipeline

ZIO State Compression Checksum DVA vdev I/O

open

start
start
start
done
done
done
assess
assess
assess

verify

decompress

done

Gang and deduplicaiton activity elided, for clarity

VDEV – Virtual Device Subsytem
• Where mirrors, RAIDZ, and Name Priority
RAIDZ2 are implemented
NOW 0
✦ Surprisingly few lines of code
SYNC_READ 0
needed to implement RAID
SYNC_WRITE 0
• Leaf vdev (physical device) I/O FREE 0
management CACHE_FILL 0
✦ Number of outstanding iops LOG_WRITE 0
✦ Read-ahead cache ASYNC_READ 4
• Priority scheduling ASYNC_WRITE 4
RESILVER 10
SCRUB 20


ARC - Adaptive
Replacement Cache

61

Object Cache
• UFS uses page cache managed by the virtual memory system
• ZFS does not use the page cache, except for mmap'ed ﬁles
• ZFS uses a Adaptive Replacement Cache (ARC)
• ARC used by DMU to cache DVA data objects
• Only one ARC per system, but caching policy can be changed
on a per-dataset basis
• Seems to work much better than page cache ever did for UFS


Traditional Cache
• Works well when data being accessed was recently added
• Doesn't work so well when frequently accessed data is evicted

Misses cause insert

MRU

Dynamic caches can change
Cache size size by either not evicting
or aggressively evicting

LRU

Evict the oldest


ARC – Adaptive Replacement Cache

Evict the oldest single-use entry

LRU
Recent
Cache
Miss
MRU
Evictions and dynamic
MFU size resizing needs to choose best
Hit
cache to evict (shrink)
Frequent
Cache

LFU

Evict the oldest multiple accessed entry


ARC with Locked Pages

Evict the oldest single-use entry

Cannot evict LRU
locked pages!
Recent
Cache
Miss
MRU
MFU size
Hit
Frequent
If hit occurs Cache
within 62 ms
LFU

Evict the oldest multiple accessed entry

ZFS ARC handles mixed-size pages


L2ARC – Level 2 ARC
• Data soon to be evicted from the ARC is added
to a queue to be sent to cache vdev
✦ Another thread sends queue to cache vdev ARC
✦ Data is copied to the cache vdev with a throttle
data soon to
to limit bandwidth consumption be evicted
✦ Under heavy memory pressure, not all evictions
will arrive in the cache vdev
• ARC directory remains in memory
• Good idea - optimize cache vdev for fast reads
✦ lower latency than pool disks
✦ inexpensive way to “increase memory”
cache
• Content considered volatile, no raid needed
• Monitor usage with zpool iostat and ARC kstats


ARC Directory
• Each ARC directory entry contains arc_buf_hdr structs
✦ Info about the entry
✦ Pointer to the entry
• Directory entries have size, ~200 bytes
• ZFS block size is dynamic, sector size to 128 kBytes
• Disks are large
• Suppose we use a Seagate LP 2 TByte disk for the L2ARC
✦ Disk has 3,907,029,168 512 byte sectors, guaranteed
✦ Workload uses 8 kByte ﬁxed record size
✦ RAM needed for arc_buf_hdr entries
✤ Need = (3,907,029,168 - 9,232) * 200 / 16 = ~48 GBytes
• Don't underestimate the RAM needed for large L2ARCs


ARC Tips
• In general, it seems to work well for most workloads
• ARC size will vary, based on usage
✦ Default target max is 7/8 of physical memory or (memory - 1
GByte)
✦ Target min is 64 MB
✦ Metadata capped at 1/4 of max ARC size
• Dynamic size can be reduced when:
✦ page scanner is running
✤ freemem < lotsfree + needfree + desfree
✦ swapfs does not have enough space so that anonymous
reservations can succeed
✤ availrmem < swapfs_minfree + swapfs_reserve + desfree
✦ [x86 only] kernel heap space more than 75% full
• Can limit at boot time

Observing ARC
• ARC statistics stored in kstats
• kstat -n arcstats
• Interesting statistics:
✦ size = current ARC size
✦ p = size of MFU cache
✦ c = target ARC size
✦ c_max = maximum target ARC size
✦ c_min = minimum target ARC size
✦ l2_hdr_size = space used in ARC by L2ARC
✦ l2_size = size of data in L2ARC


General Status - ARC


More ARC Tips
• Performance
✦ Prior to b107, L2ARC fill rate was limited to 8 MB/sec
✦ After b107, cold L2ARC fill rate increases to 16 MB/sec
• Internals tracked by kstats in Solaris
✦ Use memory_throttle_count to observe pressure to
evict
• Dedup Table (DDT) also uses ARC
✦ lots of dedup objects need lots of RAM
✦ field reports that L2ARC can help with dedup

L2ARC keeps its directory in kernel memory

Transactional
Object Layer

72

ﬂash
Source Code Structure

File system Mgmt
Device
Consumer Consumer
libzfs

Interface
ZPL ZVol /dev/zfs
Layer

Transactional ZIL ZAP Traversal
Object
Layer DMU DSL

ARC
Pooled
Storage ZIO
Layer
VDEV Configuration


Transaction Engine
• Manages physical I/O
• Transactions grouped into transaction group (txg)
✦ txg updates
✦ All-or-nothing
✦ Commit interval
✤ Older versions: 5 seconds
✤ Less old versions: 30 seconds
✤ b143 and later: 5 seconds
• Delay committing data to physical storage
✦ Improves performance
✦ A bad thing for sync workload performance – hence the ZFS
Intent Log (ZIL)

30 second delay can impact failure detection time

ZIL – ZFS Intent Log
• DMU is transactional, and likes to group I/O into transactions
for later commits, but still needs to handle “write it now”
desire of sync writers
✦ NFS
✦ Databases
• ZIL recordsize inﬂation can occur for some workloads
✦ May cause larger than expected actual I/O for sync workloads
✦ Oracle redo logs
✦ No slog: can tune zfs_immediate_write_sz,
zvol_immediate_write_sz
✦ With slog: use logbias property instead
• Never read, except at import (eg reboot), when transactions
may need to be rolled forward


Separate Logs (slogs)
• ZIL competes with pool for IOPS
✦ Applications wait for sync writes to be on nonvolatile media
✦ Very noticeable on HDD JBODs
• Put ZIL on separate vdev, outside of pool
✦ ZIL writes tend to be sequential
✦ No competition with pool for IOPS
✦ Downside: slog device required to be operational at import
✦ NexentaStor 3 allows slog device removal
✦ Size of separate log < than size of RAM (duh)
• 10x or more performance improvements possible
✦ Nonvolatile RAM card
✦ Write-optimized SSD
✦ Nonvolatile write cache on RAID array


zilstat
• http://www.richardelling.com/Home/scripts-and-programs-1/
zilstat
• Integrated into NexentaStor 3.0.3
✦ nmc: show performance zil


Synchronous Write Destination

Without separate log
Sync I/O size >
ZIL Destination
zfs_immediate_write_sz ?
no ZIL log
yes bypass to pool

With separate log
logbias? ZIL Destination
latency (default) log device
throughput bypass to pool

Default zfs_immediate_write_sz = 32 kBytes

ZIL Synchronicity Project
• All-or-nothing policies don’t work well, in general
• ZIL Synchronicity project proposed by Robert Milkowski
✦ http://milek.blogspot.com
• Adds new sync property to datasets
• Arrived in b140
sync Parameter Behaviour
Policy follows previous design: write
standard (default)
immediate size and separate logs
always All writes become synchronous (slow)
disabled Synchronous write requests are ignored


Disabling the ZIL

• Preferred method: change dataset sync property
• Rule 0: Don’t disable the ZIL
• If you love your data, do not disable the ZIL
• You can ﬁnd references to this as a way to speed up ZFS
✦ NFS workloads
✦ “tar -x” benchmarks
• Golden Rule: Don’t disable the ZIL
• Can set via mdb, but need to remount the ﬁle system
• Friends don’t let friends disable the ZIL
• Older Solaris - can set in /etc/system
• NexentaStor has checkbox for disabling ZIL
• Nostradamus wrote, “disabling the ZIL will lead to the
apocalypse”

DSL - Dataset and
Snapshot Layer

81

Dataset & Snapshot Layer
• Object
✦ Allocated storage
✦ dnode describes collection of blocks
• Object Set
Dataset Directory
✦ Group of related objects Dataset
• Dataset Object Set Childmap
✦ Snapmap: snapshot relationships
Object
Object
✦ Space usage Object Properties

• Dataset directory Snapmap
✦ Childmap: dataset relationships
✦ Properties


ﬂash
Copy on Write



z

zfs snapshot
• Create a read-only, point-in-time window into the dataset (ﬁle
system or Zvol)
• Computationally free, because of COW architecture
• Very handy feature
✦ Patching/upgrades
• Basis for time-related snapshot interfaces
✦ Solaris Time Slider
✦ NexentaStor Delorean Plugin
✦ NexentaStor Virtual Machine Data Center


Snapshot
• Create a snapshot by not free'ing COWed blocks
• Snapshot creation is fast and easy
• Number of snapshots determined by use – no hardwired limit
• Recursive snapshots also possible

Snapshot tree Current tree
root root


auto-snap service


Clones
• Snapshots are read-only
• Clones are read-write based upon a snapshot
• Child depends on parent
✦ Cannot destroy parent without destroying all children
✦ Can promote children to be parents
• Good ideas
✦ OS upgrades
✦ Change control
✦ Replication
✤ zones
✤ virtual disks


zfs clone
• Create a read-write ﬁle system from a read-only snapshot
• Solaris boot environment administation
Install Checkpoint Clone Checkpoint

OS rev1 OS rev1 OS rev1 OS rev1 OS rev1

rootfs- rootfs- rootfs- rootfs-
nmu- nmu- nmu- nmu-
001 001 001 001

patch/
OS rev1 OS rev1 OS rev1
upgrade
clone clone clone
rootfs-
nmu-
002
grubboot
manager

Origin snapshot cannot be destroyed, if clone exists


What is Deduplication?
• A $2.1 Billion feature
• 2009 buzzword of the year
• Technique for improving storage space efﬁciency
✦ Trades big I/Os for small I/Os
✦ Does not eliminate I/O
• Implementation styles
✦ ofﬂine or post processing
✤ data written to nonvolatile storage
✤ process comes along later and dedupes data
✤ example: tape archive dedup
✦ inline
✤ data is deduped as it is being allocated to nonvolatile storage
✤ example: ZFS


Dedup how-to
• Given a bunch of data
• Find data that is duplicated
• Build a lookup table of references to data
• Replace duplicate data with a pointer to the entry in the
lookup table
• Grainularity
✦ ﬁle
✦ block
✦ byte


Dedup in ZFS
• Leverage block-level checksums
✦ Identify blocks which might be duplicates
✦ Variable block size is ok
• Synchronous implementation
✦ Data is deduped as it is being written
• Scalable design
✦ No reference count limits
• Works with existing features
✦ compression
✦ copies
✦ scrub
✦ resilver
• Implemented in ZIO pipeline

Deduplication Table (DDT)
• Internal implementation
✦ Adelson-Velskii, Landis (AVL) tree
✦ Typical table entry ~270 bytes
✤ checksum
✤ logical size
✤ physical size
✤ references
✦ Table entry size increases as the number of references
increases


Reference Counts

Eggs courtesy of Richard’s chickens

Reference Counts
• Problem: loss of the referenced data affects all referrers
• Solution: make additional copies of referred data based upon a
threshold count of referrers
✦ leverage copies (ditto blocks)
✦ pool-level threshold for automatically adding ditto copies
✤ set via dedupditto pool property
# zpool set dedupditto=50 zwimming

✤ add 2nd copy when dedupditto references (50) reached
✤ add 3rd copy when dedupditto2 references (2500) reached


Veriﬁcation

write()

compress

checksum

DDT entry lookup

yes no
DDT
verify?
match?

no yes

read data

data yes
add reference
match?

no

new entry


Enabling Dedup
• Set dedup property for each dataset to be deduped
• Remember: properties are inherited
• Remember: only applies to newly written data

dedup checksum verify?
on
SHA256 no
sha256
on,verify
SHA256 yes
sha256,verify

Fletcher is considered too weak, without verify

Dedup Accounting
• ...and you thought compression accounting was hard...
• Remember: dedup works at pool level
✦ dataset-level accounting doesn’t see other datasets
✦ pool-level accounting is always correct

zfs list
NAME USED AVAIL REFER MOUNTPOINT
bar 7.56G 449G 22K /bar
bar/ws 7.56G 449G 7.56G /bar/ws
dozer 7.60G 455G 22K /dozer
dozer/ws 7.56G 455G 7.56G /dozer/ws
tank 4.31G 456G 22K /tank
tank/ws 4.27G 456G 4.27G /tank/ws

zpool list
NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT
bar 464G 7.56G 456G 1% 1.00x ONLINE -
dozer 464G 1.43G 463G 0% 5.92x ONLINE -
tank 464G 957M 463G 0% 5.39x ONLINE -

ZFS Tutorial DataUSENIX LISA’11team
courtesy of the ZFS 98

DDT Histogram

# zdb -DD tank
DDT-sha256-zap-duplicate: 110173 entries, size 295 on disk, 153 in core
DDT-sha256-zap-unique: 302 entries, size 42194 on disk, 52827 in core

DDT histogram (aggregated over all DDTs):

bucket! allocated! referenced
______ ___________________________ ___________________________
refcnt blocks LSIZE PSIZE DSIZE blocks LSIZE PSIZE DSIZE
------ ------ ----- ----- ----- ------ ----- ----- -----
1 302 7.26M 4.24M 4.24M 302 7.26M 4.24M 4.24M
2 103K 1.12G 712M 712M 216K 2.64G 1.62G 1.62G
4 3.11K 30.0M 17.1M 17.1M 14.5K 168M 95.2M 95.2M
8 503 11.6M 6.16M 6.16M 4.83K 129M 68.9M 68.9M
16 100 4.22M 1.92M 1.92M 2.14K 101M 45.8M 45.8M

ZFS Tutorial USENIX LISA’11
Data courtesy of the ZFS team 99

DDT Histogram

$ zdb -DD zwimming
DDT-sha256-zap-duplicate: 19725 entries, size 270 on disk, 153 in core
DDT-sha256-zap-unique: 52369639 entries, size 284 on disk, 159 in core

DDT histogram (aggregated over all DDTs):

bucket allocated referenced
______ ______________________________ ______________________________
refcnt blocks LSIZE PSIZE DSIZE blocks LSIZE PSIZE DSIZE
------ ------ ----- ----- ----- ------ ----- ----- -----
1 49.9M 25.0G 25.0G 25.0G 49.9M 25.0G 25.0G 25.0G
2 16.7K 8.33M 8.33M 8.33M 33.5K 16.7M 16.7M 16.7M
4 610 305K 305K 305K 3.33K 1.66M 1.66M 1.66M
8 661 330K 330K 330K 6.67K 3.34M 3.34M 3.34M
16 242 121K 121K 121K 5.34K 2.67M 2.67M 2.67M
32 131 65.5K 65.5K 65.5K 5.54K 2.77M 2.77M 2.77M
64 897 448K 448K 448K 84K 42M 42M 42M
128 125 62.5K 62.5K 62.5K 18.0K 8.99M 8.99M 8.99M
8K 1 512 512 512 12.5K 6.27M 6.27M 6.27M
Total 50.0M 25.0G 25.0G 25.0G 50.1M 25.1G 25.1G 25.1G

dedup = 1.00, compress = 1.00, copies = 1.00, dedup * compress / copies = 1.00


Over-the-wire Dedup
• Dedup is also possible over the send/receive pipe
✦ Blocks with no checksum are considered duplicates (no verify
option)
✦ First copy sent as usual
✦ Subsequent copies sent by reference
• Independent of dedup status of originating pool
✦ Receiving pool knows about blocks which have already arrived
• Can be a win for dedupable data, especially over slow wires
• Remember: send/receive version rules still apply

# zfs send -DR zwimming/stuff


Dedup Performance
• Dedup can save space and bandwidth
• Dedup increases latency
✦ Caching data improves latency
✦ More memory → more data cached
✦ Cache performance heirarchy
✤ RAM: fastest
✤ L2ARC on SSD: slower
✤ Pool HDD: dreadfully slow
• ARC is currently not deduped
• Difﬁcult to predict
✦ Dependent variable: number of blocks
✦ Estimate 270 bytes per unique block
✦ Example:
✤ 50M blocks * 270 bytes/block = 13.5 GBytes

Deduplication Use Cases

Data type Dedupe Compression
Home directories ✔✔ ✔✔

Internet content ✔ ✔

Media and video ✔✔ ✔

Life sciences ✘ ✔✔

Oil and Gas (seismic) ✘ ✔✔

Virtual machines ✔✔ ✘

Archive ✔✔✔✔ ✔







Block Device Driver

HDD SSD iSCSI ??


zpool create
• zpool create poolname vdev-configuration
• nmc: setup volume create
✦ vdev-configuration examples

✤ mirror c0t0d0 c3t6d0
✤ mirror c0t0d0 c3t6d0 mirror c4t0d0 c0t1d6
✤ mirror disk1s0 disk2s0 cache disk4s0 log disk5
✤ raidz c0d0s1 c0d1s1 c1d2s0 spare c1d3s0
• Solaris
✦ Additional checks for disk/slice overlaps or in use
✦ Whole disks are given EFI labels
• Can set initial pool or dataset properties
• By default, creates a file system with the same name
✦ poolname pool → /poolname file system

People get confused by a file system with same name as the pool

zpool destroy
• Destroy the pool and all datasets therein
• zpool destroy poolname
✦ Can (try to) force with “-f”
✦ There is no “are you sure?” prompt – if you weren't sure, you
would not have typed “destroy”
• nmc: destroy volume volumename
✦ nmc prompts for conﬁrmation, by default

zpool destroy is destructive... really! Use with caution!

zpool add
• Adds a device to the pool as a top-level vdev
• Does NOT not add columns to a raidz set
• Does NOT attach a mirror – use zpool attach instead
• zpool add poolname vdev-configuration
✦ vdev-configuration can be any combination also used for zpool

create
✦ Complains if the added vdev-configuration would cause a

different data protection scheme than is already in use
✤ use “-f” to override
✦ Good idea: try with “-n” flag first
✤ will show final configuration without actually performing the add
• nmc: setup volume volumename grow

Do not add a device which is in use as a cluster quorum device

zpool remove
• Remove a top-level vdev from the pool
• zpool remove poolname vdev
• nmc: setup volume volumename remove-lun
• Today, you can only remove the following vdevs:
✦ cache

✦ hot spare

✦ separate log (b124, NexentaStor 3.0)

Don't confuse “remove” with “detach”

zpool attach
• Attach a vdev as a mirror to an existing vdev
• zpool attach poolname existing-vdev vdev
• nmc: setup volume volumename attach-lun
• Attaching vdev must be the same size or larger than the
existing vdev

vdev Conﬁgurations
ok simple vdev → mirror
ok mirror
ok log → mirrored log
no RAIDZ
no RAIDZ2
no RAIDZ3


zpool detach
• Detach a vdev from a mirror
• zpool detach poolname vdev
• nmc: setup volume volumename detach-lun
• A resilvering vdev will wait until resilvering is complete


zpool replace
• Replaces an existing vdev with a new vdev
• zpool replace poolname existing-vdev vdev
• nmc: setup volume volumename replace-lun
• Effectively, a shorthand for “zpool attach” followed by “zpool
detach”
• Attaching vdev must be the same size or larger than the
existing vdev
• Works for any top-level vdev-conﬁguration, including RAIDZ

“Same size” literally means the same number of blocks until b117.
Many “same size” disks have different number of available blocks.

zpool import
• Import a pool and mount all mountable datasets
• Import a speciﬁc pool
✦ zpool import poolname
✦ zpool import GUID
✦ nmc: setup volume import
• Scan LUNs for pools which may be imported
✦ zpool import
• Can set options, such as alternate root directory or other
properties
✦ alternate root directory important for rpool or syspool

Beware of zpool.cache interactions

Beware of artifacts, especially partial artifacts

zpool export
• Unmount datasets and export the pool
• zpool export poolname
• nmc: setup volume volumename export
• Removes pool entry from zpool.cache
✦ useful when unimported pools remain in zpool.cache


zpool upgrade

• Display current versions
✦ zpool upgrade
• View available upgrade versions, with features, but don't
actually upgrade
✦ zpool upgrade -v
• Upgrade pool to latest version
✦ zpool upgrade poolname
✦ nmc: setup volume volumename version-
upgrade
• Upgrade pool to speciﬁc version

Once you upgrade, there is no downgrade

Beware of grub and rollback issues

zpool history
• Show history of changes made to the pool
• nmc and Solaris use same command


zpool history
• Show history of changes made to the pool
# zpool history rpool
History for 'rpool':
2009-03-04.07:29:46 zpool create -f -o failmode=continue -R /a -m legacy -o
cachefile=/tmp/root/etc/zfs/zpool.cache rpool c0t0d0s0
2009-03-04.07:29:47 zfs set canmount=noauto rpool
2009-03-04.07:29:47 zfs set mountpoint=/rpool rpool
2009-03-04.07:29:47 zfs create -o mountpoint=legacy rpool/ROOT
2009-03-04.07:29:48 zfs create -b 4096 -V 2048m rpool/swap
2009-03-04.07:29:48 zfs create -b 131072 -V 1024m rpool/dump
2009-03-04.07:29:49 zfs create -o canmount=noauto rpool/ROOT/snv_106
2009-03-04.07:29:50 zpool set bootfs=rpool/ROOT/snv_106 rpool
2009-03-04.07:29:50 zfs set mountpoint=/ rpool/ROOT/snv_106
2009-03-04.07:29:51 zfs set canmount=on rpool
2009-03-04.07:29:51 zfs create -o mountpoint=/export rpool/export
2009-03-04.07:29:51 zfs create rpool/export/home
2009-03-04.00:21:42 zpool import -f -R /a 17111649328928073943
2009-03-04.00:21:42 zpool export rpool
2009-03-04.08:47:08 zpool set bootfs=rpool rpool
2009-03-04.08:47:08 zpool set bootfs=rpool/ROOT/snv_106 rpool
2009-03-04.08:47:12 zfs snapshot rpool/ROOT/snv_106@snv_b108
2009-03-04.08:47:12 zfs clone rpool/ROOT/snv_106@snv_b108 rpool/ROOT/snv_b108
...


zpool status
• Shows the status of the current pools, including their
conﬁguration
• Important troubleshooting step
# zpool status
…
pool: zwimming
state: ONLINE
status: The pool is formatted using an older on-disk format. The pool can
still be used, but some features are unavailable.
action: Upgrade the pool using 'zpool upgrade'. Once this is done, the
pool will no longer be accessible on older software versions.
scrub: none requested
config:
NAME STATE READ WRITE CKSUM
zwimming ONLINE 0 0 0
mirror ONLINE 0 0 0
c0t2d0s0 ONLINE 0 0 0
c0t0d0s7 ONLINE 0 0 0
errors: No known data errors

Understanding status output error messages can be tricky

zpool clear
• Clears device errors
• Clears device error counters
• Starts any resilvering, as needed
• Improves sysadmin sanity and reduces sweating
• zpool clear poolname
• nmc: setup volume volumename clear-errors


zpool iostat
• Show pool physical I/O activity, in an iostat-like manner
• Solaris: fsstat will show I/O activity looking into a ZFS ﬁle
system
• Especially useful for showing slog activity
# zpool iostat -v
capacity operations bandwidth
pool used avail read write read write
------------ ----- ----- ----- ----- ----- -----
rpool 16.5G 131G 0 0 1.16K 2.80K
c0t0d0s0 16.5G 131G 0 0 1.16K 2.80K
------------ ----- ----- ----- ----- ----- -----
zwimming 135G 14.4G 0 5 2.09K 27.3K
mirror 135G 14.4G 0 5 2.09K 27.3K
c0t2d0s0 - - 0 3 1.25K 27.5K
c0t0d0s7 - - 0 2 1.27K 27.5K
------------ ----- ----- ----- ----- ----- -----

Unlike iostat, does not show latency

zpool scrub
• Manually starts scrub
✦ zpool scrub poolname
• Scrubbing performed in background
• Use zpool status to track scrub progress
• Stop scrub
✦ zpool scrub -s poolname
• How often to scrub?
✦ Depends on level of paranoia
✦ Once per month seems reasonable
✦ After a repair or recovery procedure
• NexentaStor auto-scrub features easily manages scrubs and
schedules

Estimated scrub completion time improves over time

auto-scrub service


Dataset Management





Block Device Driver

HDD SSD iSCSI ??


zfs create, destroy
• By default, a ﬁle system with the same name as the pool is
created by zpool create
• Dataset name format is: pool/name[/name ...]
• File system / folder
✦ zfs create dataset-name
✦ nmc: create folder
✦ zfs destroy dataset-name
✦ nmc: destroy folder
• Zvol
✦ zfs create -V size dataset-name
✦ nmc: create zvol
✦ zfs destroy dataset-name
✦ nmc: destroy zvol


zfs mount, unmount
• Note: mount point is a file system parameter
✦ zfs get mountpoint fs-name
• Rarely used subcommand (!)
• Display mounted file systems
✦ zfs mount
• Mount a file system
✦ zfs mount fs-name
✦ zfs mount -a
• Unmount (not umount)
✦ zfs unmount fs-name
✦ zfs unmount -a


zfs list
• List mounted datasets
• NexentaStor 2: listed everything
• NexentaStor 3: do not list snapshots
✦ See zpool listsnapshots property
• Examples
✦ zfs list
✦ zfs list -t snapshot
✦ zfs list -H -o name


Replication Services

Days
Traditional Backup NDMP

Hours Auto-Tier
Recovery rsync

Point Text
Auto-Sync
ZFS send/receive
Objective
Seconds
Auto-CDP Application Level
AVS (SNDR) Mirror Replication

Slower Faster

System I/O Performance

zfs send, receive
• Send
✦ send a snapshot to stdout
✦ data is decompressed
• Receive
✦ receive a snapshot from stdin
✦ receiving ﬁle system parameters apply (compression, et.al)
• Can incrementally send snapshots in time order
• Handy way to replicate dataset snapshots
• NexentaStor
✦ simpliﬁes management
✦ manages snapshots and send/receive to remote systems
• Only method for replicating dataset properties, except quotas
• NOT a replacement for traditional backup solutions

auto-sync Service


zfs upgrade

• Display current versions
✦ zfs upgrade
• View available upgrade versions, with features, but don't
actually upgrade
✦ zfs upgrade -v
• Upgrade pool to latest version
✦ zfs upgrade dataset
• Upgrade pool to speciﬁc version
✦ zfs upgrade -V version dataset
• NexentaStor: not needed until 3.0
You can upgrade, there is no downgrade

Beware of grub and rollback issues

Sharing
• zfs share dataset
• Type of sharing set by parameters
✦ shareiscsi = [on | off]

✦ sharenfs = [on | off | options]

✦ sharesmb = [on | off | options]

• Shortcut to manage sharing
✦ Uses external services (nfsd, iscsi target, smbshare, etc)

✦ Importing pool will also share

✦ Implementation is OS-speciﬁc

✤ sharesmb uses in-kernel SMB server for Solaris-derived OSes
✤ sharesmb uses Samba for FreeBSD


Properties

• Properties are stored in an nvlist
• By default, are inherited
• Some properties are common to all datasets, but a speciﬁc
dataset type may have additional properties
• Easily set or retrieved via scripts
• In general, properties affect future ﬁle system activity

zpool get doesn't script as nicely as zfs get

Getting Properties

• zpool get all poolname
• nmc: show volume volumename property
propertyname
• zpool get propertyname poolname

• zfs get all dataset-name
• nmc: show folder foldername property
• nmc: show zvol zvolname property


Setting Properties

• zpool set propertyname=value poolname
• nmc: setup volume volumename property
propertyname

• zfs set propertyname=value dataset-name
• nmc: setup folder foldername property
propertyname


User-deﬁned Properties
• Names
✦ Must include colon ':'
✦ Can contain lower case alphanumerics or “+” “.” “_”
✦ Max length = 256 characters
✦ By convention, module:property
✤ com.sun:auto-snapshot
• Values
✦ Max length = 1024 characters
• Examples
✦ com.sun:auto-snapshot=true
✦ com.richardelling:important_ﬁles=true


Clearing Properties
• Reset to inherited value
✦ zfs inherit compression export/home/relling
• Clear user-deﬁned parameter
✦ zfs inherit com.sun:auto-snapshot export/
home/relling
• NexentaStor doesn’t offer method in nmc


Pool Properties

Property Change? Brief Description
altroot Alternate root directory (ala chroot)
autoexpand Policy for expanding when vdev size changes
autoreplace vdev replacement policy
available readonly Available storage space
bootfs Default bootable dataset for root pool
Cache ﬁle to use other than /etc/zfs/
cacheﬁle
zpool.cache
capacity readonly Percent of pool space used
delegation Master pool delegation switch
failmode Catastrophic pool failure policy


More Pool Properties

guid readonly Unique identiﬁer
health readonly Current health of the pool
listsnapshots zfs list policy
size readonly Total size of pool
used readonly Amount of space used
version readonly Current on-disk version


Common Dataset Properties

available readonly Space available to dataset & children
checksum Checksum algorithm
compression Compression algorithm
Compression ratio – logical
compressratio readonly
size:referenced physical
copies Number of copies of user data
creation readonly Dataset creation time
dedup Deduplication policy
logbias Separate log write policy
mlslabel Multilayer security label
origin readonly For clones, origin snapshot


More Common Dataset Properties

primarycache ARC caching policy
readonly Is dataset in readonly mode?
referenced readonly Size of data accessible by this dataset
Minimum space guaranteed to a
refreservation dataset, excluding descendants
(snapshots & clones)
Minimum space guaranteed to dataset,
reservation
including descendants
secondarycache L2ARC caching policy
sync Synchronous write policy
Type of dataset (ﬁlesystem, snapshot,
type readonly
volume)


More Common Dataset Properties

used readonly Sum of usedby* (see below)
usedbychildren readonly Space used by descendants
usedbydataset readonly Space used by dataset
Space used by a refreservation for
usedbyrefreservation readonly
this dataset
Space used by all snapshots of this
usedbysnapshots readonly
dataset
Is dataset added to non-global zone
zoned readonly
(Solaris)


Volume Dataset Properties

shareiscsi iSCSI service (not COMSTAR)
volblocksize creation ﬁxed block size
volsize Implicit quota
Set if dataset delegated to non-global
zoned readonly
zone (Solaris)


File System Properties

ACL inheritance policy, when files or
aclinherit
directories are created
ACL modification policy, when chmod is
aclmode
used
atime Disable access time metadata updates
canmount Mount policy
Filename matching algorithm (CIFS client
casesensitivity creation
feature)
devices Device opening policy for dataset
exec File execution policy for dataset
mounted readonly Is file system currently mounted?


More File System Properties

export/ File system should be mounted with non-blocking
nbmand
import mandatory locks (CIFS client feature)
normalization creation Unicode normalization of ﬁle names for matching
quota Max space dataset and descendants can consume
recordsize Suggested maximum block size for ﬁles
Max space dataset can consume, not including
refquota
descendants
setuid setuid mode policy
sharenfs NFS sharing options
sharesmb Files system shared with CIFS


File System Properties

snapdir Controls whether .zfs directory is hidden
utf8only creation UTF-8 character ﬁle name policy
vscan Virus scan enabled
xattr Extended attributes policy


Forking Properties

Pool Properties
Release Property Brief Description
illumos comment Human-readable comment ﬁeld

Dataset Properties
Release Property Brief Description
Solaris 11 encryption Dataset encryption
Delphix/illumos clones Clone descendants
Delphix/illumos refratio Compression ratio for references
Solaris 11 share Combines sharenfs & sharesmb
Solaris 11 shadow Shadow copy
NexentaOS/illumos worm WORM feature
Amount of data written since last
Delphix/illumos written
snapshot

Dataset Space Accounting
• used = usedbydataset + usedbychildren + usedbysnapshots +
usedbyrefreservation
• Lazy updates, may not be correct until txg commits
• ls and du will show size of allocated ﬁles which includes all
copies of a ﬁle
• Shorthand report available

$ zfs list -o space
NAME AVAIL USED USEDSNAP USEDDS USEDREFRESERV USEDCHILD
rpool 126G 18.3G 0 35.5K 0 18.3G
rpool/ROOT 126G 15.3G 0 18K 0 15.3G
rpool/ROOT/snv_106 126G 86.1M 0 86.1M 0 0
rpool/ROOT/snv_b108 126G 15.2G 5.89G 9.28G 0 0
rpool/dump 126G 1.00G 0 1.00G 0 0
rpool/export 126G 37K 0 19K 0 18K
rpool/export/home 126G 18K 0 18K 0 0
rpool/swap 128G 2G 0 193M 1.81G 0


Pool Space Accounting
• Pool space accounting changed in b128, along with
deduplication
• Compression, deduplication, and raidz complicate pool
accounting (the numbers are correct, the interpretation is
suspect)
• Capacity planning for remaining free space can be challenging

$ zpool list zwimming
NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT
zwimming 100G 43.9G 56.1G 43% 1.00x ONLINE -


zfs vs zpool Space Accounting
• zfs list != zpool list
• zfs list shows space used by the dataset plus space for
internal accounting
• zpool list shows physical space available to the pool
• For simple pools and mirrors, they are nearly the same
• For RAIDZ, RAIDZ2, or RAIDZ3, zpool list will show space
available for parity

Users will be confused about reported space available


NexentaStor Snapshot Services


Accessing Snapshots
• By default, snapshots are accessible in .zfs directory
• Visibility of .zfs directory is tunable via snapdir property
✦ Don't really want ﬁnd to ﬁnd the .zfs directory
• Windows CIFS clients can see snapshots as Shadow Copies
for Shared Folders (VSS)

# zfs snapshot rpool/export/home/relling@20090415
# ls -a /export/home/relling
…
.Xsession
.xsession-errors
# ls /export/home/relling/.zfs
shares snapshot
# ls /export/home/relling/.zfs/snapshot
20090415
# ls /export/home/relling/.zfs/snapshot/20090415
Desktop Documents Downloads Public


Time Slider - Automatic Snapshots
• Solaris feature similar to OSX's Time Machine
• SMF service for managing snapshots
• SMF properties used to specify policies: frequency (interval)
and number to keep
• Creates cron jobs
• GUI tool makes it easy to select individual ﬁle systems
• Tip: take additional snapshots for important milestones to
avoid automatic snapshot deletion
Service Name Interval (default) Keep (default)
auto-snapshot:frequent 15 minutes 4
auto-snapshot:hourly 1 hour 24
auto-snapshot:daily 1 day 31
auto-snapshot:weekly 7 days 4
auto-snapshot:monthly 1 month 12

Nautilus
• File system views which can go back in time


Resilver & Scrub
• Can be read IOPS bound
• Resilver can also be bandwidth bound to the resilvering device
• Both work at lower I/O scheduling priority than normal
work, but that may not matter for read IOPS bound devices
• Dueling RFEs:
✦ Resilver should go faster
✦ Resilver should go slower
✤ Integrated in b140


Time-based Resilvering
• Block pointers contain
birth txg number
73
• Resilvering begins with 73

oldest blocks ﬁrst
73 55
• Interrupted resilver will still 73 27

result in a valid ﬁle system 27 27
68 73
view 73 68

Birth txg = 27
Birth txg = 68
Birth txg = 73


ACL – Access Control List
• Based on NFSv4 ACLs
• Similar to Windows NT ACLs
• Works well with CIFS services
• Supports ACL inheritance
• Change using chmod
• View using ls
• Some changes in b146 to make behaviour more consistent


Checksums for Data
• DVA contains 256 bits for checksum
• Checksum is in the parent, not in the block itself
• Types
✦ none
✦ ﬂetcher2: truncated 2nd order Fletcher-like algorithm
✦ ﬂetcher4: 4th order Fletcher-like algorithm
✦ SHA-256
• There are open proposals for better algorithms


Checksum Use

Pool Algorithm Notes
Uberblock SHA-256 self-checksummed
Metadata fletcher4
Labels SHA-256
Gang block SHA-256 self-checksummed

Dataset Algorithm Notes
Metadata fletcher4
fletcher2
Data zfs checksum parameter
fletcher4 (b114)
fletcher2
ZIL log self-checksummed
fletcher4 (b135)
Send stream fletcher4
Note: ZIL log has additional checking beyond the checksum

Compression
• Builtin
✦ lzjb, Lempel-Ziv by Jeff Bonwick
✦ gzip, levels 1-9
✦ lze, run-length encoding (zeros)
• Extensible
✦ new compressors can be added
✦ backwards compatibility issues
• Uses taskqs to take advantage of multi-processor systems
• Do you have a better compressor in mind?
✦ http://richardelling.blogspot.com/2009/08/justifying-new-
compression-algorithms.html

Solaris cannot boot from gzip compressed root (RFE is open)

Encryption
• Placeholder – details TBD, reported putback in b148
• http://opensolaris.org/os/project/zfs-crypto
• Complicated by:
✦ Oracle
✦ Block pointer rewrites
✦ Deduplication


Quotas
• File system quotas
✦ quota includes descendants (snapshots, clones)
✦ refquota does not include descendants
• User and group quotas
✦ b114, Solaris 10 10/09 (patch 141444-03 or 141445-03)
✦ Works like refquota, descendants don't count
✦ Not inherited
✦ zfs userspace and groupspace subcommands show quotas
✤ Users can only see their own and group quota, but can delegate
✦ Managed like properties
✤ [user|group]quota@[UID|username|SID name|SID number]
✤ not visible via zfs get all


zpool.cache
• Old way
✦ mount /
✦ read /etc/[v]fstab
✦ mount file systems
• ZFS
✦ import pool(s)
✦ find mountable datasets and mount them
• /etc/zpool.cache is a cache of pools to be imported at
boot time
✦ No scanning of all available LUNs for pools to import
✦ Binary: dump contents with zdb -C
✦ cachefile property permits selecting an alternate zpool.cache
✤ Useful for OS installers
✤ Useful for clusters, where you don't want a booting node to

Mounting ZFS File Systems
• By default, mountable file systems are mounted when the pool
is imported
✦ Controlled by canmount policy (not inherited)
✤ on – (default) file system is mountable
✤ off – file system is not mountable
- if you want children to be mountable, but not the parent
✤ noauto – file system must be explicitly mounted (boot
environment)
• Can zfs set mountpoint=legacy to use /etc/vfstab
• By default, cannot mount on top of non-empty directory
✦ Can override explicitly using zfs mount -O or legacy
mountpoint
• Mount properties are persistent, use zfs mount -o for
temporary changes

Solaris Swap and Dump
• Swap
✦ Solaris does not have automatic swap resizing
✦ Swap as a separate dataset
✦ Swap device is raw, with a refreservation
✦ Blocksize matched to pagesize: 8 KB SPARC, 4 KB x86
✦ Don't really need or want snapshots or clones
✦ Can resize while online, manually
• Dump
✦ Only used during crash dump
✦ Preallocated
✦ No refreservation
✦ Checksum off
✦ Compression off (dumps are already compressed)


recordsize
• Dynamic
✦ Max: 128 kBytes
✦ Min: sector size (usually 512 bytes)
✦ Power of 2
• For most workloads, don't worry about it
• For ﬁxed size workloads, can set to match workloads
✦ Databases
✦ iSCSI Zvols serving NTFS or ext3 (use 4 or 8 KB)
• File systems or Zvols
• zfs set recordsize=8k dataset


Delegated Administration
• Fine grain control
✦ users or groups of users
✦ subcommands, parameters, or sets
• Similar to Solaris' Role Based Access Control (RBAC)
• Enable/disable at the pool level
✦ zpool set delegation=on mypool (default)
• Allow/unallow at the dataset level
✦ zfs allow relling snapshot mypool/relling
✦ zfs allow @backupusers snapshot,send
mypool/sw
✦ zfs allow mypool/relling


Delegation Inheritance
• Beware of inheritance
• Local
✦ zfs allow -l username snapshot mypool
• Local + descendants
✦ zfs allow -d username mount mypool

Make sure permissions are set at the correct level

Delegatable Subcommands

• allow • receive
• clone • rename
• create • rollback
• destroy • send
• groupquota • share
• groupused • snapshot
• mount • userquota
• promote • userused


Delegatable Parameters

• aclinherit • nbmand • sharesmb
• aclmode • normalization • snapdir
• atime • quota • userprop
• canmount • readonly • utf8only
• casesensitivity • recordsize • version
• checksum • refquota • volsize
• compression • refreservation • vscan
• copies • reservation • xattr
• devices • setuid • zoned
• exec • shareiscsi
• mountpoint • sharenfs


Nondelagable Parameters
• mlslabel


Browser User Interface
• Solaris 10 – WebConsole
• Nexenta
• OpenStorage


Solaris WebConsole


NexentaStor


Oracle’s Sun OpenStorage


General Comments
• In general, performs well out of the box
• Standard performance improvement techniques apply
• Lots of DTrace knowledge available
• Typical areas of concern:
✦ ZIL
✤ check with zilstat, improve with slogs
✦ COW “fragmentation”
✤ check iostat, improve with L2ARC
✦ Memory consumption
✤ check with arcstat
✤ set primarycache property
✤ can be capped
✤ can compete with large page aware apps
✦ Compression, or lack thereof

Hybrid Storage Pool

Adaptive Replacement Cache (ARC)

separate Main Pool
Main Pool Level 2 ARC
intent log

Write optimized HDD
HDD Read optimized
device (SSD) HDD device (SSD)

Size (GBytes) 1 - 10 GByte large big
Cost write iops/$ size/$ size/$

Use sync writes persistent storage read cache

Performance secondary
low-latency writes low-latency reads
optimization
Need more
speed? stripe more, faster devices stripe


Storage Architecture

Log Main Pool L2ARC

HDD
Minimal

HDD
Good SSD HDD
mirror

HDD HDD HDD SSD
Better SSD
SSD raidz, raidz2, raidz3
mirror

SSD SSD HDD
HDD HDD
HDD HDD
HDD SSD SSD
Best SSD SSD
mirror mirror mirror mirror mirror stripe
stripe stripe


Storage Architecture

Log Main Pool L2ARC

HDD HDD HDD SSD
Better SSD
SSD raidz, raidz2, raidz3
mirror

SSD SSD HDD
HDD HDD
HDD HDD
HDD SSD SSD
Best SSD
mirror
SSD
mirror mirror mirror mirror stripe
stripe stripe

SSD SSD SSD
Extreme raidz2, raidz3

Over the Top SSD
HDD SSD
HDD SSD
HDD
mirror mirror mirror
stripe


ZIL Performance : NFS
• Big performance increases demonstrated
✦ especially with SSDs
✦ for RAID arrays with nonvolatile RAM cache, not so much
• NFS servers
✦ 32kByte threshold (zfs_immediate_write_sz) also corresponds
to NFSv3 write size
✤ May cause more work than needed
✤ See CR6686887
✦ Use fast, separate log device


iSCSI Performance Hiccup
• Prior to b107, iSCSI over Zvols didn’t properly handle sync
writes
• b107-b113, iSCSI over Zvols made all writes sync (read: slow)
✦ Workaround: enable write cache enable in the iSCSI target, see
CR6770534
✦ OpenSolaris 2009.06 is b111
• b114, write cache enable works automatically iSCSI over Zvol
• NexentaStor
✦ not affected by hiccup
✦ easy management of iSCSI performance tunables


ARC Size
• The ARC maximum size of 7/8 memory or memory - 1 GB is
effectively 7/8 memory
• For storage servers, this can be too conservative
• If RAM > 8 GB, consider changing swapfs limits
✦ swapfs_minfree limit is 1/8 memory, by default
✦ units are pages (4 KB for x86, 8 KB for SPARC)
✦ set swapfs_minfree=8192


ZIL Performance : Databases
• The logbias property can be set on a dataset to control
threshold for writing to pool when a slog is used
✦ logbias=latency (default) all writes go to slog
✦ logbias=throughput all writes go to pool
✦ Settable on-the-fly
✤ Consider changing policy during database loads
• Can have different sync policies for logs and data
✦ Oracle, separate latency-sensitive redo log traffic from
✤ Redo logs: logbias=latency
✤ Indexes: logbias=latency
✤ Data files: logbias=throughput
✦ MySQL with InnoDB
✤ logbias=latency


More ZIL Performance : Databases

• I/O size inflation
✦ Once a file grows to use a block size, it will keep that block size
✤ Block size is capped by recordsize
✤ recordsize is a power of 2: 512 bytes, 1 KB, 2 KB, 4 KB, ... 128 KB
✦ Can be inefficient if the workload is sync and writes variable
sized data
• Oracle performance work: Roch reports 40% improvement
for JBOD (HDD) + separate log (SSD) with:
File system or Zvol Role recordsize logbias
data files 8 KB throughput
redo logs 128 KB (default) latency (default)
indices 8-32 KB? latency (default)


Network Architecture

NAS Network Clients
Minimal

Good

Better HA cluster

Best HA cluster

November 8, 2010 USENIX LISA’10

Compression Performance
• Metadata – you won't notice
• Data
✦ LZJB is barely noticeable
✦ gzip-9 can be very noticeable
• Geriatric hardware ???


vdev Cache
• vdev cache occurs at the SPA level
✦ readahead
✦ 10 MBytes per vdev
✦ only caches metadata (b70 or later)
• Stats collected as Solaris kstats

# kstat -n vdev_cache_stats
module: zfs instance: 0
name: vdev_cache_stats class: misc
crtime 38.83342625
delegations 14030
hits 105169
misses 59452
snaptime 4564628.18130739

Hit rate = 59%, not bad...

Intelligent Prefetching
• Intelligent ﬁle-level prefetching occurs at the DMU level
• Feeds the ARC
• In a nutshell, prefetch hits cause more prefetching
✦ Read a block, prefetch a block
✦ If we used the prefetched block, read 2 more blocks
✦ Up to 256 blocks
• Recognizes strided reads
✦ 2 sequential reads of same length and a ﬁxed distance will be
coalesced
• Fetches backwards
• Seems to work pretty well, as-is, for most workloads


Unintelligent Prefetch?
• Some workloads don't do so well with intelligent prefetch
✦ CR6859997, zfs caching performance problem, ﬁxed in NV b124
• Look for time spent in zfetch_* functions using lockstat
✦ lockstat -I sleep 10
• Easy to disable in mdb for testing on Solaris
✦ echo zfs_prefetch_disable/W0t1 | mdb -kw
• Re-enable with
✦ echo zfs_prefetch_disable/W0t0 | mdb -kw
• Set via /etc/system
✦ set zfs:zfs_prefetch_disable = 1


Impedance Matching
• RAID arrays & columns
• format label offsets
✦ Older Solaris starting block = 34
✦ Newer Solaris starting block = 256
• Verify with format or prtvtoc


I/O Queues
• By default, up to 35 I/Os are queued to each vdev
✦ b127 changes default to 4-10, limit based on response
✦ Upper limit tunable with zfs_vdev_max_pending, set to 10 with:
✦ echo zfs_vdev_max_pending/W0t10 | mdb -kw
• Implies that more vdevs is better
✦ Consider avoiding RAID array with a single, large LUN
✦ Beware: ZFS doesn’t know two vdevs can be on same disk
• ZFS I/O scheduler loses control once iops are queued
✦ CR6471212 proposes reserved slots for high-priority iops
• May need to match queues for the entire data path
✦ zfs_vdev_max_pending
✦ Fibre channel, SCSI, SAS, SATA driver
✦ RAID array controller
• Fast disks → small queues, slow disks → larger queues

Slow disk?
• What is going on here?

extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
5948.9 349.3 40322.3 5238.1 0.1 16.7 0.0 2.7 0 330 c9
3.7 0.0 230.7 0.0 0.0 0.1 0.0 13.5 0 2 c9t1d0
845.0 0.0 5497.4 0.0 0.0 0.9 0.0 1.1 1 32 c9t2d0
3.8 0.0 230.7 0.0 0.0 0.0 0.0 10.6 0 1 c9t3d0
845.2 0.0 5495.4 0.0 0.0 0.9 0.0 1.1 1 32 c9t4d0
3.8 0.0 237.1 0.0 0.0 0.0 0.0 10.4 0 1 c9t5d0
841.4 0.0 5519.7 0.0 0.0 0.9 0.0 1.1 1 32 c9t6d0
3.8 0.0 237.3 0.0 0.0 0.0 0.0 9.2 0 1 c9t7d0
843.5 0.0 5485.2 0.0 0.0 0.9 0.0 1.1 1 31 c9t8d0
3.7 0.0 230.8 0.0 0.0 0.1 0.0 15.2 0 2 c9t9d0
850.2 0.0 5488.6 0.0 0.0 0.9 0.0 1.1 1 31 c9t10d0
3.1 0.0 211.2 0.0 0.0 0.0 0.0 13.2 0 1 c9t11d0
847.9 0.0 5523.4 0.0 0.0 0.9 0.0 1.1 1 31 c9t12d0
3.1 0.0 204.9 0.0 0.0 0.0 0.0 9.6 0 1 c9t13d0
847.2 0.0 5506.0 0.0 0.0 0.9 0.0 1.1 1 31 c9t14d0
3.4 0.0 224.1 0.0 0.0 0.0 0.0 12.3 0 1 c9t15d0
0.0 349.3 0.0 5238.1 0.0 9.9 0.0 28.4 1 100 c9t16d0


Slow disk?

NAME STATE READ WRITE CKSUM
tank DEGRADED 0 0 0
raidz2-0 ONLINE 0 0 0
c9t1d0 ONLINE 0 0 0
c9t3d0 ONLINE 0 0 0
c9t5d0 ONLINE 0 0 0
c9t7d0 ONLINE 0 0 0
c9t9d0 ONLINE 0 0 0
c9t11d0 ONLINE 0 0 0
raidz2-1 DEGRADED 0 0 0
c9t2d0 ONLINE 0 0 0
c9t4d0 ONLINE 0 0 0
c9t6d0 ONLINE 0 0 0
c9t8d0 ONLINE 0 0 0
replacing-7 DEGRADED 0 0 0
c9t16d0s0/o FAULTED 0 0 0 corrupted data
c9t16d0 ONLINE 0 0 0 1.28G resilvered


Slow disk?
• IOPS bound, big time
• zfs_vdev_max_pending too big?

extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
428.0 103.8 27361.9 65.9 0.0 14.4 0.0 27.1 0 446 c9
28.5 0.0 1824.1 0.0 0.0 0.3 0.0 10.6 0 23 c9t1d0
28.7 0.0 1836.9 0.0 0.0 0.3 0.0 9.6 0 23 c9t2d0
28.2 0.0 1804.9 0.0 0.0 0.3 0.0 10.6 0 23 c9t3d0
28.5 0.0 1824.1 0.0 0.0 0.3 0.0 10.6 0 24 c9t4d0
28.5 0.0 1824.1 0.0 0.0 0.3 0.0 10.8 0 23 c9t5d0
28.3 0.0 1805.0 0.0 0.0 0.3 0.0 10.1 0 22 c9t6d0
28.8 0.0 1843.3 0.0 0.0 0.3 0.0 10.8 0 23 c9t7d0
29.2 0.0 1862.6 0.0 0.0 0.3 0.0 10.0 0 23 c9t8d0
28.0 0.0 1792.1 0.0 0.0 0.3 0.0 10.7 0 23 c9t9d0
28.3 0.0 1805.0 0.0 0.0 0.3 0.0 9.8 0 21 c9t10d0
28.3 0.0 1811.3 0.0 0.0 0.3 0.0 10.8 0 24 c9t11d0
29.2 0.0 1862.6 0.0 0.0 0.3 0.0 9.8 0 23 c9t12d0
27.8 0.0 1779.3 0.0 0.0 0.3 0.0 10.2 0 23 c9t13d0
29.1 0.0 1856.2 0.0 0.0 0.3 0.0 9.8 0 23 c9t14d0
28.6 0.0 1830.5 0.0 0.0 0.3 0.0 10.9 0 23 c9t15d0
0.0 103.8 0.0 65.9 0.0 10.0 0.0 96.2 0 100 c9t16d0


COW Penalty
• COW can negatively affect workloads which have updates and
sequential reads
✦ Initial writes will be sequential
✦ Updates (writes) will cause seeks to read data
• Lots of people seem to worry a lot about this
• Only affects HDDs
• Very difﬁcult to speculate about the impact on real-world apps
✦ Large sequential scans of random data hurt anyway
✦ Reads are cached in many places in the data path
✦ Databases can COW, too
• Sysbench benchmark used to test on MySQL w/InnoDB
engine
✦ One hour read/write test


COW Penalty

Performance seems to level at about 25% penalty

Results compliments of Allan Packer & Neelakanth Nadgir
http://blogs.sun.com/realneel/resource/MySQL_Conference_2009_ZFS_MySQL.pdf


About Disks...
• Disks still the most important performance bottleneck
✦ Modern processors are multi-core
✦ Default checksums and compression are computationally
efﬁcient
Average
Max Size Rotational Average Seek
Disk Size RPM (GBytes) Latency (ms) (ms)
HDD 2.5” 5,400 500 5.5 11
HDD 3.5” 5,900 2,000 5.1 16
HDD 3.5” 7,200 1,500 4.2 8 - 8.5
HDD 2.5” 10,000 300 3 4.2 - 4.6
HDD 2.5” 15,000 146 2 3.2 - 3.5
SSD (w) 2.5” N/A 800 0 0.02 - 0.25
SSD (r) 2.5” N/A 1,000 0 0.02 - 0.15

Where is my disk?


DirectIO
• UFS forcedirectio option brought the early 1980s design of
UFS up to the 1990s
• ZFS designed to run on modern multiprocessors
• Databases or applications which manage their data cache may
beneﬁt by disabling ﬁle system caching
• Expect L2ARC to improve random reads (secondarycache)
• Prefetch disabled by primarycache=none|metadata
UFS DirectIO ZFS
primarycache=metadata
Unbuffered I/O
primarycache=none
Concurrency Available at inception
Improved Async I/O code path Available at inception


RAID-Z Bandwidth
• Traditional RAID-Z had a “mind the gap” feature
• Impacts possible bandwidth
• Mirrors could show higher bandwidth
• Now RAID-Z shows better bandwidth, when channel
bandwidth is the constrained resource


Troubleshooting

205

Checking Status
• zpool status
• zpool status -v
• Solaris
✦ fmadm faulty
✦ fmdump
✦ fmdump -ev or fmdump -eV
✦ format or rmformat


ﬂash
Copy on Write



z

ﬂash
Copy on Write

What happens if the uberblock is updated prior to leaves?



z

What if flush is ignored?
• Some devices ignore cache flush commands (!)
✦ Virtualization default=ignore flush:VirtualBox, others?
✦ Some USB/Firewire to IDE/SATA converters
• Problem: uberblock could be updated before leaves
• Symptom: can’t import pool, uberblock points to random data
• Affected systems
✦ Many OSes and file systems
✦ Laptops - rarely because of battery
✦ Enterprise-class systems - rarely because of power redundancy
and solid design
✦ Desktops - more frequently


What if ﬂush is ignored?
• Solution
✦ Check integrity of recent transaction groups
✦ If damaged, rollback to older uberblock
✤ zpool import -F
✦ Best recovery methods in later releases


Can't Import Pool?
• Check device paths with zpool import
✦ Be aware of /etc/zfs/zpool.cache
✦ May need zpool -d directory option
✦ “phantom paths”?
• Check for 4 labels
✦ zdb -l /dev/dsk/c0t0d0s0
• Try rollback import
✦ zpool imports -F poolname

Beware of device short names: c0d0 != c0d0s0


Slow Pool Import?
• Case: zvols with snapshots
✦ Symptom: reboot or zpool import is really
slllooooowwwwwww...
✦ Cause: inefﬁcient incrementing over all zvols creating entries
in /dev/zvol/dsk
✦ Cure: CR6761786 integrated in b125
• Case: system is iSCSI initiator
✦ Symptom: format or zpool import wait for tens of seconds
to start
✦ Cause: iSCSI target presents in-band management LUs that are
slow to respond to SCSI inquiry
✦ Cure: map iSCSI targets so that in-band management is not
presented to ZFS hosts


File System Mounts B0rken?
• Prevention
✦ Avoid complex heirarchies (KISS)
✦ Be aware of legacy mounts
✦ Be aware of alternate boot environments (Solaris)
• Check mountpoint properties
✦ zfs list -o name,mountpoint
• Shared ﬁle systems
✦ Be aware of inherited shares
✦ Some clients do not mirror mount (Linux)
✦ NFS version differences?
✦ Check name services


Can't Boot?
• Check if BIOS/OBP supports booting from device
• Make sure LUN has SMI label, not EFI
✦ Common mistake when mirroring root
✦ OK: zpool attach rpool c0t0d0s0 c0t1d0s7
✦ Not OK: zpool attach rpool c0t0d0s0 c0t1d0
• installboot?
• grub issues
✦ Boot environments usually handled by grub
✦ Check grub menu.lst
• Know how to do a failsafe boot
• Be aware of LiveCD import
• Be aware of zpool.cache interactions


Stuck at grub> Prompt?

• Latent fault
• Typical scenario
✦ Upgrade NexentaStor
✦ Go to expert mode shell
✦ # zpool upgrade syspool
✦ ... wait a few months ...
✦ Reboot
✦ grub>
• What happened?
✦ grub has a miniature ZFS implementation that can’t grok
syspool version and menu.lst ﬁle is in the syspool
• Recovery
✦ Boot CD


Future Plans
• Stay tuned...


Now you know...
• ZFS structure: pools, datasets
• Data redundancy: mirrors, RAIDZ, copies
• Data veriﬁcation: checksums
• Data replication: snapshots, clones, send, receive
• Hybrid storage: separate logs, cache devices, ARC
• Security: allow, deny, encryption
• Resource management: quotas, references, I/O scheduler
• Performance: latency, COW, zilstat, arcstat, logbias, recordsize
• Troubleshooting: FMA, zdb, importance of cache ﬂushes


Thank You!

Questions?

Richard.Elling@RichardElling.com
Richard.Elling@Nexenta.com

218

ZFS Tutorial LISA 2011

More Related Content

What's hot (19)

Viewers also liked (9)

Similar to ZFS Tutorial LISA 2011 (20)

Recently uploaded (20)

ZFS Tutorial LISA 2011

Editor's Notes