Balena: a Moby-based container engine for IoT
0 likes936 views
The document outlines the history and features of balena, released in October 2017, highlighting its small footprint, minimal disk writes, and durable image pull capabilities, which enhance efficiency compared to Docker. It details technical aspects such as its architecture, use of the go linker, and implementation of container deltas for optimized bandwidth usage. Challenges include customization difficulties and the entanglement of the Docker name, with a focus on future optimizations for balena.
Balena: a Moby-based container engine for IoT
- 1. Petros Angelatos resin.io Moby Summit - October 2017
- 2. History ● December 2013 ○ Ported Docker 0.7.2 to ARM ● October 2016 ○ Released resinOS 2.0 ● April 2017 ○ Moby Project announcement ● October 2017 ○ Released Balena
- 3. Key features ● Small footprint ○ On average 3x to 4x smaller than Docker CE ● Minimal disk writes ○ On-the-fly extraction from the registry to the layer store ● Durable image pull ○ Disk syncing on every step ● Container deltas ○ 10-70 times more bandwidth efficient than docker pull
- 4. Small footprint ● Busybox-style architecture ○ Deduplicate shared go libs between components ○ The binary includes multiple entrypoints ○ Select appropriate main() function based on argv0 ● Dropped non-IoT relevant features* ○ Swarm ○ Plugins ○ Cloud logging drivers ○ Etc * based on the scenarios we’re focusing on
- 6. Docker architecture containerd runc swarmkit gostdlib dockerd runc swarmkit gostdlib docker-cli runc dockerd gostdlib containerd dockerd gostdlib runc notary logrus logrus ● Size = 4 x (gostdlib) + 2 x (swarmkit) + 2 x (logrus) ...
- 7. Balena architecture containerd runc dockerd gostdlib balena ● Go Linker ensures only required routines are compiled ● There is only one copy of each library ● Entrypoint selects based on argv0 docker-cli entrypoint
- 8. Minimal disk writes ● Serially download each layer ● Extract the files directly in the layer store ● Allows balena to pull under low space situations ● Hash verification still occurs ○ Layer is discarded if a problem is found
- 9. Durable image pull ● Run file.Sync() after every file on disk ● Sync() the filesystem before committing layers ○ This ensures metadata is also synced to disk ● Syncing enables us to purge page cache during pull ○ man 2 posix_fadvise
- 11. Container deltas ● Based on librsync ○ Ported in Golang https://github.com/resin-os/librsync-go ● Output is still a valid docker image ○ Cannot run though ○ Can be pushed and pulled to normal docker registries
- 12. Container deltas layer 1 Image config layer 2 layer 3 layer 4 Source image
- 13. Container deltas layer 1 Image config layer 2 layer 3 layer 4 Source image layer 1 layer 2 layer 3 layer 4 Serialized image stream
- 14. Container deltas layer 1 Image config layer 2 layer 3 layer 4 Source image layer 1 layer 2 layer 3 layer 4 Serialized image stream librsync-go Source fingerprint
- 15. Container deltas layer 1 Image config layer 2 layer 3 layer 4 Source image Source fingerprint
- 16. Container deltas layer 1 Image config layer 2 layer 3 layer 4 Source image Source fingerprint layer 1 Image config’ layer 2’ Target image layer 3’
- 17. Container deltas layer 1 Image config layer 2 layer 3 layer 4 Source image Source fingerprint layer 1 Image config’ layer 2’ Target image Image config’ Delta image layer 3’
- 18. Container deltas layer 1 Image config layer 2 layer 3 layer 4 Source image Source fingerprint layer 1 Image config’ layer 2’ Target image empty Image config’ Delta image Identical layer 3’ Image config’
- 19. Container deltas layer 1 Image config layer 2 layer 3 layer 4 Source image Source fingerprint layer 1 Image config’ layer 2’ Target image empty Image config’ Delta image layer 3’ Image config’
- 20. Container deltas layer 1 Image config layer 2 layer 3 layer 4 Source image Source fingerprint layer 1 Image config’ layer 2’ Target image empty Image config’ Delta image Rsync delta layer 3’ Image config’
- 21. Container deltas layer 1 Image config layer 2 layer 3 layer 4 Source image Source fingerprint layer 1 Image config’ layer 2’ Target image empty Image config’ Delta image layer 3’ Image config’
- 22. Container deltas - How layer 1 Image config layer 2 layer 3 layer 4 Source image Source fingerprint layer 1 Image config layer 2’ Target image empty Image config Delta image layer 3’ Rsync delta Image config’
- 23. Container deltas layer 1 Image config layer 2 layer 3 layer 4 Source image Source fingerprint layer 1 Image config’ layer 2’ Target image empty Image config’ Delta image layer 3’ Image config’
- 24. Container deltas layer 1 Image config layer 2 layer 3 layer 4 Source image Source fingerprint layer 1 Image config’ layer 2’ Target image Image config’ Delta image layer 3’ 10x smaller! Image config’
- 25. Challenges and the future ● Currently it’s a bit hard to customise a moby assembly ○ Will improve as moby becomes more modular ● The Docker name is entangled in various places ○ We’ve done some initial work on separating it out ● Other than that, continue to optimise balena :)
- 26. Thanks!
Editor's Notes
- #3: Having seen IoT devices used in production for tens of millions of hours, we’ve become aware of the unique needs of the embedded world. Until recently, we addressed these by either making small modifications to Docker itself or building larger components outside of it, but that approach had reached its limits. For example having external programs to manipulate /var/lib/docker in a special way The Moby announcement enabled us to move forward in a way that is compatible with the docker community Last Friday we released Balena
- #5: The way we approached this was by making a graph of all dependencies and finding losely connected clusters of packages
- #7: There are a lot of shared libraries between the components
- #8: To do this we have to do align the versions of shared libraries to a version that works with everything The way we set argv0 is by symlinking different names to the balena binary
- #9: Normally the free space needed for a docker pull is the size of the new layers plus the compressed size of those layers, depending on the order they happen to download Balena always uses exactly as much space as the size of the new layers Reduces wear and tear of flash devices
- #10: Two reasons for page cache Application IO performance Preventing device stress that can cause filesystem failures
- #14: This construction needs to be random access
- #15: The fingerprint is a map of hashes to their location in the stream
- #16: We’ll use the source fingerprint to calculate the delta for all the layers
- #18: The delta image contains the same image config as the target, but with a special marker that allows balena to recognise it when it pulls
- #27: Come talk to me if you have questions