JUC Europe 2015: A Reproducible Build Environment with Jenkins
1 like145 views
The document discusses the concept of binary reproducibility in software builds, emphasizing the need for consistent outputs from the same inputs over time. It outlines the challenges to achieving reproducibility, including issues with source code management, compiler behavior, and environmental dependencies. The presentation also details Technisat's approach to building reproducible systems using Jenkins and virtual machine infrastructure to ensure reliable software development and deployment processes.
JUC Europe 2015: A Reproducible Build Environment with Jenkins
- 1. Jenkins World Tour 2015 London, UK, June 2015 1
- 2. #jenkinsconf A Reproducible Build Environment with Jenkins Robert Fach, TechniSat Digital GmbH 2
- 3. #jenkinsconf Footer Agenda • Some words about me • Some words about TechniSat • What is binary code reproducibility? • Why do we need it? • Why is it so difficult to achieve? • Reproducible Build System • Questions?
- 4. #jenkinsconf Footer Some words about me • Degree in computer science • Research background in H/A, Scalability, Big Data • Today Software Integrator, DevOps and Systems Engineer at TechniSat • Contributing to Salt Stack + Salt Cookbook Reviewer • High technical affinity – was contributing in 3D printing area – currently building fpv drones
- 5. #jenkinsconf Footer Some words about TechniSat • TechniSat group develops and produces consumer and information technology products • Founded in 1987, today locations in Germany, Hungary, Poland, USA, China • In-House hardware & software development and production • Tier-1 supplier for well known car manufacturers • Solutions for navigation, tuners, telematics systems
- 6. #jenkinsconf Footer Some words about TechniSat • Develop digital receivers and digital television receivers • Develop customer-specific OEM concepts for electronics and automotive industry Technisat Digital GmbH Dresden Research & Development
- 7. #jenkinsconf Footer What is binary reproducibility? • Same “inputs” produce same binary outputs! – today, tomorrow, next month, in 15 years – Difficulty: Identify & reproduce everything which has influence on the output Sources (C,C+ +,Java, …) Target BinaryTransform Compiler, Scripts, …
- 8. #jenkinsconf Footer Why do we need it? • Customer requests a limited change in a software for a specific software release • Software bugfixes • Reduce or avoid side effects in build environment • Toolchain & build system validation • Achieve a new level of trust to our customers (changes are proof-able) • Independent verification of binaries • Validation of cross-builds vs. native builds • Finding bugs in software modules • Good for automotive spice (e.g., for tracking of changes/requirements)
- 9. #jenkinsconf Footer Why do we need it? Release XY Release XYZ Rebuild with changes
- 10. #jenkinsconf Footer Why is it so difficult to achieve? Sources (C,C+ +,Java, …) Target BinaryTransform Compiler, Scripts, …
- 11. #jenkinsconf Footer Why is it so difficult to achieve? Sources (C,C+ +,Java, …) Target BinaryTransform Compiler, Scripts, … Stable Sources SVN,GIT
- 12. #jenkinsconf Footer Why is it so difficult to achieve? Sources (C,C+ +,Java, …) Target BinaryTransform Compiler, Scripts, … Stable Sources SVN,GIT Version Issues
- 13. #jenkinsconf Footer Why is it so difficult to achieve? Sources (C,C+ +,Java, …) Target BinaryTransform Compiler, Scripts, … Stable Sources SVN,GIT Version Issues Operating System Filesystem Issues
- 14. #jenkinsconf Footer Why is it so difficult to achieve? Sources (C,C+ +,Java, …) Target BinaryTransform Compiler, Scripts, … Stable Sources SVN,GIT Version Issues Operating System Filesystem Issues
- 15. #jenkinsconf Footer Why is it so difficult to achieve? Sources (C,C+ +,Java, …) Target BinaryTransform Compiler, Scripts, … Stable Sources Stable Compiler & Scripts Random Decisions SVN,GIT Version Issues Operating System Filesystem Issues Sorting Issues
- 16. #jenkinsconf Footer Why is it so difficult to achieve? Sources (C,C+ +,Java, …) Target BinaryTransform Compiler, Scripts, … Stable Sources Stable Compiler & Scripts Random Decisions SVN,GIT Version Issues Operating System Filesystem Issues Encoding Issues Date & Time Issues Random Values, UUIDs Sorting Issues
- 17. #jenkinsconf Footer Why is it so difficult to achieve? • Workspace paths compiled into the binary • Random compiler decisions for optimizations, for namespace mangling etc. – frandom-seed • Timestamps, UUIDs in object files and libraries – deterministic mode • Timestamps, dates generated by macros (__DATE__, __TIME__, __TIMESTAMP__) – Faketime/libfaketime (submillis not covered) – Overriding these macros with defined value • File processing order depends on inode ordering • Locales can have influence on sorting order
- 18. #jenkinsconf Footer Why is it so difficult to achieve? • Dealing with complexity & scale – Multiple integration stages – Modular software – Load distribution (virtualization & build slaves)
- 19. #jenkinsconf Footer Why is it so difficult to achieve? • Dealing with complexity & scale – Multiple integration stages – Modular software – Load distribution (virtualization & build slaves) Release Management extra external dependencies
- 20. #jenkinsconf Footer Why is it so difficult to achieve? Sources (C,C+ +,Java, …) Target BinaryTransform Compiler, Scripts, … Sources & Binaries Target BinaryTransform Compiler, Scripts Sources & Binaries Sources & Binaries Sources & Binaries Stage 0 Stage >0 SVN,GIT Artefact Repository
- 21. #jenkinsconf Footer Why is it so difficult to achieve? Sources (C,C+ +,Java, …) Target BinaryTransform Compiler, Scripts, … Sources & Binaries Target BinaryTransform Compiler, Scripts Sources & Binaries Sources & Binaries Sources & Binaries Stage 0 Stage >0 SVN,GIT Artefact Repository Artefact & Dependency Management
- 22. #jenkinsconf Footer Reproducible Build System Release Manager Release Mgmt System Release XY Software Container Module M1 Library L1 Library L2 Library L3 Library L4 Module M2 Module M2 specify defines Software Release Configuration
- 23. #jenkinsconf Footer Reproducible Build System Release XY Jenkins Job source references script references parameters, etc. Jenkins Job source references script references parameters, etc. Jenkins Job source references script references parameters, etc. Jenkins Job source references script references parameters, etc. Release Mgmt System SCM (SVN,GIT) Artefact Repository define map
- 24. #jenkinsconf Footer Reproducible Build System Jenkins Job source references script references parameters, etc. Jenkins Job source references script references parameters, etc. Jenkins Job source references script references parameters, etc. Jenkins Job source references script references parameters, etc. Release XY Infrastructure Jenkins Master XY Jenkins Slave XY Jenkins Slave XY Jenkins Slave XY Release Mgmt System map
- 25. #jenkinsconf Footer Reproducible Build System Release XYZ Infrastructure Salt(manageseverything) KVM Host KVM Cluster (libvirt, OpenStack planned) KVM Host … KVM Host Release Mgmt System SCM Artefact Repository DNS Q/A Systems Jenkins Master XY Jenkins Slave XY Jenkins Slave XY Jenkins Slave XY Release XY Infrastructure Jenkins Master XY Jenkins Slave XY Jenkins Slave XY Jenkins Slave XY
- 26. #jenkinsconf Footer Reproducible Build System • Each release is built in a new bootstrapped vm infrastructure completely described by SALT and Release Mgmt system • Archive complete infrastructure including os, tools, jenkins, etc. • Restore or recreate systems from specification or backup) for rebuild (1st, 2nd and 3rd LOD) • Infrastructure Bootstrapping – node spec. + Salt conf. + release spec. defines release infrastructure – images connect to salt cm to finalize bootstrap according their tasks – jenkins instance is a “clone” from release predecessor – Release Jenkins jobs include a “copy” of the job history of predecessor • Finally map release configuration to Jenkins jobs
- 27. #jenkinsconf Footer Reproducible Build System • Optimizing VM Performance – Use tmpfs inside VMs for fast random I/O filesystems – Try to use nfs read-only cache for saving network bandwidth – Put Jenkins workspace on a dedicated lvm volume on the host instead of on a filesystem based disk – Kvm Optimizations