LCA 2014 project-builder.org presentation

Project Builder:
A GPL continuous packaging solution
1

Bruno Cornec

V2.1 6th of January 2014
© Copyright 2012 Hewlett-Packard Development Company, L.P. - Licensed under CC-by-SA 3.0

HP EMEA Open Source Profession Lead

Introducing Myself
Software engineering and Unices since 1988

●

–

Mostly Configuration Management Systems, Build systems, quality tools, on multiple commercial Unix
systems

–

Discover Open Source & Linux (OSL) & first contributions in 1993

–

Full time on OSL since 1995, first as HP reseller then @HP

Currently:

●

–

Master Technology Architect on OSL for the HP/Intel Solution Center, Grenoble

–

HP OSL Advocate and Converged Infrastructure Ambassador

–

HP EMEA OSL Profession Lead

–

Solutions Linux Conference and AFUL board member. Conferences at WW level in LinuxCon, Linux.conf.au

–

MondoRescue, Project-Builder.org, UUWL and PUSK Project Lead

–

LinuxCOE, mrepo, tellico, rinse, fossology, collectl contributor

–

FOSSBazaar and OSL Governance enthusiast

–

Mandriva, Mageia, Fedora packager

2

●

And also:
–

Amateur singer (Alto / Tenor) and recorder player since 1976 and Choir director since 1987

–

CD collector since 1981 (5000+ and counting) – Concert attendance since 1976

–

Amateur photograph since 1976


What is project-builder.org’s goal ?

Make upstream projects
life easier with regards to
packaging their software
3


What is project-builder.org’s itch ?

Make *my*
life easier with regards to
packaging *my* software
4


Users / Sysadmins want packaged software
●
Ease of use: GUI, CLI
>= 90%
●Distribution compliance
●Smooth Integration with
deployment tools

U
5

< 10%
< 1%


Users / Sysadmins want packaged software
●
Ease of use: GUI, CLI
>= 90%
●Distribution compliance
●Smooth Integration with
deployment tools

U

< 10%
< 1%

6

D

Lag between SW and Pkg
availability
●Test of alpha/beta/VCS SW
●Too many:
● distributions,
● versions,
● package formats,
● tools to generate packages
● tools to manage repositories
●

Upstream benefits from Continuous packaging
Packaging should be a project concern as well as coding, testing, installing, ....
especially for smaller projects
Use packaging as your only way of delivery (not a dream)
Minial overhead, slightly longer than providing a tgz, maximum benefit:
Consistancy and reproduceability for devs and users
Distribution & deployment server integration,
Improved deployment without risk of screwing up the system
7

Packaging is also a marketing activity for the upstream project based on a
technical content. It's an easy way to extend your user base, and improve your
community relationship and is a “competitive advantage”.
New mantra: “Package early, package always”
THE SOLUTION IS INDEED CONTINUOUS PACKAGING (whatever the tool)


Continuous Packaging Architecture
Local Build Server (may host VMs and/or VEs)
Packagers

VM or VE Build

Repository
Server

Build +
metadata

Local build

8

Project
RM Build
Farm
Developers

(may host VMs)


Remote build

Provisioning the Continuous Packaging Cloud
Local Build Server (may host VMs and/or VEs)
VM or VE Build

Build +
9

metadata

Local build

Project
RM Build Farm (may
host VMs)

Remote build

Repository
Server

Goals
Project-builder main goal is to help you package continuously being
agnostic:
VCS agnostic: no VCS but guys it's 21st century now, SVN, CVS,
Mercurial, GIT and GIT/SVN, SVK....
OS agnostic: Linux: RPM, deb, ebuild, slack based, ... 150+ distro
tuples made and counting – repositories for yum, urpmi, apt. Solaris
pkg. HP-UX sd in roadmap
10

Build environment agnostic: local, VM (QEMU, KVM...), VE
(rpmbootstrap, rinse, mock, debootstrap...), RM (external build farm)
No project impact: preserves the md5sum of the delivered upstream
sources. Can be completely external to the upstream project.
Avoids duplication of code and metadata
THE SOLUTION IS INDEED CONTINUOUS PACKAGING (with projectbuilder.org !)

Goodies
Project-builder provides additional goodies:
Easy VMs/VEs/RMs installation/setup. More on that later
Macro system with perl variables to avoid duplication
Skeleton generation to help starting
Manages package delivery up to your repository (ssh based) with
repository management (yum, apt, urpmi)
Manages announces on mailing lists
11

Integration of tests in the process
Manages patches/additional sources when not upstream
Checks validity of packages built (lintian, rpmlint)
Easy creation of new versions for upstream management
Manages website delivery

VM, VE & RM Management
VE Creation:
debootstrap for .deb based ones.
rpmbootstrap (modeled after debootstrap): create chroot for multiple
RPM based distributions (Mageia, Fedora, Mandriva, CentOS,
OpenSuSE)
VM|RM Creation:
KVM or QEMU iso install possibilities.
12

LinuxCOE/OpenStack as provisioning tool for your VM|RM
VM|VE|RM setup: adds pb tools and build account
VM|VE Snapshot feature – Update of VM|VE|M optional
Building in VM|VE|RM == building locally (build2vm|ve|rm vs build2pkg).
ssh communication for VM/RM

Metadata Management
Packaging process independent of the project and external
Metadata isolated - No duplication

13


Configuration file usage
Macro example:
./rpmbootstrap/pbfilter/all.pbf:filter PBSUMMARY = rpmbootstrap is a tool similar
to debootstrap for RPM based distributions
./rpmbootstrap/rpm/rpmbootstrap.spec:Summary: PBSUMMARY
./rpmbootstrap/deb/control:Description: PBSUMMARY
./rpmbootstrap/pkg/pkginfo:NAME="PBSUMMARY"
14

Instantiation possible from distro-ver-arch, distro-ver, distro,
distro-family, distro-type, os
Similar approach for all other parameters (Cf: man pb.conf)


Commands
[cms|sbx]2build: Create tar files for the project under your CMS. Parameters are
packages to build if not using default list
build2pkg: Create packages for your running distribution
build2ssh: Send the tar files to a SSH host
pkg2ssh: Send the packages built to a SSH host
build2vm|ve|rm: Create packages in VMs/VEs/RMs, launching them if needed and
send those packages to a SSH host once built
launchvm: Launch one virtual machine
script2vm|ve|rm: Launch a VM/VE/RM if needed and executes a script on it
test2vm|ve|rm: Test inside a VM/VE/RM
newvm|ve: Create a new VM/VE/RM
setupvm|ve|rm: Setup a VM/VE/RM for pb usage
newver: Create a new version of the project derived from the current one
newproj: Create a new project and a template set of configuration files under
pbconf

15


Multi platform packaging cycle

VMs/VEs/RMs

Developers

UUWL

Packages

17

VM/VE/RM

Packagers

Q & A up to now ?
18


DEMONSTRATION
19


Special announce for LCA 2014 !

20


More Special announce for LCA 2014 !
I hate time release delivery
Of course, 0.12.3 has bugs ;-)
Lack of time to test thoroughly
So I hate time release delivery
So expect a 0.12.4 later this week, when more tests have been done !
So back to feature set release in fact
21

Did I say I hate time release delivery ?


To be done
LSB chroot support. (or not)
LXC support (which one ?)
Support for libvirt, virsh, openstack API
Other CMS (Bazaar, …) only when/if needed
Other VMs (VMWare, Xen, ...) only when/if needed
Multiple delivery means
22

Look at interactions with Buildbot
Add non-interactive signature support for .deb
Config-Model for configuration file management
REST API and Web Interface would be nice. Any contributor ?

Learn Project-Builder.org
Start with the Lab (63 pages)
Use man (pb, pb.conf + 8
ProjectBuilder::* man pages)
Use the mailing-list pb-announce
and pb-devel at
http://www.mondorescue.org/sympa

Use examples from
23

http://trac.project-builder.org/browser/projects/


Web Resources
Project-Builder Web site / Trac / Wiki:
●

http://www.project-builder.org

●

http://trac.project-builder.org

Projects using project-builder.org:
●

http://www.project-builder.org ( of course :-)

●

http://www.mondorescue.org

24

●

Http://www.linuxcoe.org

●

http://www.fossology.org


Related tools
Project-Builder is mostly suited for upstream projects wanting to package their
applications
Distributions provide each their build tools
● SuSE: Open Build Service (Multi distro, Web based, BaaS)
● Fedora (Koji)
● Mandriva/Mageia (Youri, mdvsys, mgarepo...)
Other complementary tools:
● Buildbot
● KVM/QEMU
● rpmbootstrap, rinse, mock
● Parallel::ForkManager, DBI, DB::SQLite, File::MimeInfo, Mail::Sendmail
25

Project-Builder.org is one of the tool of the vcs-pkg.org initiative.


Contact - Thanks
Bruno.Cornec@hp.com
(Open Source and Linux Technology
Architect at the HP/Intel Solution Center)
http://www.hp.com/linux
http://opensource.hp.com

Thanks goes to:

26

Linus Torvalds, Richard Stallman, Eric
Raymond, Nat Makarevitch, René
Cougnenc, Eric Dumas, Rémy Card,
Bdale Garbee, Bryan Gartner, Craig
Lamparter, Lee Mayes, Gallig Renaud,
Andree Leidenfrost, Phil Robb, Bob
Gobeille, Martin Michlmayr among
others, for their work and devotion to
the Open Source Software cause... and
my family for their patience :-)


”Changes are never easy to make.
There is comfort and safety in
tradition, but change must come, no
matter how painful or expensive it
may be.”
Bill Hewlett

LCA 2014 project-builder.org presentation

More Related Content

What's hot (17)

Viewers also liked (8)

Similar to LCA 2014 project-builder.org presentation (20)

More from Bruno Cornec (20)

Recently uploaded (20)

LCA 2014 project-builder.org presentation