Rule jenkins with configuration as code

An example on how to introduce the Job DSL Plugin in a real life company
Rule Jenkins with Configuration as Code
Jenkins World, 14.09.2016

….…............................ An example on how to introduce Job DSL in a real life company
What this talk is about
• Configuration as code with the Job DSL Plugin
• How to take it to the next level
• Pitfalls and how to make them foolproof
Intro Configuration as Code Test your codeThe next level

An example on how to introduce Job DSL in a real life company
What this talk is not about
• Job DSL in detail – please refer to:
Adoption of the Job DSL Plugin at Netflix
by Justin Ryan
Configuration as Code: The Job DSL Plugin
by Daniel Spilker at JUC Europe 2015
Jenkins + Groovy with the Job DSL Plugin
by Matt Sheehan at GR8Conf US 2015
….…............................Intro Configuration as Code Test your codeThe next level

Christian Rasp / christian.rasp@PAYBACK.net / @crasp
• Software Developer DevOps
• Jenkins & Nexus Admin
• Build Engineer for CI & CD

PAYBACK is continuously expanding into new countries
2000
2009
2014 2010
2012
2015

PAYBACK – Biggest loyalty program and largest multichannel marketing platform
In 2015, 20 bn coupons were distributed to
customers in Germany
Points to the value of € 338 m were
collected in 2015
95% of all PAYBACK points get
redeemed!
8 out of 10 Germans are familiar
with the PAYBACK brand
Sales of more than € 27 bn
are generated with PAYBACK cards
at partner stores (2015)
The PAYBACK card is the third most
important card in German wallets
(after debit cards and credit cards, TNS Emnid survey, 2014)

In May 2015 PAYBACK was launched in the US under the name "Plenti"

….…............................ An example on how to introduce Job DSL in a real life company
Intro Configuration as Code Test your codeThe next level
Configuration as code with the Job DSL Plugin

All Jenkins job configuration is setup manually:
The initial problem

• What is this job doing?
The initial problem

• Is the job still in use?
The initial problem

• Who is responsible for this job?
The initial problem

• Why is it configured that way?
The initial problem

• What are the standard settings?
The initial problem

• Who changed what and when?
The initial problem

• How to change settings for a group of jobs?
The initial problem

Possible solutions

Possible solutions
• Script console

Possible solutions
• Script console
• Templates

Possible solutions
• Script console
• Templates
• Job generator Plugin

Possible solutions
• Script console
• Templates
• REST API or CLI

Possible solutions
• Script console
• Templates
• REST API or CLI
• Pipeline Plugin

Possible solutions
• Script console
• Templates
• REST API or CLI
• Pipeline Plugin
• Job DSL Plugin

Describe jobs using a Groovy-based language

Describe jobs using a Groovy-based language
mavenJob('HAL9000') {
description('I'm sorry Dave, I'm afraid I can't do that')
logRotator(-1, 30)
jdk('jdk-1.8.0_45')
wrappers {
mavenRelease {
releaseGoals('-Dresume=false release:prepare release:perform')
dryRunGoals('-Dresume=false -DdryRun=true release:prepare')
numberOfReleaseBuildsToKeep(2)
selectScmCredentials(true)
}
}
goals('clean install --batch-mode --update-snapshots')
mavenInstallation('maven-3.2.2')
scm {
svn {
location("${Repositories.DISCOVERY.url()}/dave/trunk") {
checkoutStrategy(SvnCheckoutStrategy.UPDATE_WITH_REVERT)
}
}
[...]

It still does not fit our needs
• How to change settings for subset of jobs?

It still does not fit our needs
• New scripting language
• New DSL reference to learn
• How to change settings for subset of jobs?

How to take it to the next level

Make a developers life as easy as possible
Things a Jenkins user should not need to know about:

• Standard log rotation

• Configuration pitfalls

• Credentials for tech. users

• Naming conventions

• Jenkins Maven magic

• Jenkins Maven magic
• Default settings

So what IS the next level?
Build your own tool with standard software engineering approaches:

• Inheritance

• Inheritance
• Standards

• Inheritance
• Standards
• Convention

• Inheritance
• Standards
• Convention
• Clean project structure

• Inheritance
• Standards
• Convention
• Dependency management

• Inheritance
• Standards
• Convention
• Reviews

• Inheritance
• Standards
• Convention
• Reviews
• Tests!

• Inheritance
• Standards
• Convention
• Reviews
• Tests!
• More tests!

logRotator(-1, 30)
jdk('jdk-1.8.0_45')
wrappers {
mavenRelease {
}
}
scm {
svn {
location("${Repositories.DISCOVERY.url()}/dave/trunk")
}
}
[...]
Reduce the configuration to a bare minimum

jdk('jdk-1.8.0_45')
wrappers {
mavenRelease {
}
}
scm {
svn {
}
}
[...]

wrappers {
mavenRelease {
}
}
scm {
svn {
}
}
[...]

scm {
svn {
}
}
[...]

new MavenJobBuilder(this, 'HAL9000').make {
description("I'm sorry Dave, I'm afraid I can't do that")
svn(Repositories.DISCOVERY, '/trunk')
}

Use builder pattern to create simple job builders
http://mrhaki.blogspot.de/2011/11/groovy-goodness-create-simple-builders.html
Groovy Goodness from
the Blog of MrHaki
….…..................….......Intro Configuration as Code Test your codeThe next level

// Build your own closures to expand Job DSL
private runClosure(Closure runClosure) {
// Create clone of closure for threading access.
Closure runClone = runClosure.clone()
// Set delegate of closure to this builder.
runClone.delegate = this
// And only use this builder as the closure delegate.
runClone.resolveStrategy = Closure.DELEGATE_ONLY
// Run closure code.
runClone()
}
the Blog of MrHaki

// Build your own closures to expand Job DSL
private runClosure(Closure runClosure) {
// Create clone of closure for threading access.
Closure runClone = runClosure.clone()
// Set delegate of closure to this builder.
runClone.delegate = this
// And only use this builder as the closure delegate.
runClone.resolveStrategy = Closure.DELEGATE_ONLY
// Run closure code.
runClone()
}
the Blog of MrHaki
// Delegate everything else to Job DSL
def methodMissing(String name, argument) {
job.invokeMethod(name, (Object[]) argument)
}

the Blog of MrHaki
new JobBuilder(this, 'adjust-permissions').make {
// Own function via runClosure
restrictPermissions('ernie', 'bert', 'jenkins-admins')
// Original Job DSL syntax via methodMissing
steps {
shell('ls -l')
}
}

Set some standards automatically
public class JobBuilder extends Builder {
protected Job job
// Create a new job object
JobBuilder(DslFactory dslFactory, String jobName) {
this.job = dslFactory.job(jobName)
}
}

public class JobBuilder extends Builder {
protected Job job
// Create a new job object
JobBuilder(DslFactory dslFactory, String jobName) {
this.job = dslFactory.job(jobName)
}
// Defines initial state for a job
Job make(Closure additionalConfig) {
// Standards for all jobs
job.logRotator
job.jdk
job.label
// every other configurations
runClosure additionalConfig
job
}
}

job.logRotator
job.jdk
job.label
job
}
[...]
logRotator(-1, 30)
jdk('jdk-1.8.0_45')
[...]
}

job.logRotator
job.jdk
job.label
job
}
[...]
logRotator(-1, 30)
jdk('jdk-1.8.0_45')
[...]
}
What are the standard settings?

Hide complex configuration settings
class MavenJobBuilder extends JobBuilder {
MavenJobBuilder(DslFactory dslFactory, String jobName) {
super(dslFactory.mavenJob(jobName))
}
MavenJob make(Closure additionalConfig) {
job.wrappers {
mavenRelease {
releaseGoals
numberOfReleaseBuildsToKeep
}
}
job.goals
job.mavenInstallation
[...]
super.make(additionalConfig)
job as MavenJob
}
}

job.wrappers {
mavenRelease {
releaseGoals
numberOfReleaseBuildsToKeep
}
}
job.goals
job.mavenInstallation

new MavenJobBuilder(this, 'YourEasyMavenJobHere').make {
[...]
}

Encapsulate common DSL patterns
void svn(Repositories repository,
String loc,
String dir = '.',
SvnCheckoutStrategy strategy = SvnCheckoutStrategy.UPDATE_WITH_REVERT) {

void svn(Repositories repository,
String loc,
String dir = '.',
SvnCheckoutStrategy strategy = SvnCheckoutStrategy.UPDATE_WITH_REVERT) {
job.scm {
svn {
location("${repository.url()}${loc}") {
credentials(repository.credential())
directory(dir)
}
checkoutStrategy(strategy)
}
}
}

svn(Repositories.FOO, '/UTIL/trunk')

svn(Repositories.FOO, '/UTIL/trunk')
Why is it configured that way?

void cron(Cron c) {
job.triggers {
cron(c.getTab())
}
}

void cron(Cron c) {
job.triggers {
cron(c.getTab())
}
}
enum Cron {
FIVE_MINUTES( 'H/5 * * * *', 'Once in five minutes'),
FIFTEEN_MINUTES( 'H/15 * * * *', 'Once in fifteen minutes'),
THIRTY_MINUTES( 'H/30 * * * *', 'Once every thirty minutes'),
HOURLY( 'H * * * *', 'Once an hour'),
WEEKDAY_TWO_HOURLY('H 8-18/2 * * 1-5', 'Once every two hours every weekday'),
MIDNIGHT( 'H H(0-2) * * *', 'Once right after midnight'),
DAILY( 'H H * * *', 'Once a day'),
WEEKLY( 'H H * * H', 'Once a week'),
MONTHLY( 'H H H * *', 'Once a month')
}

cron(Cron.FIFTEEN_MINUTES)

Easy-to-read configurations
new JobBuilder(this, 'run-script').make {
steps {
shell('''ssh $KEY_FILE $USER@FOO <<'END' java -jar cli.jar PROJECTIMPORT
-host localhost
-port 12345
-HTTP
-user "${USER}"
-password "${PASSWORD}"
-project "${PROJECT_NAME}"
-projectFile "${PROJECT_FILE}"
-dbuser '"$(grep "content.jdbc.USER" database.conf | cut -d "=" -f2-)"'
-dbpassword '"$(grep "content.jdbc.PASSWORD" database.conf | cut -d "=" -f2-)"'
-dburl '"$(grep "content.jdbc.URL" database.conf | cut -d "=" -f2-)"'
-dbschema '"$(grep "content.jdbc.SCHEMA" database.conf | cut -d "=" -f2-)"'
END''')
}
}

new JobBuilder(this, 'run-script').make {
def scriptfile = readFileFromWorkspace('src/main/resources/bin/scriptfile.sh')
steps {
shell(scriptfile)
}
}

new MavenJobBuilder(this, 'build-new-deathstar').make {
svn(Repositories.DARKEMPIRE, '/spaceships/deathstar/shapes/ball')
cronScm(Cron.EVERY_MOVIE)
sendMail('darth.vader@empire.dark.net')
sendMail('sheev.palpatine@empire.dark.net')
}

}
new MavenJobBuilder(this, 'build-super-star-destroyers').make {
svn(Repositories.DARKEMPIRE, '/spaceships/destroyers')
cronScm(Cron.WEEKLY)
expandMavenGoals('--activate-profiles super')
}

}
}
new MavenJobBuilder(this, 'build-star-destroyers').make {
cronScm(Cron.DAILY)
reportEmpireAcceptanceTests()
}

}
}
cronScm(Cron.DAILY)
}
Who is responsible for this job?

}
}
cronScm(Cron.DAILY)
}
What is this job doing?

}
}
cronScm(Cron.DAILY)
}
How to change settings for
a group of jobs?
What is this job doing?

DevEnvironment.values().each { environment ->
new DeployJobBuilder(this, 'Deploy-${environment.getEnvName()}').make {
}

job.environmentVariables {
env('ENVIRONMENT_NAME', environment.getEnvName())
env('BRANCH', environment.getBranch())
env('TEST_DATA', environment.getTestData())
if (DevEnvironment.UAT.equals(environment)) {
env('UAT', true)
}
[...]
}
}

job.environmentVariables {
env('ENVIRONMENT_NAME', environment.getEnvName())
env('BRANCH', environment.getBranch())
env('TEST_DATA', environment.getTestData())
if (DevEnvironment.UAT.equals(environment)) {
env('UAT', true)
}
[...]
}
}
enum DevEnvironment {
STAGE1('system-integration', Branch.SIT, '#customer'),
STAGE2('performance', Branch.PRF, '#customer'),
UAT( 'external-test', Branch.ET, '#customer#partner'),
PROD( 'production', Branch.MASTER, '#full')
}

Project structure
.
├── pom.xml # build file

Project structure
.
├── src
│ ├── main
│ │ ├── groovy # job builder and support classes

Project structure
.
├── src
│ ├── main
│ │ └── resources
│ │ ├── bin # common scripts (shell, batch, groovy etc.)
│ │ ├── jobs # DSL script files for jobs
│ │ └── views # DSL script files for views

Project structure
.
├── src
│ ├── main
│ └── test
│ ├── groovy # specs for the job builders and their functionality
│ └── resources # script files for tests

Project structure
.
├── src
│ ├── main
│ └── test
│ ├── groovy # specs for the job builders and their functionality
│ └── resources # script files for tests
└── target
└── debug-xml # output directory for all generated items during test
├── jobs # Useful to diff job changes
└── views

Project dependencies
<dependency>
<groupId>net.payback.ci</groupId>
<artifactId>dslbuilder</artifactId>
<version>${dslbuilder.version}</version>
</dependency>
<modules>
<module>dev-bi</module>
<module>dev-core</module>
<module>dev-performance</module>
<module>dev-portal</module>
[…]
<module>dev-vagrant</module>
</modules>

Main project with common builders, functions
and tests. Has its own release cycle.
<dependency>
</dependency>
<modules>
[…]
</modules>

Main project with common builders, functions
and tests. Has its own release cycle.
Own modules for every team, department or
the same area of responsibility
<dependency>
</dependency>
<modules>
[…]
</modules>

Resolve ALL the issues
http://knowyourmeme.com/memes/x-all-the-y

Minimum config, easy to read

Look at the commits

Look at the commits
Everyone can read it, own modules

Look at the commits

Look at the commits
Simple methods for complex stuff

Look at the commits

Look at the commits
Set basics in job builders

Look at the commits

Look at the commits
Get all the benefits from VCS

• How to change settings for
a group of jobs?
Look at the commits

a group of jobs?
Look at the commits
Refactor or search and replace

a group of jobs?
Look at the commits

a group of jobs?
Look at the commits
Simplified with custom builders

a group of jobs?
Look at the commits

a group of jobs?
Look at the commits
Outstanding API viewer ;)

How to make it foolproof
.........................….......Intro Configuration as Code Test your codeThe next level

People will do the same thing in the code, as they did before via the UI

• Copy & Paste of job builders

• Write jobs in pure dsl syntax, but forget about standards

• Use anti-patterns in code, configuration will be unreadable again

• Use anti-patterns in code, configuration will be unreadable again
WRITE TESTS

Test Example from https://github.com/sheehan/job-dsl-gradle-example
class JobScriptsSpec extends Specification {
@Unroll
void 'test script #file.name'(File file) {
given:
when:
then:
where:
[...]
}
...................................Intro Configuration as Code Test your codeThe next level

@Unroll
given:
JobManagement jm = Spy(MemoryJobManagement)
[...]
when:
then:
where:
[...]
}

@Unroll
given:
[...]
when:
DslScriptLoader(jm).runScript(file.text)
then:
where:
[...]
}

@Unroll
given:
[...]
when:
then:
noExceptionThrown()
where:
[...]
}

@Unroll
given:
[...]
when:
then:
noExceptionThrown()
where:
file << jobFiles
}
[...]
}

What can be tested?
def 'this one has a timer trigger set, but no scm trigger'() {
setup:
Job job = new JobBuilder(jobParent, 'foo').make { cron(Cron.HOURLY) }
expect:
with(job.node) {
........................…........Intro Configuration as Code Test your codeThe next level

What can be tested?
setup:
expect:
with(job.node) {
<project>
<triggers>
<hudson.triggers.TimerTrigger>
<spec>H * * * *</spec>
</hudson.triggers.TimerTrigger>
</triggers>
</project>

What can be tested?
setup:
expect:
with(job.node) {
// is any trigger set?
triggers
<project>
<triggers>
</triggers>
</project>

What can be tested?
setup:
expect:
with(job.node) {
triggers
// is trigger a timer trigger?
triggers.'hudson.triggers.TimerTrigger'
<project>
<triggers>
</triggers>
</project>

What can be tested?
setup:
expect:
with(job.node) {
triggers
// trigger is NOT an scm trigger?
!triggers.'hudson.triggers.SCMTrigger'
<project>
<triggers>
</triggers>
</project>

What can be tested?
setup:
expect:
with(job.node) {
triggers
// has trigger a specific value?
triggers.text().equals('H * * * *')
<project>
<triggers>
</triggers>
</project>

What can be tested?
setup:
expect:
with(job.node) {
triggers
// has timer trigger a specific value?
triggers.'hudson.triggers.TimerTrigger'.text().equals('H * * * *')
<project>
<triggers>
</triggers>
</project>

What can be tested?
setup:
expect:
with(job.node) {
triggers
// has timer trigger a specific value?
triggers.'hudson.triggers.TimerTrigger'.text().equals('H * * * *')
// trigger has a tag with value timer trigger
triggers.get(0).value().get(0).name().equals('hudson.triggers.TimerTrigger')
}
}
<project>
<triggers>
</triggers>
</project>

Write tests for all job configurations
def setupSpec() {
jobManagement = new MemoryJobManagement()
List<ScriptRequest> scriptRequests = []
ResourceHelper.getAllJobDslFiles().each { File script ->
URL scriptURL = new File(script.getPath()).toURI().toURL()
scriptRequests.add(new ScriptRequest(script.name, null, scriptURL))
}
new DslScriptLoader(jobManagement).runScripts(scriptRequests)
}

Write tests for all job configurations
def setupSpec() {
jobManagement = new MemoryJobManagement()
List<ScriptRequest> scriptRequests = []
ResourceHelper.getAllJobDslFiles().each { File script ->
URL scriptURL = new File(script.getPath()).toURI().toURL()
scriptRequests.add(new ScriptRequest(script.name, null, scriptURL))
}
new DslScriptLoader(jobManagement).runScripts(scriptRequests)
}
@Unroll
def 'Always activate log rotation for "#config.key"'(Map.Entry<String, String> config) {
expect:
with(new XmlParser().parse(new StringReader(config.value))) {
!logRotator.text().isEmpty()
}
where:
script << jobManagement.savedConfigs
}

Overall Testing: Enforce scripting standards
@Unroll
def 'Turn off debug output for shell steps in "#config.key"'(Map.Entry<String, String> config) {
expect:
}
where:
}

@Unroll
expect:
def scriptCommands = builders.get(0).children().findAll { buildStep ->
buildStep.name().equals('hudson.tasks.Shell')
}.collect { shellStep ->
shellStep.get('command').text()
}
}
where:
}

@Unroll
expect:
def scriptCommands = builders.get(0).children().findAll { buildStep ->
buildStep.name().equals('hudson.tasks.Shell')
}.collect { shellStep ->
shellStep.get('command').text()
}
// make sure every script has debug disabled by default
scriptCommands.findAll {
!it.startsWith('#!/bin/sh -en')
}.findAll {
!it.startsWith('@echo offn')
}.isEmpty()
}
where:
}

Functional tests
def 'test functional stuff for your job builders'() {
when:
Job job = new JobBuilder(jobParent, 'foo').make {
parameters {
StringParam('IMPORTANT_PARAMETER', 'wrong!')
}
}
then:
thrown IllegalStateException('wrong parameter used!')
}

Some examples for the impact in the actual teams

• Web development: Huge release process with a bunch of manual branches
-> Now it's a one-klick release

• Business Intelligence: Hardly any automated processes via Jenkins
-> Now they automated everything

• Core development: Huge deploy job blob that no one understood
-> One-klick deploy jobs for every environment (350+ jobs)

• Core development: Huge deploy job blob that no one understood
-> One-klick deploy jobs for every environment (350+ jobs)
Teams can work
autonomously

Benefits
• Refactor jobs without fear due to regression testing and debug output
• Reduced configuration mistakes due to functional testing
• No conflicts between distributed teams
• Easy change rollout
• Minimal configuration
• Maximal convention

THANK YOU!
PAYBACK GmbH
Christian rasp
Software Developer DevOps
Theresienhöhe 12
80339 München
Phone +49 (0) 89 997 41 – 1637
christian.rasp@PAYBACK.net
PAYBACK.net | PAYBACK.de

Rule jenkins with configuration as code

More Related Content

What's hot (18)

Viewers also liked (7)

Similar to Rule jenkins with configuration as code (19)

Recently uploaded (20)

Rule jenkins with configuration as code