Jenkins Pipeline @ Scale. Building Automation Frameworks for Systems Integration

Jenkins Pipeline @ Scale
Building Automation Frameworks for
Systems Integration
CloudBees | Jenkins Automotive
and Embedded Day
Böblingen, April 05, 2017
Oleg Nenashev, CloudBees

© 2017 CloudBees, Inc. All Rights Reserved. 2
About me. Hardware/Embedded
@oleg_nenashev
oleg-nenashev LibreCoresSt. Petersburg
Polytechnic University

About me. Jenkins project
@oleg_nenashev
oleg-nenashev

Oleg’s
“Hall of
Shame”(c)
• Jenkins Core
• Remoting
• Plugins

2016
http://bit.ly/nenashev-automotive-day-2016

ØHow/why do we build frameworks for Embedded?
ØWhat has changed in Pipeline since 2016?
ØHow to effectively use and develop Pipeline?
ØLessons learnt
Agenda

• Pipeline 101
• Pipeline Generic Features
• Blue Ocean
• Declarative Pipeline
• Automation Infrastructure
• Agents, Tools, Hardware Peripherals
• Added-value Pipeline features in CloudBees Jenkins Enterprise
What is NOT in the talk?
Ask the Experts,
Jenkins Online Meetup
Birds of Feather,
Ask the Experts

What is a typical Embedded Product?
`
Hardware
External IPFirmware /
Embedded
SW
External
SW
Development
tools / EDAs
Demo software Demo hardware
Developer kits
“Main product”
Documentation

Embedded area.
Welcome to the automation hell!
• Almost everything needs to be automated
• Multiple technologies and tools
• Integration testing is important
• Unstable hardware & networks
• Legacy stuff is everywhere
• …

OK, Automotive…
Source: http://www.popsautoelectric.com/auto-electric/
• System of systems
• High level of integration
• Components by 3rd
parties
• Many standards
• High quality requirements

1. The most popular CI automation
server in the world
2. It’s a framework => flexible
• Swiss Army Knife of Automation
3. It’s open source => customizable
4. Commercial support is available
5. Useful plugins
Why Jenkins?

Система
Docs
Plugins Configs
Infra-
structure
Samples
Demos
How do we build
CI Systems?
Infra-
structure
Infra-
structure
• Agents (FKA ”slaves”)
• Tools
• Hardware peripherals
• Computing grids

Approach: Using “classic” Job types:
• Freestyle, Matrix, JobDSL, …
Classic Approaches
Keywords:
• Script steps to run the logic
• Custom Tools Plugin to manage tools
• Throttle Concurrent Builds and Lockable
Resources to dispatch access to hardware
• Naginator Plugin to restart builds on-demand
• Parameterized Trigger / Copy artifacts to
parallelize tests and to collect results
JUC2015/London
https://jenkins.io/solutions/embedded/

Classic Approaches
Spaghetti Automation
• Barely traceable
dependencies
• Complex build flow
• Complex
configurations
• Duplication
• Expensive
maintenance

ü Store build definitions with the project
ü Track changes together with the project
ü Tests together with the project
Configuration as Code

Configuration as Code in Jenkins
Job ConfigurationJenkins system
configuration

Jenkins System Configuration… as Code
External Tools
Jenkins CLI
and REST
API
python-
jenkins
jenkins-client
(java)
Configuration
Management
systems
Ansible, Chef
Docker
Solutions
within Jenkins
project
Groovy Boot
Hooks
Scriptler
Plugin
SCM Sync
Configuration
Unstable
Just examples…

Configuration as Code for jobs
Job DSL
Job Builder Plugin
Jenkins Pipeline
Pipeline is offered by default in Jenkins 2.0
…
Groovy
YAML
Groovy

Pipeline is a…
Ø Groovy DSL for automatic flow description *
• Alternatives – Job DSL, Jenkins Job Builder
Ø Job type in Jenkins (Pipeline Job)
• Job definitions can be stored in SCM
• Automatic job creation from repositories and organizations
Ø Ecosystem for automation as code
• ~20 dedicated plugins
• Integration in many other plugins
• Suggested by default in Jenkins 2
// Run on a node having the “my_fpga” label
node("linux && ml509") {
git url:"http://github.com/myorg/myproject.git"
sh "make all"
} http://bit.ly/pipeline-tutorial
19

Why Jenkins Pipeline?

Why Jenkins Pipeline?
• Robustness - master restarts & network
• Parallelization across multiple nodes:
• parallel()
• node()
• Ability to store the shared logic in
standalone Pipeline libraries
• Security
• Two modes: Scripted and Declarative
• Visualization: Stage View, Blue Ocean

Система
Docs
Plugins Configs
Infra-
structure
Samples
Demos
Automation
Framework
Vision
Infra-
structure
Infra-
structure
SCM
Infrastructure as Code
Configuration as Code
Documentation as Code
Pipeline Libraries
Pipelines

Developer Experience
• Encapsulation
• Simple UI
• Most of the code is in
GitHub
• No need in classic UI
for 90% of users
• Notifications to
email/GitHub

Jenkins Pipeline in Embedded
2016

2016
http://bit.ly/nenashev-automotive-day-2016

2016 – Summary
• Jenkins can be used to run tests on hardware in
embedded and automotive projects
• Jenkins Pipeline is helpful for test runs
• Pipeline lacks some critical features

• Management of resources.
• Integration with Throttle Concurrent Builds plugin
• Use-case: limiting the license utilization in automation
infrastructures (JENKINS-31801)
• Better support of tools.
• Common build flow – dozens of tools
• Integration with Custom Tools Plugin seems to be the best option,
especially in the case of multiple tool versions (JENKINS-30680)
• Pipeline package manager (JENKINS-34186)
• Lack of plugins for EDA tools in Jenkins, code duplications
• A common approach is to keep the shared "functions" in libraries
• Pipeline Global Library and Pipeline Remote Loader can be used
2016 – Pipeline improvement areas
From my talk at
CloudBees |
Jenkins Automotive
and Embedded Day
2016

Management of resources
• Lockable Resources Plugin – since 2016
• Throttle Concurrent Builds – coming soon
• Pipeline Milestones

PARTIAL - Better support of tools
• withEnv() for environment variables
• tool() step in Pipeline was improved
• Tool reference in the Job
• Section in Declarative Pipeline
• Custom tools is not fully integrated
• Follow JENKINS-30680
• ETA – Q2, 2017

DONE – Pipeline Libs (Pipeline Package Manager)
• Early 2016
• Global Pipeline Library
• Pipeline Remote Loader Plugin
• Late 2016
• Pipeline Shared Libraries
• Support of @Grab – Pipeline libs
can pull common Groovy libs
Library Structure:

• Pipeline Shared Libraries
• Libs are located in SCM
• Versioning
• Definition for scopes:
▸ Jenkins system-wide
▸ Folder
▸ Job

Pipeline Library Definition [1/2]

Pipeline Library Definition [2/2]

Using Pipeline Libraries
• Implicit Loading
• Via Jenkins Global or Folder configuration
• Explicit Loading
• @Library

Using Pipeline Libraries
com.mycorp.libs.make
Jenkinsfile
makeProject()
makeProject() implementation
- global variable
com.mycorp.libs.coverity
com.mycorp.libs.cppunit
Globally-defined
library, automatic
update
@Library() definition

Early adoption of
any framework
http://devopsreactions.tumblr.c
om/post/150674592321/pettin
g-your-incomprehensible-
piece-of-code
• It is working!
• In several pet projects…

Lessons Learnt.
What does NOT work well?

Lessons Learnt.
What does NOT work well?
• Development Tools
• Non-Success Path handling
• Declarative Pipeline
• Manual release management

Pipeline Developer Experience
• No Debugger
• No IDE
• No built-in test framework
• Limited static analysis features
•Pipeline DSL != Groovy

How do I work?
Intellij IDEA
Filesystem
SCM Plugin
(with patch)
Disclaimer: There
may be better ways
• Syntax file
• Built-in documentation
Jenkins Test
Instance
Source Code
(local .git repos)
• Production
System Snapshot
• Custom Pipeline
Library Settings
• Repositories for libraries
• Jenkinsfile repos

Getting the Syntax File

Filesystem SCM Plugin
• Pending pull request:
• https://github.com/jenkinsci/filesystem_scm-plugin/pull/2
Library definition in
the test instance
!!!

Pipeline Testing
• No standard test framework offered by the Jenkins project
• External project: https://github.com/lesfurets/JenkinsPipelineUnit
“mvn verify” has
been invoked…
And the execution
passed…
For the source
Jenkinsfile…

Error Handling
• IMHO: Pipeline is designed for Success Path scenarios
When it
“Just works”
When it does not

Error Handling. Top-level
Scripted Pipeline Declarative Pipeline
• Global try/catch()
When it does not…

Error Handling
When it
“Just works”
When it does not
The interesting part…
Failover, repeat

Error Handling. Hardware failover
• Pipeline allows re-running unreliable
stages on other nodes.
System failure
(script returns 253)
Node #1 Node #2. . .
. . .
for (def board : boards) {
echo "trying board " + board;
try {
node(board) {
checkout scm
sh ‘./bin/run.sh’
// Call passed => DONE
break;
}
} catch (Exception ex) {
if (ex.message.contains
("exit code 253")) {
// Fatal error
fail("Test run failed")
}
}
}
Try all possible boards:

Error handling.
Hardware failover
for (def board : boards) {
echo "trying board " + board;
try {
node(board) {
checkout scm
sh ‘./bin/run.sh’
// Call passed => DONE
break;
}
} catch (Exception ex) {
if (ex.message.contains
("exit code 255")) {
// Fatal error
fail("Test run failed")
} } }
???
• Hard to extract error cause
• Special handling is required for
every case
• Cannot just use existing parsers
• Build Failure Analyzer
• Log Parser
• Warnings
• Try-catch blocks pollute the code
• Hard to test

Context retention
• No way to propagate any data
• Solution - @NonCPS steps within the library steps

Error handling.
Hardware failover
Declarative
Pipeline
Encapsulation via
Libraries
Simple use-case
Other
• Exception try/catch
• timeout() wrapper
• Proper failover
• Logic in @NonCPS

Declarative Pipeline
• It is great for simple use-cases
• Publishers & Notifiers
• We do not need a framework for simple use-cases
• Lack of features
• “How do I pass option for the Docker Agent”
• “How do I declare a resource requirement for Throttling?”
• …
• Scripted Pipeline is a recommended fallback, but
there is no automatic migration

Declarative Pipeline. Recommendations
• Use Declarative Pipeline with caution
• Complex agent definitions – use Scripted Pipeline
• Complex throttling – use Scripted Pipeline
• Post-build steps – work well
• Tool definitions – work well
• …

Pipeline vs. Application Lifecycle

Pipeline vs. Application Lifecycle
BUT:
• Non-CD process with manual promotion
• Dependent products
• Snowflake development stages

Solutions
• Input() step + milestone() for synchronization
• Multiple Pipelines
• Release() step is coming soon
• Release Plugin
Pipeline 1 Pipeline 2
Run Selector
Staging
Artifact
Storage
External ALM tool
CD for the Component 1
CI/CD for the Component 2

Lessons learnt
• No area-specific issues so far
• High barrier to entry
• It worth investment

What I need? 2017
• Development Tools for Pipeline
• IDE, Debugger, Test Tools
• Community Pipeline Libraries
• Common Embedded cases without plugins
• Promotion engine for Pipeline
• Ideally: Merge of standard Promoted
Builds and Release plugins
• Plugins for area-specific tools

Takeaways
• Jenkins Pipeline is a powerful engine for
highly-customized Jenkins instances
• Jenkins Pipeline can be effectively used
in Embedded/Automotive use-cases
• Barrier to entry is still high

More info
• Embedded solutions page:
• https://jenkins.io/solutions/embedded/
• Links to the articles and videos
• Some categorization
Embedded Solutions
Page
…

• Jenkins project:
• Website: https://jenkins.io
• Embedded Solutions page: https://jenkins.io/solutions/embedded/
• Pipeline:
• Tutorial: https://github.com/jenkinsci/workflow-plugin/blob/master/TUTORIAL.md
• Compatibility: https://github.com/jenkinsci/pipeline-plugin/blob/master/COMPATIBILITY.md
• Examples: https://github.com/jenkinsci/pipeline-examples
• CloudBees website:
• Website: https://www.cloudbees.com
• CloudBees Jenkins Platform: https://www.cloudbees.com/products/cloudbees-jenkins-platform
Useful Links

Thank you!
Contacts:
E-mail: onenashev@cloudbees.com
GitHub: oleg-nenashev
Twitter: @oleg_nenashev

Software at the speed of ideas
THANK YOU!
www.cloudbees.com

Jenkins Pipeline @ Scale. Building Automation Frameworks for Systems Integration

More Related Content

What's hot (20)

Similar to Jenkins Pipeline @ Scale. Building Automation Frameworks for Systems Integration (20)

Recently uploaded (20)

Jenkins Pipeline @ Scale. Building Automation Frameworks for Systems Integration