JUC Europe 2015: Orchestrating Your Pipelines with Jenkins, Python and the Jenkins API
1 like491 views
The document is about a presentation on using Python with the Jenkins API to orchestrate Jenkins pipelines. It covers the motivation for the library, installation instructions, main modules, and provides code examples demonstrating how to manage jobs, builds, and artifacts within Jenkins. Additionally, it discusses built-in exceptions and opportunities for further contribution to the library's documentation.
![#jenkinsconf
Jobs
l
>>> jenkins = get_jenkins_instance()
l
>>> jenkins.keys()
l
>>> ['Beefy Job', 'Simple Job']
l
>>> jobs = jenkins.get_jobs()
l
>>> for job in jobs:
l
... print job
l
...
l
('Beefy Job', <jenkinsapi.job.Job Beefy Job>)
l
('Simple Job', <jenkinsapi.job.Job Simple Job>)](https://image.slidesharecdn.com/juc-2015-europe-orchestrating-your-bhattacharya-160623162848/85/JUC-Europe-2015-Orchestrating-Your-Pipelines-with-Jenkins-Python-and-the-Jenkins-API-11-320.jpg)
![#jenkinsconf
Jobs
l
>>> job.get_scm_type()
l
'git'
l
>>> job.get_scm_branch()
l
['*/master']
l
>>> job.get_scm_url()
l
['http://github.com/jenkinsci/mesos-plugin']](https://image.slidesharecdn.com/juc-2015-europe-orchestrating-your-bhattacharya-160623162848/85/JUC-Europe-2015-Orchestrating-Your-Pipelines-with-Jenkins-Python-and-the-Jenkins-API-14-320.jpg)
![#jenkinsconf
Build
l
>>> build.get_console()
l
'Started by user anonymousnBuilding on master in workspace /var/lib/jenkins/workspace/Beefy
Jobn > git rev-parse --is-inside-work-tree # timeout=10nFetching changes from the remote Git
repositoryn > git config remote.origin.url http://github.com/jenkinsci/mesos-plugin #
timeout=10nFetching upstream changes from http://github.com/jenkinsci/mesos-pluginn > git
--version # timeout=10n > git -c core.askpass=true fetch --tags --progress
http://github.com/jenkinsci/mesos-plugin +refs/heads/*:refs/remotes/origin/*n > git rev-parse
refs/remotes/origin/master^{commit} # timeout=10n > git rev-parse
refs/remotes/origin/origin/master^{commit} # timeout=10nChecking out Revision
b0c30d15b4c0ac55d003db7533c00e889f74891e (refs/remotes/origin/master)n > git config
core.sparsecheckout # timeout=10n > git checkout -f
b0c30d15b4c0ac55d003db7533c00e889f74891en > git rev-list
b0c30d15b4c0ac55d003db7533c00e889f74891e # timeout=10n[Beefy Job] $ /bin/sh -xe
/tmp/hudson1394675615697321597.shn+ set -xn+ echo This job takes a lot of time and
consumes a lot of resources.nThis job takes a lot of time and consumes a lot of resources.n+
which pythonn/usr/bin/pythonn+ python --versionnPython 2.7.6n+ sleep 180n+ echo Done
building beefy job.nDone building beefy job.nFinished: SUCCESSn'](https://image.slidesharecdn.com/juc-2015-europe-orchestrating-your-bhattacharya-160623162848/85/JUC-Europe-2015-Orchestrating-Your-Pipelines-with-Jenkins-Python-and-the-Jenkins-API-19-320.jpg)
![#jenkinsconf
Artifacts
l
>>> job = jenkins.get_job('Simple Job')
l
>>> build = job.get_build(33)
l
>>> build.get_artifact_dict()
l
{'artifact.txt': <jenkinsapi.artifact.Artifact http://10.10.20.100:8080/job/Simple
%20Job/33/artifact/artifact.txt>}
l
>>> build.get_artifact_dict()['artifact.txt'].get_data()
l
'This is artifactn'](https://image.slidesharecdn.com/juc-2015-europe-orchestrating-your-bhattacharya-160623162848/85/JUC-Europe-2015-Orchestrating-Your-Pipelines-with-Jenkins-Python-and-the-Jenkins-API-21-320.jpg)
![#jenkinsconf
Plugins
l
>>> jenkins.get_plugins().keys()
l
['git-client', 'matrix-auth', 'mesos', 'maven-plugin', 'javadoc', 'external-monitor-job', 'ant',
'ssh-slaves', 'pam-auth', 'windows-slaves', 'git', 'scm-api', 'subversion', 'antisamy-
markup-formatter', 'ldap', 'junit', 'mailer', 'credentials', 'translation', 'ssh-credentials',
'matrix-project', 'cvs', 'script-security']](https://image.slidesharecdn.com/juc-2015-europe-orchestrating-your-bhattacharya-160623162848/85/JUC-Europe-2015-Orchestrating-Your-Pipelines-with-Jenkins-Python-and-the-Jenkins-API-23-320.jpg)
JUC Europe 2015: Orchestrating Your Pipelines with Jenkins, Python and the Jenkins API
- 1. #jenkinsconf Orchestrating Your Jenkins Pipelines With Python and Jenkins API London, 23 & 24 June 2015
- 2. #jenkinsconf whoami Pradeepto Bhattacharya / @pradeepto
- 3. #jenkinsconf Agenda l Why do we need this library? l Installation l Library Modules l Code Examples l Documentations l Q & A
- 5. #jenkinsconf Why? l Sometimes you want more control over how you want run your jobs. l It is true that you have many plugins to so many things like - copy artifacts, run child jobs etc. No doubt about it. l But then there are times you have to develop systems or CI pipelines where you need more fine grained control. l Thanks to the developers of Jenkins, there is an excellent REST API. l Open source being open source, some good dudes have written a Python library encapsulating the REST API l This talk is about this library and what can we do with it.
- 7. #jenkinsconf Installation l pip install jenkinsapi l easy_install jenkinsapi l sudo apt-get install python-jenkinsapi l Note : There are other similar libraries. I found this one best for all my use cases. It definitely was the most exhaustive.
- 9. #jenkinsconf Main modules l Jenkins : The Jenkins instance l Job(s) : Represents Jenkins jobs. l User API : Consists of helpful high-level functions l Build : This module encapsulates the single run of a Jenkins job l Artifact : Artifacts are created by Jenkins build. This module encapsulates that. l Node : Encapsulates Jenkins Node
- 10. #jenkinsconf Connect to Jenkins l from jenkinsapi.jenkins import Jenkins l def get_jenkins_instance(): l return Jenkins('http://10.10.20.100:8080')
- 11. #jenkinsconf Jobs l >>> jenkins = get_jenkins_instance() l >>> jenkins.keys() l >>> ['Beefy Job', 'Simple Job'] l >>> jobs = jenkins.get_jobs() l >>> for job in jobs: l ... print job l ... l ('Beefy Job', <jenkinsapi.job.Job Beefy Job>) l ('Simple Job', <jenkinsapi.job.Job Simple Job>)
- 12. #jenkinsconf Jobs l >>> job = jenkins.get_job('Simple Job') l >>> job.get_description() l 'A very simple job' l >>> job.is_enabled() l True l >>> job.is_running() l False l >>> job.get_last_stable_buildnumber() l 29
- 13. #jenkinsconf Jobs l >>> job = jenkins.get_job('Beefy Job') l >>> job.get_last_failed_buildnumber() l 3 l >>> job.get_last_completed_build() l <jenkinsapi.build.Build Beefy Job #53> l >>> job.invoke() l <jenkinsapi.invocation.Invocation object at 0x7f0f75733050> l >>> job.is_running() l True
- 14. #jenkinsconf Jobs l >>> job.get_scm_type() l 'git' l >>> job.get_scm_branch() l ['*/master'] l >>> job.get_scm_url() l ['http://github.com/jenkinsci/mesos-plugin']
- 15. #jenkinsconf Jobs l >>> job.disable() l >>> job.is_enabled() l False l >>> job.enable() l >>> job.is_enabled() l True
- 16. #jenkinsconf Jobs l 'get_first_build', 'get_first_buildnumber', 'get_jenkins_obj', 'get_last_build', 'get_last_buildnumber', 'get_last_completed_build', 'get_last_completed_buildnumber', 'get_last_failed_buildnumber', 'get_last_good_build', 'get_last_good_buildnumber', 'get_last_stable_build', 'get_last_stable_buildnumber', 'get_next_build_number', 'get_params', 'get_params_list', 'get_revision_dict', 'get_scm_branch', 'get_scm_type', 'get_scm_url', 'get_upstream_job_names', 'get_upstream_jobs', 'has_queued_build', 'invoke', 'is_enabled', 'is_queued', 'is_queued_or_running', 'is_running'
- 17. #jenkinsconf User API l from jenkinsapi.api import * l >>> get_latest_build('http://10.10.20.100:8080','Simple Job') l <jenkinsapi.build.Build Simple Job #29> l >>> get_latest_complete_build('http://10.10.20.100:8080','Beefy Job') l <jenkinsapi.build.Build Beefy Job #53>
- 18. #jenkinsconf Build l >>> build = job.get_build(54) l >>> build.is_running() l True l >>> build.get_revision() l 'b0c30d15b4c0ac55d003db7533c00e889f74891e' l >>> build.name l 'Beefy Job #57' l >>> build.get_status() l 'SUCCESS'
- 19. #jenkinsconf Build l >>> build.get_console() l 'Started by user anonymousnBuilding on master in workspace /var/lib/jenkins/workspace/Beefy Jobn > git rev-parse --is-inside-work-tree # timeout=10nFetching changes from the remote Git repositoryn > git config remote.origin.url http://github.com/jenkinsci/mesos-plugin # timeout=10nFetching upstream changes from http://github.com/jenkinsci/mesos-pluginn > git --version # timeout=10n > git -c core.askpass=true fetch --tags --progress http://github.com/jenkinsci/mesos-plugin +refs/heads/*:refs/remotes/origin/*n > git rev-parse refs/remotes/origin/master^{commit} # timeout=10n > git rev-parse refs/remotes/origin/origin/master^{commit} # timeout=10nChecking out Revision b0c30d15b4c0ac55d003db7533c00e889f74891e (refs/remotes/origin/master)n > git config core.sparsecheckout # timeout=10n > git checkout -f b0c30d15b4c0ac55d003db7533c00e889f74891en > git rev-list b0c30d15b4c0ac55d003db7533c00e889f74891e # timeout=10n[Beefy Job] $ /bin/sh -xe /tmp/hudson1394675615697321597.shn+ set -xn+ echo This job takes a lot of time and consumes a lot of resources.nThis job takes a lot of time and consumes a lot of resources.n+ which pythonn/usr/bin/pythonn+ python --versionnPython 2.7.6n+ sleep 180n+ echo Done building beefy job.nDone building beefy job.nFinished: SUCCESSn'
- 20. #jenkinsconf Build l 'block', 'block_until_complete', 'buildno', 'get_actions', 'get_artifact_dict', 'get_artifacts', 'get_console', 'get_data', 'get_downstream_builds', 'get_downstream_job_names', 'get_downstream_jobs', 'get_duration', 'get_jenkins_obj', 'get_master_build', 'get_master_build_number', 'get_master_job', 'get_master_job_name', 'get_matrix_runs', 'get_number', 'get_result_url', 'get_resultset', 'get_revision', 'get_revision_branch', 'get_status', 'get_timestamp', 'get_upstream_build', 'get_upstream_build_number', 'get_upstream_job', 'get_upstream_job_name', 'has_resultset', 'is_good', 'is_running'
- 21. #jenkinsconf Artifacts l >>> job = jenkins.get_job('Simple Job') l >>> build = job.get_build(33) l >>> build.get_artifact_dict() l {'artifact.txt': <jenkinsapi.artifact.Artifact http://10.10.20.100:8080/job/Simple %20Job/33/artifact/artifact.txt>} l >>> build.get_artifact_dict()['artifact.txt'].get_data() l 'This is artifactn'
- 22. #jenkinsconf Nodes l >>> for node in jenkins.get_nodes().keys(): l ... print node l ... l master l Big l Small l >>> small.is_online() l True
- 23. #jenkinsconf Plugins l >>> jenkins.get_plugins().keys() l ['git-client', 'matrix-auth', 'mesos', 'maven-plugin', 'javadoc', 'external-monitor-job', 'ant', 'ssh-slaves', 'pam-auth', 'windows-slaves', 'git', 'scm-api', 'subversion', 'antisamy- markup-formatter', 'ldap', 'junit', 'mailer', 'credentials', 'translation', 'ssh-credentials', 'matrix-project', 'cvs', 'script-security']
- 24. #jenkinsconf Exceptions l This library comes with bunch of built-in exceptions. I highly recommend you use them in your code and do stuff as you want in case you catch those whilst running your CI pipelines. l JenkinsAPIException, UnknownJob, UnknownView, UnknownNode, UnknownPlugin, NoBuildData, NoResults ...
- 25. #jenkinsconf Other things you could ... l Create Jobs l Create Views l Monitor and query builds/jobs/nodes l Search and manipulate artifacts
- 26. #jenkinsconf Documentation l Some (very few) of the modules are documented. l There are a few examples in documentation. l Great opportunity to contribute to this wonderful library. l https://github.com/salimfadhley/jenkinsapi
- 28. #jenkinsconf Agenda CloudBees and organisers of JUC Europe 2015 Contributors of Jenkins, Mesos, Mesos plugin My Family Kalpak, Shikha, Lenin Girija and Rupali (my partners in CI related crimes)
- 29. #jenkinsconf Please Share Your Feedback • Did you find this session valuable? • Please share your thoughts in the • Jenkins User Conference Mobile App. • Find the session in the app and click • on the feedback area.