Continuous Integration with Robot Sweatshop
1 like572 views
The document outlines a Jenkins configuration for a continuous integration system focusing on automation with Buildbot, including settings for running Cucumber tests and managing build results. It features commands for creating directories, running tests, and sending notification emails upon build outcomes. The text also discusses challenges in maintaining tools and the architecture supporting the distribution of workloads in a CI/CD environment.
![BUILDBOT
from buildbot.process.factory import BuildFactory
from buildbot.steps.source import SVN
from buildbot.steps.shell import ShellCommand
checkout = SVN(baseURL = 'git@myrepos.com:portal-project.git',
mode = "update",
username = "foo",
password = "bar",
haltOnFailure = True )
mkdir = ShellCommand(name = "mkdir -p telus/target",
command = ["mkdir", "-p", "telus/target"],
description = "make the results path")
cucumber = ShellCommand(name = "cucumber -p json_report",
command = ["cucumber", "-p", "json_report"],
description = "run Cucumber tests")
logresults = ShellCommand(name = "log-cucumber telus/target",
command = ["log-cucumber", "telus/target"],
description = "upload results")
factory = BuildFactory()
factory.addStep(checkout)
factory.addStep(mkdir)
factory.addStep(cucumber)
factory.addStep(logresults)
c['builders'] = [
BuilderConfig(name = "portal-project",
slavenames = ['slave1', 'slave2', 'slave3'],
factory = factory)
]](https://image.slidesharecdn.com/robotsweatshop-150709184349-lva1-app6891/85/Continuous-Integration-with-Robot-Sweatshop-8-320.jpg)
![SIDE-BY-SIDE CONFIGS
<?xml version='1.0' encoding='UTF-8'?>
<project>
<actions/>
<description>requires these gems:
statsd-ruby
selenium-webdriver
cucumber
json
RMagick</description>
<logRotator class="hudson.tasks.LogRotator">
<daysToKeep>10</daysToKeep>
<numToKeep>-1</numToKeep>
<artifactDaysToKeep>-1</artifactDaysToKeep>
<artifactNumToKeep>-1</artifactNumToKeep>
</logRotator>
<keepDependencies>false</keepDependencies>
<properties/>
<scm class="hudson.plugins.git.GitSCM" plugin="git@2.2.1">
<configVersion>2</configVersion>
<userRemoteConfigs>
<hudson.plugins.git.UserRemoteConfig>
<url>git@bitbucket.org:webchannel/uss-cucumber.git</url>
<credentialsId>ad5d9dcc-cb2f-42a4-8e9a-f47b3208ce0c</credentialsId>
</hudson.plugins.git.UserRemoteConfig>
</userRemoteConfigs>
<branches>
<hudson.plugins.git.BranchSpec>
<name>*/develop</name>
</hudson.plugins.git.BranchSpec>
</branches>
<doGenerateSubmoduleConfigurations>false</doGenerateSubmoduleConfigurations>
<submoduleCfg class="list"/>
<extensions>
<hudson.plugins.git.extensions.impl.CloneOption>
<shallow>true</shallow>
<reference></reference>
</hudson.plugins.git.extensions.impl.CloneOption>
</extensions>
</scm>
<canRoam>true</canRoam>
<disabled>false</disabled>
<blockBuildWhenDownstreamBuilding>false</blockBuildWhenDownstreamBuilding>
<blockBuildWhenUpstreamBuilding>false</blockBuildWhenUpstreamBuilding>
<triggers>
<hudson.triggers.TimerTrigger>
<spec>H 11 * * *</spec>
</hudson.triggers.TimerTrigger>
</triggers>
<concurrentBuild>false</concurrentBuild>
<builders>
<hudson.tasks.Shell>
<command>cd telus
mkdir -p target
cucumber -p json_report -p remote_ff</command>
</hudson.tasks.Shell>
</builders>
<publishers>
<org.jenkinsci.plugins.cucumber.jsontestsupport.CucumberTestResultArchiver
plugin="cucumber-testresult-plugin@0.4">
<testResults>telus/target/features_report.json</testResults>
</org.jenkinsci.plugins.cucumber.jsontestsupport.CucumberTestResultArchiver>
<hudson.tasks.Mailer plugin="mailer@1.8">
<recipients>krystal.jackson2@telus.com leon.yu@telus.com
jonathan.shin@telus.com Brendan.Brow@telus.com</recipients>
<dontNotifyEveryUnstableBuild>false</dontNotifyEveryUnstableBuild>
<sendToIndividuals>false</sendToIndividuals>
</hudson.tasks.Mailer>
</publishers>
<buildWrapp
from buildbot.process.factory import BuildFactory
from buildbot.steps.source import SVN
from buildbot.steps.shell import ShellCommand
checkout = SVN(baseURL = 'git@myrepos.com:portal-project.git',
mode = "update",
username = "foo",
password = "bar",
haltOnFailure = True )
mkdir = ShellCommand(name = "mkdir -p telus/target",
command = ["mkdir", "-p", "telus/target"],
description = "make the results path")
cucumber = ShellCommand(name = "cucumber -p json_report",
command = ["cucumber", "-p", "json_report"],
description = "run Cucumber tests")
logresults = ShellCommand(name = "log-cucumber telus/target",
command = ["log-cucumber", "telus/target"],
description = "upload results")
factory = BuildFactory()
factory.addStep(checkout)
factory.addStep(mkdir)
factory.addStep(cucumber)
factory.addStep(logresults)
c['builders'] = [
BuilderConfig(name = "portal-project",
slavenames = ['slave1', 'slave2', 'slave3'],
factory = factory)
]
---
branch_whitelist:
- develop
commands:
- mkdir -p telus/target
- cucumber -p json_report
- log-cucumber telus/target
67
33+
8](https://image.slidesharecdn.com/robotsweatshop-150709184349-lva1-app6891/85/Continuous-Integration-with-Robot-Sweatshop-12-320.jpg)
Continuous Integration with Robot Sweatshop
- 1. ROBOT SWEATSHOP Lightweight, non-opinionated Continuous Integration
- 6. JENKINS … <builders> <hudson.tasks.Shell> <command>cd telus mkdir -p target cucumber -p json_report</command> </hudson.tasks.Shell> </builders> …
- 7. – a Jenkins user “I’d rather work around a bad tool than maintain something new.”
- 8. BUILDBOT from buildbot.process.factory import BuildFactory from buildbot.steps.source import SVN from buildbot.steps.shell import ShellCommand checkout = SVN(baseURL = 'git@myrepos.com:portal-project.git', mode = "update", username = "foo", password = "bar", haltOnFailure = True ) mkdir = ShellCommand(name = "mkdir -p telus/target", command = ["mkdir", "-p", "telus/target"], description = "make the results path") cucumber = ShellCommand(name = "cucumber -p json_report", command = ["cucumber", "-p", "json_report"], description = "run Cucumber tests") logresults = ShellCommand(name = "log-cucumber telus/target", command = ["log-cucumber", "telus/target"], description = "upload results") factory = BuildFactory() factory.addStep(checkout) factory.addStep(mkdir) factory.addStep(cucumber) factory.addStep(logresults) c['builders'] = [ BuilderConfig(name = "portal-project", slavenames = ['slave1', 'slave2', 'slave3'], factory = factory) ]
- 9. http://docs.buildbot.net/current/tutorial/fiveminutes.html “Typically, at first sight it looks like a bunch of complicated concepts that make no sense and whose relationships with each other are unclear.”
- 10. STOP SUFFERING BADTOOLS We can do better
- 11. ROBOT SWEATSHOP jobs/portal-project.yaml --- branch_whitelist: - develop commands: - mkdir -p telus/target - cucumber -p json_report - log-cucumber telus/target
- 12. SIDE-BY-SIDE CONFIGS <?xml version='1.0' encoding='UTF-8'?> <project> <actions/> <description>requires these gems:
 statsd-ruby
 selenium-webdriver
 cucumber
 json
 RMagick</description> <logRotator class="hudson.tasks.LogRotator"> <daysToKeep>10</daysToKeep> <numToKeep>-1</numToKeep> <artifactDaysToKeep>-1</artifactDaysToKeep> <artifactNumToKeep>-1</artifactNumToKeep> </logRotator> <keepDependencies>false</keepDependencies> <properties/> <scm class="hudson.plugins.git.GitSCM" plugin="git@2.2.1"> <configVersion>2</configVersion> <userRemoteConfigs> <hudson.plugins.git.UserRemoteConfig> <url>git@bitbucket.org:webchannel/uss-cucumber.git</url> <credentialsId>ad5d9dcc-cb2f-42a4-8e9a-f47b3208ce0c</credentialsId> </hudson.plugins.git.UserRemoteConfig> </userRemoteConfigs> <branches> <hudson.plugins.git.BranchSpec> <name>*/develop</name> </hudson.plugins.git.BranchSpec> </branches> <doGenerateSubmoduleConfigurations>false</doGenerateSubmoduleConfigurations> <submoduleCfg class="list"/> <extensions> <hudson.plugins.git.extensions.impl.CloneOption> <shallow>true</shallow> <reference></reference> </hudson.plugins.git.extensions.impl.CloneOption> </extensions> </scm> <canRoam>true</canRoam> <disabled>false</disabled> <blockBuildWhenDownstreamBuilding>false</blockBuildWhenDownstreamBuilding> <blockBuildWhenUpstreamBuilding>false</blockBuildWhenUpstreamBuilding> <triggers> <hudson.triggers.TimerTrigger> <spec>H 11 * * *</spec> </hudson.triggers.TimerTrigger> </triggers> <concurrentBuild>false</concurrentBuild> <builders> <hudson.tasks.Shell> <command>cd telus mkdir -p target cucumber -p json_report -p remote_ff</command> </hudson.tasks.Shell> </builders> <publishers> <org.jenkinsci.plugins.cucumber.jsontestsupport.CucumberTestResultArchiver plugin="cucumber-testresult-plugin@0.4"> <testResults>telus/target/features_report.json</testResults> </org.jenkinsci.plugins.cucumber.jsontestsupport.CucumberTestResultArchiver> <hudson.tasks.Mailer plugin="mailer@1.8"> <recipients>krystal.jackson2@telus.com leon.yu@telus.com jonathan.shin@telus.com Brendan.Brow@telus.com</recipients> <dontNotifyEveryUnstableBuild>false</dontNotifyEveryUnstableBuild> <sendToIndividuals>false</sendToIndividuals> </hudson.tasks.Mailer> </publishers> <buildWrapp from buildbot.process.factory import BuildFactory from buildbot.steps.source import SVN from buildbot.steps.shell import ShellCommand checkout = SVN(baseURL = 'git@myrepos.com:portal-project.git', mode = "update", username = "foo", password = "bar", haltOnFailure = True ) mkdir = ShellCommand(name = "mkdir -p telus/target", command = ["mkdir", "-p", "telus/target"], description = "make the results path") cucumber = ShellCommand(name = "cucumber -p json_report", command = ["cucumber", "-p", "json_report"], description = "run Cucumber tests") logresults = ShellCommand(name = "log-cucumber telus/target", command = ["log-cucumber", "telus/target"], description = "upload results") factory = BuildFactory() factory.addStep(checkout) factory.addStep(mkdir) factory.addStep(cucumber) factory.addStep(logresults) c['builders'] = [ BuilderConfig(name = "portal-project", slavenames = ['slave1', 'slave2', 'slave3'], factory = factory) ] --- branch_whitelist: - develop commands: - mkdir -p telus/target - cucumber -p json_report - log-cucumber telus/target 67 33+ 8
- 14. THE FUTURE Robot Sweatshop • Output node • Multiple workers easily • Implicit job (done!) • Non-TCP connections • Worker Docker support Sweatshop Gears • Scaffolding scripts • Custom repos • Service hub • Update/uninstall • List/show • Docker helper scripts
- 15. QUESTIONS? robotsweat.com But what about all the crunchy parts…?
- 16. UNDERTHE HOOD overcoming some interesting challenges
- 19. GEARS DEMO package management tool for non-core bits https://asciinema.org/a/22878