SlideShare a Scribd company logo
Doing More WithYour HudsonKohsuke Kawaguchi        kohsuke@infradna.comInfraDNA, Inc.
Hudson ProjectOpen-source CI serverWritten in Java but used for all kinds of languagesEmphasis on ease of installation and use“java -jar hudson.war” executionOr your OS-specific packageConfigure everything from browsers2https://hudson-labs.org/
EcosystemThe real reason why Hudson thrivesThe same reason why Eclipse thrivesExtensibility330+ community-developed public pluginsUp from 230+ last year in JavaOne3
Growing Hudson AdoptionEclipse Community Survey9.1% in 2009 -> 21.8%  in 2010Polling by John Ferguson4
Install Base is Growing
Growing Hudson Adoption6Source: indeed.com
InfraDNAhttp://infradna.com/I left Oracle this April and started a company around HudsonSupport SubscriptionsOn open-source Hudson + pluginsOr on ICHCI: a commercial value-added distributionConsultingTroubleshooting, custom plugins, etc.Visit us at partner kiosk outside exhibit floor7
First steps to CIHudson notices that there is a changeCheck out the source codeDo some beefy computationBuildsRun testsRecord and publish resultsReport to peopleOnly when necessary8
But You Can Do So Much More!Once you automate, you can do a lotYet average Hudson users only use very small part of HudsonHence this talk9
Interconnecting Jobs               aka Workflows10
Inter-Project InteractionsYou can host many projects on 1 HudsonLots of ways to have them interact with each otherIncreased parallelism is good11
Building Block #1: triggeringWhen a build is done, kick off something else“upstream”: the build that’s kicking off“downstream”: the build being kicked offTriggering is asynchronousA large graph forms a cascading behaviourUpstream doesn’t wait for downstreamDownstream can tell who is the upstream12Foo  #10Bar #153
Test/Build SeparationDecompose test run into suite runsTrigger them all from buildtimeBuild #10Build #11Build #12…Test X Test YTest  Z13
Test/Build Separation TechniqueDo not recompile the bits during testsInstead, copy the pre-built bitsUse  the copy artifact pluginOr wget/curl  via permalink14
Build/Deploy Separation1 job to build software, another to deploy itDeployment can be rerun without rebuildFailed build, new server, etc.Helps promote separate responsibilities15
Building Block #2: join pluginFan out, and then join back16timeBuild #10Build #11Build #12…Test X Test YReport Generation
Join Configuration17
Building Block #3: FingerprintsMD5 checksums recorded during buildsx.jar in product Z #112 had a bug. Where did it come from? Did it pass all tests?18
FingerprintsThe same mechanism works for …Tracking 3rd party dependenciesTracking the version of the build tools usedFingerprinting is critical when you have inter-related jobsUsed to show relationship in UIUsed to aggregate test reports19
Build PromotionSometimes plain-vanilla triggering is not desirableExample:Build XRun smoke test suites X1 and X2If they both pass, deploy X to QA siteRun QA testIf it pass, deploy X to staging site20
Build PromotionHudson’s solution: build promotionModelDefine a promotion level“Cleared Smoke tests”Define a promotion criteria“Successful completion of X1test and X2test”Define what happens after promotion“Tag the source tree and trigger Xqa”21
More Gimmicks in Build PromotionDefine multiple promotionsE.g., “cleared smoke test” & “cleared QA test”Convenient for exposing builds to progressively more expensive workPromotion criteria can be “manual approval”We often can’t automate away everything!Last promoted build gets a permalink22
Demo
Getting More From Your Maven Builds24Photo: http://www.flickr.com/photos/seeminglee/4556156477/
Maven Integration HighlightsPOM comprehensionSimplified configurationMore gets inferred automaticallySnapshot dependencies among builds25
Dealing With Large Maven BuildParallel module buildMost projects have short but wide dependency chainLet Hudson build on module-by-module basisWherever possible, build in parallel26
Dealing With Large Maven BuildIncremental buildsBuild changed modules and their descendantsSCM determines what has changed27
Private Maven RepositoryCause~/.m2/repository is not concurrent safeMultiple builds of the same projectDifferent branches, test environments, JDK, etc.ProblemArtifacts overwrite each otherArtifacts change under the carpetSolutionAllocate ~/.m2/repository per jobConsider a repository manager28
After-the-fact DeploymentDefault “mvn deploy” behavior harmfulThink about a failure in the middleBetter to let the build complete, then deploy29
Really After-the-fact DeploymentDeploy much after the build has finished30
Matrix Project31Photo: http://www.flickr.com/photos/amarilloposters/4380080793/
MotivationYou often do the same thing with slight variationsCompile C++ code for different platformsTest with different browsersOr more generally, think of it as32for (x in [a,b,c]) {  for (y in [d,e,f]) {    …doSomethingWith(x,y,z,..);    …  }}
ModelDefine axesOne axis ≈ one for loopChoose from pre-defined types of AxisSlave axis: pick slaves by their names or their labelse.g., linux, solaris, and windowsGeneric axis: arbitrary values exposed as environment variables33
Matrix Project GimmicksFilteringOtherwise combinations increase exponentiallyNot all combinations make senseUse boolean expression to trim down the sizeOr tell Hudson to cut the workload to N%Hudson will thin out the combinations by itself34(label==“windows”).implies(browser==“iexplore”) &&(label==“mac”).implies(browser==“safari”)
Demo
36Doing Selenium Tests on Hudson
Selenium GridAutomatically installs Selenium grid binariesOn all Hudson slavesAutomatically start Selenium RCs and connect themStarts Hub on Hudson master port 4444Hudson cluster now accessible as Selenium RC farm from anywhere37
Selenium Grid38Use Hudson slaves asSelenium RC nodesHudson slavesHudson master(selenium hub)
Choosing EnvironmentUse Hudson’s labels to choose the slave that runs the browser*firefoxRun Firefox somewhere (but no guarantee that the chosen slave has it)windows:*firefoxPick a slave that has the “windows” label, then run Firefox windows&sfbay:*firefoxPick a slave that has both “windows” and “sfbay” label, then run Firefox39
Testing with multiple browsersUse “matrix project” typeRun the same tests with different browsersAggregate the results40Firefox testsIE testsSafari tests
Selenium Grid w/Hudson BenefitsNo sysadmin overhead for setting up Selenium RCsEspecially nice on heterogeneous networkReuse your existing investment in Hudson cluster41
Launching browsers inside HudsonHudson typically runs without GUIaka “Headless”Browsers need GUISo you need to make GUI environment accessible to Hudson and its slaves42
On UnixUse Hudson xvncpluginImplements X protocol w/o a real GUINeeds xvnc binaries to be installed everywhereapt-get install vnc4serverHudson will start/stop xvnc and control $DISPLAY automatically43
On WindowsWindows has the concept called “WindowStation”Equivalent of X server in WindowsWinSta0is the only interactive desktopSo even headless processes can launch browsers. In theory.44
On Windows, in practiceGUI programs can hang with a dialogFor example, IE8 setup wizardThus it’s often easier to just let them access WinSta045
On Windows, really in practiceBut only SYSTEM user can interact with desktopDifferent user account creates different environments that often trip other thingsFallbackConfigure Windows for auto-loginUse start up script to start JNLP slaves46
On MacUser that runs Hudson slaves needs to be also logged in to the desktopThere appears to be no way to run GUI processes headlessly47
ConclusionsIf you aren’t using Hudson yet…What are you waiting for?If you are using Hudson…You probably can get more out of it48
Q&AKohsuke Kawaguchi                 kohsuke@infradna.comInfraDNA, Inc.

More Related Content

PPTX
Hudson
 
PDF
Improving Engineering Processes using Hudson - Spark IT 2010
PPTX
Eclipse DemoCamp Bucharest 2014 - Continuous Integration Jenkins/Hudson
PDF
2013 10-28 php ug presentation - ci using phing and hudson
PDF
Hudson: Your robotic butler
PDF
Gr8conf EU 2013 Speed up your development: GroovyServ and Grails Improx Plugin
ODP
Unit Test Android Without Going Bald
PPTX
Visual Studio Release Management - New weltanschauung or natural evolution? @...
Hudson
 
Improving Engineering Processes using Hudson - Spark IT 2010
Eclipse DemoCamp Bucharest 2014 - Continuous Integration Jenkins/Hudson
2013 10-28 php ug presentation - ci using phing and hudson
Hudson: Your robotic butler
Gr8conf EU 2013 Speed up your development: GroovyServ and Grails Improx Plugin
Unit Test Android Without Going Bald
Visual Studio Release Management - New weltanschauung or natural evolution? @...

What's hot (20)

PDF
You can now use PVS-Studio with Visual Studio absent; just give it the prepro...
PDF
IBM Connect 2014 BP204: It's Not Infernal: Dante's Nine Circles of XPages Heaven
PPT
CI and CD with Jenkins
PPT
Continuous Integration (Jenkins/Hudson)
PDF
Dr. Strangelove, or how I learned to love plugin development
PDF
Jenkins tutorial
PPTX
Jenkins introduction
PPTX
FIWARE Cloud Exercises (Developer's week)
PPTX
TYPO3 & Composer
PDF
How to write a Dockerfile
PPTX
Automated Builds And UI Testing in SharePoint 2010 Development
PPS
10 gui 14
PDF
Building Good Containers for Python Applications
PDF
Trying to Sell PVS-Studio to Google, or New Bugs in Chromium
PDF
codemotion-docker-2014
PPT
Part 2 improving your software development v1.0
PPTX
Jenkins CI
PDF
Mad&pwa practical no. 1
PDF
Yocto no 1 IoT Day da Telefonica/Vivo
PDF
Dependent things dependency management for apple sw - slideshare
You can now use PVS-Studio with Visual Studio absent; just give it the prepro...
IBM Connect 2014 BP204: It's Not Infernal: Dante's Nine Circles of XPages Heaven
CI and CD with Jenkins
Continuous Integration (Jenkins/Hudson)
Dr. Strangelove, or how I learned to love plugin development
Jenkins tutorial
Jenkins introduction
FIWARE Cloud Exercises (Developer's week)
TYPO3 & Composer
How to write a Dockerfile
Automated Builds And UI Testing in SharePoint 2010 Development
10 gui 14
Building Good Containers for Python Applications
Trying to Sell PVS-Studio to Google, or New Bugs in Chromium
codemotion-docker-2014
Part 2 improving your software development v1.0
Jenkins CI
Mad&pwa practical no. 1
Yocto no 1 IoT Day da Telefonica/Vivo
Dependent things dependency management for apple sw - slideshare
Ad

Similar to Hudson@java one2010 (20)

ODP
Jbossworld Presentation
PPTX
SwissJUG_Bringing the cloud back down to earth.pptx
PPTX
Learn About Continuous Integration With Hudson Directly From the Source
PDF
Jenkins CI
PDF
Agile Software Development & Tools
PPTX
Jenkins talk at Silicon valley DevOps meetup
PPT
Continous Integration: A Case Study
PDF
Accelerate Your Automation Testing Effort using TestProject & Docker | Docker...
PDF
DCSF 19 Building Your Development Pipeline
PPTX
Continuous integration ( jen kins travis ci)
PPT
Continuous Integration: A Case Study
PDF
Magento Docker Setup.pdf
PPTX
OpenSouthCode 2016 - Accenture DevOps Platform 2016-05-07
PPT
What's new in p2 (2009)?
DOCX
CodeShip
PDF
Demystifying Docker
PPTX
Demystifying Docker101
PPT
Eclipse summit-2010
PPTX
Continuous integration
PDF
TestWorks Conf Scalable QA with docker - Maarten van den Ende and Adé Mochtar
Jbossworld Presentation
SwissJUG_Bringing the cloud back down to earth.pptx
Learn About Continuous Integration With Hudson Directly From the Source
Jenkins CI
Agile Software Development & Tools
Jenkins talk at Silicon valley DevOps meetup
Continous Integration: A Case Study
Accelerate Your Automation Testing Effort using TestProject & Docker | Docker...
DCSF 19 Building Your Development Pipeline
Continuous integration ( jen kins travis ci)
Continuous Integration: A Case Study
Magento Docker Setup.pdf
OpenSouthCode 2016 - Accenture DevOps Platform 2016-05-07
What's new in p2 (2009)?
CodeShip
Demystifying Docker
Demystifying Docker101
Eclipse summit-2010
Continuous integration
TestWorks Conf Scalable QA with docker - Maarten van den Ende and Adé Mochtar
Ad

Hudson@java one2010

  • 1. Doing More WithYour HudsonKohsuke Kawaguchi kohsuke@infradna.comInfraDNA, Inc.
  • 2. Hudson ProjectOpen-source CI serverWritten in Java but used for all kinds of languagesEmphasis on ease of installation and use“java -jar hudson.war” executionOr your OS-specific packageConfigure everything from browsers2https://hudson-labs.org/
  • 3. EcosystemThe real reason why Hudson thrivesThe same reason why Eclipse thrivesExtensibility330+ community-developed public pluginsUp from 230+ last year in JavaOne3
  • 4. Growing Hudson AdoptionEclipse Community Survey9.1% in 2009 -> 21.8% in 2010Polling by John Ferguson4
  • 7. InfraDNAhttp://infradna.com/I left Oracle this April and started a company around HudsonSupport SubscriptionsOn open-source Hudson + pluginsOr on ICHCI: a commercial value-added distributionConsultingTroubleshooting, custom plugins, etc.Visit us at partner kiosk outside exhibit floor7
  • 8. First steps to CIHudson notices that there is a changeCheck out the source codeDo some beefy computationBuildsRun testsRecord and publish resultsReport to peopleOnly when necessary8
  • 9. But You Can Do So Much More!Once you automate, you can do a lotYet average Hudson users only use very small part of HudsonHence this talk9
  • 10. Interconnecting Jobs aka Workflows10
  • 11. Inter-Project InteractionsYou can host many projects on 1 HudsonLots of ways to have them interact with each otherIncreased parallelism is good11
  • 12. Building Block #1: triggeringWhen a build is done, kick off something else“upstream”: the build that’s kicking off“downstream”: the build being kicked offTriggering is asynchronousA large graph forms a cascading behaviourUpstream doesn’t wait for downstreamDownstream can tell who is the upstream12Foo #10Bar #153
  • 13. Test/Build SeparationDecompose test run into suite runsTrigger them all from buildtimeBuild #10Build #11Build #12…Test X Test YTest Z13
  • 14. Test/Build Separation TechniqueDo not recompile the bits during testsInstead, copy the pre-built bitsUse the copy artifact pluginOr wget/curl via permalink14
  • 15. Build/Deploy Separation1 job to build software, another to deploy itDeployment can be rerun without rebuildFailed build, new server, etc.Helps promote separate responsibilities15
  • 16. Building Block #2: join pluginFan out, and then join back16timeBuild #10Build #11Build #12…Test X Test YReport Generation
  • 18. Building Block #3: FingerprintsMD5 checksums recorded during buildsx.jar in product Z #112 had a bug. Where did it come from? Did it pass all tests?18
  • 19. FingerprintsThe same mechanism works for …Tracking 3rd party dependenciesTracking the version of the build tools usedFingerprinting is critical when you have inter-related jobsUsed to show relationship in UIUsed to aggregate test reports19
  • 20. Build PromotionSometimes plain-vanilla triggering is not desirableExample:Build XRun smoke test suites X1 and X2If they both pass, deploy X to QA siteRun QA testIf it pass, deploy X to staging site20
  • 21. Build PromotionHudson’s solution: build promotionModelDefine a promotion level“Cleared Smoke tests”Define a promotion criteria“Successful completion of X1test and X2test”Define what happens after promotion“Tag the source tree and trigger Xqa”21
  • 22. More Gimmicks in Build PromotionDefine multiple promotionsE.g., “cleared smoke test” & “cleared QA test”Convenient for exposing builds to progressively more expensive workPromotion criteria can be “manual approval”We often can’t automate away everything!Last promoted build gets a permalink22
  • 23. Demo
  • 24. Getting More From Your Maven Builds24Photo: http://www.flickr.com/photos/seeminglee/4556156477/
  • 25. Maven Integration HighlightsPOM comprehensionSimplified configurationMore gets inferred automaticallySnapshot dependencies among builds25
  • 26. Dealing With Large Maven BuildParallel module buildMost projects have short but wide dependency chainLet Hudson build on module-by-module basisWherever possible, build in parallel26
  • 27. Dealing With Large Maven BuildIncremental buildsBuild changed modules and their descendantsSCM determines what has changed27
  • 28. Private Maven RepositoryCause~/.m2/repository is not concurrent safeMultiple builds of the same projectDifferent branches, test environments, JDK, etc.ProblemArtifacts overwrite each otherArtifacts change under the carpetSolutionAllocate ~/.m2/repository per jobConsider a repository manager28
  • 29. After-the-fact DeploymentDefault “mvn deploy” behavior harmfulThink about a failure in the middleBetter to let the build complete, then deploy29
  • 30. Really After-the-fact DeploymentDeploy much after the build has finished30
  • 32. MotivationYou often do the same thing with slight variationsCompile C++ code for different platformsTest with different browsersOr more generally, think of it as32for (x in [a,b,c]) { for (y in [d,e,f]) { …doSomethingWith(x,y,z,..); … }}
  • 33. ModelDefine axesOne axis ≈ one for loopChoose from pre-defined types of AxisSlave axis: pick slaves by their names or their labelse.g., linux, solaris, and windowsGeneric axis: arbitrary values exposed as environment variables33
  • 34. Matrix Project GimmicksFilteringOtherwise combinations increase exponentiallyNot all combinations make senseUse boolean expression to trim down the sizeOr tell Hudson to cut the workload to N%Hudson will thin out the combinations by itself34(label==“windows”).implies(browser==“iexplore”) &&(label==“mac”).implies(browser==“safari”)
  • 35. Demo
  • 37. Selenium GridAutomatically installs Selenium grid binariesOn all Hudson slavesAutomatically start Selenium RCs and connect themStarts Hub on Hudson master port 4444Hudson cluster now accessible as Selenium RC farm from anywhere37
  • 38. Selenium Grid38Use Hudson slaves asSelenium RC nodesHudson slavesHudson master(selenium hub)
  • 39. Choosing EnvironmentUse Hudson’s labels to choose the slave that runs the browser*firefoxRun Firefox somewhere (but no guarantee that the chosen slave has it)windows:*firefoxPick a slave that has the “windows” label, then run Firefox windows&sfbay:*firefoxPick a slave that has both “windows” and “sfbay” label, then run Firefox39
  • 40. Testing with multiple browsersUse “matrix project” typeRun the same tests with different browsersAggregate the results40Firefox testsIE testsSafari tests
  • 41. Selenium Grid w/Hudson BenefitsNo sysadmin overhead for setting up Selenium RCsEspecially nice on heterogeneous networkReuse your existing investment in Hudson cluster41
  • 42. Launching browsers inside HudsonHudson typically runs without GUIaka “Headless”Browsers need GUISo you need to make GUI environment accessible to Hudson and its slaves42
  • 43. On UnixUse Hudson xvncpluginImplements X protocol w/o a real GUINeeds xvnc binaries to be installed everywhereapt-get install vnc4serverHudson will start/stop xvnc and control $DISPLAY automatically43
  • 44. On WindowsWindows has the concept called “WindowStation”Equivalent of X server in WindowsWinSta0is the only interactive desktopSo even headless processes can launch browsers. In theory.44
  • 45. On Windows, in practiceGUI programs can hang with a dialogFor example, IE8 setup wizardThus it’s often easier to just let them access WinSta045
  • 46. On Windows, really in practiceBut only SYSTEM user can interact with desktopDifferent user account creates different environments that often trip other thingsFallbackConfigure Windows for auto-loginUse start up script to start JNLP slaves46
  • 47. On MacUser that runs Hudson slaves needs to be also logged in to the desktopThere appears to be no way to run GUI processes headlessly47
  • 48. ConclusionsIf you aren’t using Hudson yet…What are you waiting for?If you are using Hudson…You probably can get more out of it48
  • 49. Q&AKohsuke Kawaguchi kohsuke@infradna.comInfraDNA, Inc.

Editor's Notes

  • #3: Plugins --- up 80 from last year in JavaOne, contributers – up 70 from last year
  • #5: Plugins --- up 80 from last year in JavaOne, contributers – up 70 from last year