Puppet Release Workflows at Jive Software
5 likes3,326 views
The document describes the release workflows for three different domains of Puppet code at Jive: Core, Hosted, and Continuous Deployment. Core Puppet manages all infrastructure and shared services across about 1000 nodes. It uses a bi-weekly change control process. Hosted Puppet manages customer installations across 14,000 nodes and releases are tied to the application release cycle. Continuous Deployment uses a fully automated deployment pipeline to deploy over 70 services across less than 200 nodes, with puppet code changes triggering test and deployment jobs.
Puppet Release Workflows at Jive Software
- 1. Puppet Release Workflows at Jive Devon Peters
- 2. What is a Release? 2 © Jive confidential
- 3. Push to production, always, immediately 3 © Jive confidential
- 5. Releases… • Software Release Cycle – “A software release life cycle is the sum of the stages of development and maturity for a piece of computer software: ranging from its initial development to its eventual release…” – Wikipedia • Release Management – “the process of managing software releases from development stage to software release” – Wikipedia • Release Engineering – “Automation of release management throughout all stages of release” – Me 5 © Jive confidential
- 6. More Dev in our Ops RY ATO LIG OB PS EVO D AGE IM 6 © Jive confidential
- 7. We have formal release management processes for all of our puppet Domains 7 © Jive confidential
- 8. Different Releases for Different Domains • Core puppet – All infrastructure – Many shared services • Hosted puppet – Hosted Jive installations – Puppet is tied closely to Administration tooling • Continuous Deployment – New-style services – Some infrastructure 8 © Jive confidential
- 9. Releasing Core Puppet 9 © Jive confidential
- 10. Core Puppet • All core infrastructure systems • Most shared services • Puppet+hiera does everything on these systems (very few exceptions) • Simple deployment environment breakdown – Dev, QA, Production (multiple production environments) • Dynamic puppet environments – ‘production’ puppet environment is default – ad-hoc environments are used for testing/staging • Use git flow branching strategy 10 © Jive confidential
- 12. Git Details • Wrappers to simplify and standardize common tasks – j-tech (j-new, j-hotfix, j-commit, many more) – Tied into Jira and Crucible – Feature branches are named after Jira tickets • Code Deployments – Dev/QA: develop is deployed to ‘production’ puppet environment – Prod: master is deployed to ‘production’ puppet environment 12 © Jive confidential
- 13. Testing Changes • Automated – Pre-commit: puppet/erb/ruby/yaml syntax, puppetlint, hiera-gpg • Local – Vagrant VM – Push branch out to dev puppet master • Environment Specific – Push branch to env-specific puppet master 13 © Jive confidential
- 14. Merging Changes • j-review – Submit a code review of your branch • j-commit – Merges feature branch to develop – Merges hotfix branch to master and develop • develop branch is deployed to Dev on commit – jenkins • develop branch is deployed to QA Mon-Fri @ 10am – jenkins • If it’s not a hotfix it won’t go to production yet… – Technically, this because the change isn’t in the ‘master’ branch yet, but there’s more to it than that 14 © Jive confidential
- 15. Change Control for Production • Bi-weekly CC meetings – Monday and Thursday • Puppet changes go through CC process – Hotfixes can be promoted outside of CC process – Weekly change windows for high-impact changes • If it’s a puppet change, it’s done as a hotfix • Puppet release started every Thursday @ 4pm – j-release -S: starts a release branch – Jenkins runs this, and generates a CCR ticket with all commits – Changes are reviewed in Monday CC meeting • Puppet release finished every Tuesday – j-release -F: finishes a release branch (manual) – Jenkins code deployment jobs are triggered manually 15 © Jive confidential
- 16. Core Release Overview 16 © Jive confidential
- 17. Releasing Hosted Puppet 17 © Jive confidential
- 18. Hosted Puppet • Nodes that run hosted customer installations • Very homogenous • Relatively simple puppet code – Puppet mostly supplements an in house administration tool (JCA) • Deployment environment breakdown – Dev, QA, Prod • Uses static puppet environments – ENC dictates the environment for a given node – Jenkins jobs deploy from git to appropriate puppet environment • Uses the git flow branching strategy – All the same j-tech tools 18 © Jive confidential
- 19. Testing Changes • Automated – Pre-commit: puppet/erb/ruby/yaml syntax, puppetlint, hiera-gpg • Get your own installation setup in Dev – Commit changes, and don’t walk away till you know they work • It’s somewhat acceptable to break dev, but try not to – Most OS related changes are just plucked from Core, and were likely tested more thoroughly there 19 © Jive confidential
- 20. Merging Changes • j-review – Submit a code review of your branch • j-commit – Merges feature branch to develop – Merges hotfix branch to master and develop • develop branch is deployed to Dev on commit – jenkins – make sure you don’t break it • develop branch is deployed to QA ad-hoc – QA changes are tied to JCA application release QA cycles – jenkins 20 © Jive confidential
- 21. Creating Releases • Puppet releases are typically tied to JCA application releases – The app releases every 2 weeks – A puppet release could be a part of it • Doing a release – j-release -A: create, and finish a release branch 21 © Jive confidential
- 22. Staged Production Deployments • UAT – UAT nodes use the same puppet infrastructure as production – Jenkins deploys master to the ‘hosted_uat’ puppet environment • Production – Jenkins deploys master to the ‘hosted’ puppet environment 22 © Jive confidential
- 23. Hosted Release Overview 23 © Jive confidential
- 24. Continuous Deployment 24 © Jive confidential
- 25. CD Overview • Deployable – Framework for deploying java (and other) services • Many java services (over 70) • Data Infrastructure – Kafka, Hadoop, HBase, SenseiDB, Elasticsearch • Other Infrastructure – Puppet, Nginx, Sensu, OpenTSDB • Gerrit – Code reviews are mandatory – Branch strategy is still git flow style • Complex Puppet Code • Complex Deployment Pipeline 25 © Jive confidential
- 26. Environment Overview • We call them clusters – virtual, dev, integ, test, release, preprod, prod • We also have geo-specific datacenters – local, intinteg, inttest, intrelease, phxpreprod, phxprod, amsprod • Hiera hierarchy includes all of these – These exist in hiera for other Puppet domains as well • Deployable configuration hierarchy includes all of these 26 © Jive confidential
- 27. Puppet Details 27 © Jive confidential
- 28. Puppet Overview • Dedicated puppet master(s) per cluster • Puppet agent is run on-demand by the deployment process • CD pipeline determines if puppet code can be promoted to the next cluster • “Special” module and hiera trees – We don’t want to duplicate everything in Core – Developers need the ability to change puppet code or hiera data – We setup something we call Puppet for Projects • Combines Core puppet code with Project puppet code 28 © Jive confidential
- 29. Puppet for Projects - Layout • Core puppet – Basic layout is: • hiera/ • manifests/ • modules/ – manifests/site.pp is basically: hiera_include(‘classes’) – Every commit triggers an artifact build job (jenkins) • Artifacts are uploaded to a Nexus repo, as puppet-0.0.1-<count>-<committish> • Project puppet – A project is basically a repo – The project repo contains the following directories: • puppet/hiera • puppet/modules – Contains maven configuration for Core puppet artifact and Combined puppet artifact 29 © Jive confidential
- 30. Puppet for Projects – Configuration • puppet.conf – modulepath = /path/modules:/path/project/modules • hiera.yaml – project/%{some-hierarchy} – %{some-hierarchy} 30 © Jive confidential
- 31. Puppet for Projects - Artifact • Combined artifact – A commit to the puppet code in the project triggers a combined artifact build – Artifact contains: • • • • • hiera/ hiera/project manifests/ modules/ project/modules (from Core artifact) (from puppet/hiera in the project repo) (from Core artifact) (from Core artifact) (from puppet/modules in the project repo) – Module Collisions • If a module with the same name exists in both, the project always wins and the Core module is excluded from the final artifact 31 © Jive confidential
- 33. Deployable Framework • Provides standardized… – Configuration • j-config – CLI or service • Hierarchical JSON – Logging • log-publisher service, writes to Logstash – Metrics • metric-publisher service, writes to OpenTSDB – Monitoring • Autogenerated sensu checks – Deployment • Supports multiple run phases – Service Management • j-status, j-start, j-stop 33 © Jive confidential
- 34. Puppet as a Deployable • puppetmaster-deployable – Target: puppet master – puppet tree is packaged into a <release-ver> artifact – Artifact is deployed to /etc/puppet/environments/jive_<release-ver> on the puppet master(s) • puppet-deployable – Target: all systems – j-start executes: puppet agent --environment jive_<release-ver> – If puppet fails, the deploy fails and stops • <release-ver> is always the version of the artifact being built/deployed – 0.0.1-<count>-<committish> – The deployment process converts the string to a puppet safe string • 0_0_1_<count>_<committish> 34 © Jive confidential
- 35. Deployment Run Phases • ops-tools – Deploy j-tech first so everything else will work • puppetmaster – Get out teh codes • puppet – Run puppet – Includes Hadoop, HBase, Zookeeper, Elasticsearch • pre – Deploy core/base services – Includes Kafka, and SenseiDB • Main – Everything else 35 © Jive confidential
- 36. Making a Puppet Change 36 © Jive confidential
- 37. Making a Puppet Change – Virtual • Vagrant VM – It’s big • minimum 4CPU, 8GB – for the VM alone – Full stack gets deployed and run • Hadoop, Kafka, etc, etc, etc – even all 70+ services if you want – j-vm -r • Fetches proper Core puppet artifact • Builds and deploys a puppet tree for vagrant, based on your local git repo • Executes vagrant puppet provisioner – Once it works on the vm, submit a review to Gerrit 37 © Jive confidential
- 38. Making a Puppet Change – Integ • Once the review is submitted, jenkins will: – Build Artifacts – Launch a VM-test job • Validating that what you did works on a vagrant VM there – Launch an integ deployment • Validating that things work in a multi-node non-vagrant environment – Comment on your review with Validated +1, or -1 (depending on results) – Once other reviewers give you +2 Gerrit will merge your commit – Once it’s merged… 38 © Jive confidential
- 39. Making a Puppet Change – Test • Once the commit is merged, jenkins will: – Build artifacts – Launch a Test deployment – Run more extensive tests – If the deployment and all tests pass, it’s ready for the next step 39 © Jive confidential
- 40. Making a Puppet Change – Release • Daily at 8am all commits that have passed Test are merged from develop to master • Once this happens, jenkins will: – Build artifacts – Trigger a Release deployment – Rerun all of the tests – If all tests pass… 40 © Jive confidential
- 41. Making a Puppet Change – Preprod • If all tests pass, jenkins will: – Promote artifacts to preprod Nexus repo – Trigger a Preprod deploy – At this point, everything should be stable 41 © Jive confidential
- 42. Making a Puppet Change – Prod • During the next scheduled production release, someone will: – Trigger a Prod deploy – Currently done manually 42 © Jive confidential
- 44. That’s about it… 44 © Jive confidential
- 45. Review • Core – Complex puppet code to manage everything – Releases tied to Change Control – ~1000 nodes • Hosted – Relatively simple puppet code – Releases tied to Administration tool’s application releases – ~14000 nodes • Continuous Deployment – Complex puppet code – Fully Automated Release and Deployment – <200 nodes (but growing) 45 © Jive confidential
- 46. Thank You! © Jive confidential 46