![Feel the pain [1]
• Unknown status of PR](https://image.slidesharecdn.com/devopsstage-181014171913/85/CI-doesn-t-start-with-Jenkins-26-320.jpg)
![Feel the pain [1]
• Unknown status of PR
• Lack of notifications](https://image.slidesharecdn.com/devopsstage-181014171913/85/CI-doesn-t-start-with-Jenkins-27-320.jpg)
![Feel the pain [1]
• Unknown status of PR
• Lack of notifications
• No CD](https://image.slidesharecdn.com/devopsstage-181014171913/85/CI-doesn-t-start-with-Jenkins-28-320.jpg)
![Feel the pain [1]
• Unknown status of PR
• Lack of notifications
• No CD
• A lot of manual work](https://image.slidesharecdn.com/devopsstage-181014171913/85/CI-doesn-t-start-with-Jenkins-29-320.jpg)
![Feel the pain [1]
• Unknown status of PR
• Lack of notifications
• No CD
• A lot of manual work
• Unknown time for QA](https://image.slidesharecdn.com/devopsstage-181014171913/85/CI-doesn-t-start-with-Jenkins-30-320.jpg)
![Feel the pain [1]
• Unknown status of PR
• Lack of notifications
• No CD
• A lot of manual work
• Unknown time for QA
• Unstable tests](https://image.slidesharecdn.com/devopsstage-181014171913/85/CI-doesn-t-start-with-Jenkins-31-320.jpg)
![Feel the pain [2]
• Conflicts between PRs](https://image.slidesharecdn.com/devopsstage-181014171913/85/CI-doesn-t-start-with-Jenkins-45-320.jpg)
![Feel the pain [2]
• Conflicts between PRs
• Need to wait for QA](https://image.slidesharecdn.com/devopsstage-181014171913/85/CI-doesn-t-start-with-Jenkins-46-320.jpg)
![Feel the pain [2]
• Conflicts between PRs
• Need to wait for QA
• You cannot merge
directly](https://image.slidesharecdn.com/devopsstage-181014171913/85/CI-doesn-t-start-with-Jenkins-47-320.jpg)
![Feel the pain [2]
• Conflicts between PRs
• Need to wait for QA
• You cannot merge
directly
• Hotfixes are harmful](https://image.slidesharecdn.com/devopsstage-181014171913/85/CI-doesn-t-start-with-Jenkins-48-320.jpg)
CI doesn’t start with Jenkins
- 1. CI doesn’t start with Jenkins DevOps Stage - 2018: October 12-13
- 2. Bio •CatOps Engineer @ Preply Inc •Twitter: @grem11n •Telegram: @grem1in • CatOps • Kyiv HashiCorp User Group
- 3. Agenda Pain CICD evolution @ Preply External dependencies for CI Tools Cats
- 4. Let’s start with Jenkins, tho
- 6. In the very beginning
- 7. In the very beginning
- 8. In the very beginning Separate teams
- 9. In the very beginning Separate teams Monolith
- 12. Skullcandy flow (code) https://www.endpoint.com/blog/2014/05/02/git-workflows-that-work 1. Create a pull request to the master
- 13. Skullcandy flow (code) https://www.endpoint.com/blog/2014/05/02/git-workflows-that-work 1. Create a pull request to the master 2. Label it once ready
- 14. Skullcandy flow (code) https://www.endpoint.com/blog/2014/05/02/git-workflows-that-work 1. Create a pull request to the master 2. Label it once ready 3. Gather all “ready PRs” into ephemeral branch
- 15. Skullcandy flow (code) https://www.endpoint.com/blog/2014/05/02/git-workflows-that-work 1. Create a pull request to the master 2. Label it once ready 3. Gather all “ready PRs” into ephemeral branch 4. Test it!
- 16. Skullcandy flow (code) https://www.endpoint.com/blog/2014/05/02/git-workflows-that-work 1. Create a pull request to the master 2. Label it once ready 3. Gather all “ready PRs” into ephemeral branch 4. Test it! 5. Merge QA branch to the master
- 17. Skullcandy flow (artifact) https://www.endpoint.com/blog/2014/05/02/git-workflows-that-work 1. QA artifact is created from the ephemeral QA branch
- 18. Skullcandy flow (artifact) https://www.endpoint.com/blog/2014/05/02/git-workflows-that-work 1. QA artifact is created from the ephemeral QA branch 2. Production artifact is created after QA branch is merged to master
- 19. Skullcandy flow (pros) https://www.endpoint.com/blog/2014/05/02/git-workflows-that-work • We can live with the monolith
- 20. Skullcandy flow (pros) https://www.endpoint.com/blog/2014/05/02/git-workflows-that-work • We can live with the monolith • Release candidates every day
- 21. Skullcandy flow (pros) https://www.endpoint.com/blog/2014/05/02/git-workflows-that-work • We can live with the monolith • Release candidates every day • Works pretty well with GitHub
- 22. Skullcandy flow (pros) https://www.endpoint.com/blog/2014/05/02/git-workflows-that-work • We can live with the monolith • Release candidates every day • Works pretty well with GitHub • Clear priorities
- 23. Skullcandy flow (cons) https://www.endpoint.com/blog/2014/05/02/git-workflows-that-work • PRs could have conflicts with each other
- 24. Skullcandy flow (cons) https://www.endpoint.com/blog/2014/05/02/git-workflows-that-work • PRs could have conflicts with each other • You need to wait QA before resolve the conflicts
- 25. Dependencies • A tool • Development flow
- 26. Feel the pain [1] • Unknown status of PR
- 27. Feel the pain [1] • Unknown status of PR • Lack of notifications
- 28. Feel the pain [1] • Unknown status of PR • Lack of notifications • No CD
- 29. Feel the pain [1] • Unknown status of PR • Lack of notifications • No CD • A lot of manual work
- 30. Feel the pain [1] • Unknown status of PR • Lack of notifications • No CD • A lot of manual work • Unknown time for QA
- 31. Feel the pain [1] • Unknown status of PR • Lack of notifications • No CD • A lot of manual work • Unknown time for QA • Unstable tests
- 32. Dependencies • A tool • Development flow • Alignment
- 33. Dependencies • A tool • Development flow • Alignment • Sane tests
- 34. Skullcandy 2.0 • GitHub API
- 35. Skullcandy 2.0 • GitHub API • Merge constraints
- 36. Skullcandy 2.0 • GitHub API • Merge constraints • Code ownership
- 37. Skullcandy 2.0 • GitHub API • Merge constraints • Code ownership • Codestyle
- 38. Skullcandy 2.0 • GitHub API • Merge constraints • Code ownership • Codestyle • Sane tests
- 39. Skullcandy 2.0 • GitHub API • Merge constraints • Code ownership • Codestyle • Sane tests • UI tests
- 40. Skullcandy 2.0 • GitHub API • Merge constraints • Code ownership • Codestyle • Sane tests • UI tests • QA session limits
- 41. Skullcandy 2.0 • GitHub API • Merge constraints • Code ownership • Codestyle • Sane tests • UI tests • QA session limits • More notifications
- 42. Dependencies • A tool • Development flow • Alignment • Sane tests • UI tests
- 43. Dependencies • A tool • Development flow • Alignment • Sane tests • UI tests • Discipline
- 44. So, are we done?
- 45. Feel the pain [2] • Conflicts between PRs
- 46. Feel the pain [2] • Conflicts between PRs • Need to wait for QA
- 47. Feel the pain [2] • Conflicts between PRs • Need to wait for QA • You cannot merge directly
- 48. Feel the pain [2] • Conflicts between PRs • Need to wait for QA • You cannot merge directly • Hotfixes are harmful
- 49. Gitflow? https://nvie.com/posts/a-successful-git-branching-model/ • Everybody likes Gitflow • Easy to automate • Clear enough
- 50. Gitflow https://nvie.com/posts/a-successful-git-branching-model/ • Everybody likes Gitflow • Easy to automate • Clear enough • Broken development branch is a company- wide problem
- 51. Trunk-based development! https://www.toptal.com/software/trunk-based-development-git-flow • Only one mainline • Clear releases (tags) • Everyone is independent • Hotfixes are the same entities
- 52. Trunk-based development https://www.toptal.com/software/trunk-based-development-git-flow 1. Feature branch is created and tested separately
- 53. Trunk-based development https://www.toptal.com/software/trunk-based-development-git-flow 1. Feature branch is created and tested separately 2. Feature is merged directly into the master when it’s ready
- 54. Trunk-based development https://www.toptal.com/software/trunk-based-development-git-flow 1. Feature branch is created and tested separately 2. Feature is merged directly into the master when it’s ready 3. New tag is created from the master branch, which goes to production
- 55. Trunk-based development https://www.toptal.com/software/trunk-based-development-git-flow 1. Each feature is a separate artifact for QA
- 56. Trunk-based development https://www.toptal.com/software/trunk-based-development-git-flow 1. Each feature is a separate artifact for QA 2. Production artifact is created from tag
- 57. Wait, but … • What if trunk is broken?
- 58. Wait, but … • What if trunk is broken? • Test better!
- 59. Wait, but … • What if trunk is broken? • Test better! • Where can I test?!
- 60. Wait, but … • What if trunk is broken? • Test better! • Where can I test?! • …
- 62. Welcome, Kubernetes! You cannot just put Kubernetes to fix everything!11 Me:
- 63. Dynamic env for each PR • Kubernetes • Helm • Jenkins
- 64. DB for each environment! Django Anonymizer: • Preply & OpenSource • Utilizes Django ORM • Safe data for development
- 65. Dependencies • A tool • Development flow • Alignment • Sane tests • Discipline • UI testes • Infrastructure
- 66. Deal with it!
- 67. Let’s add some services! • Monorepo or not? • Repeatable CI • Env for the whole thing
- 68. And we also have Lambdas • CI for Lambdas • LocalStack • Docker
- 69. And we also have Lambdas • CI for Lambdas • LocalStack • Docker • CD for Lambdas • Serverless framework • Terraform
- 70. Dependencies • A tool • Development flow • Alignment • Sane tests • Discipline • UI testes • Infrastructure • Architectural planning
- 71. Dependencies • A tool (tech, ops) • Development flow (tech, mgmt) • Alignment (cult, mgmt) • Sane tests (tech, dev) • Discipline (cult, all) • UI testes (tech, qa) • Infrastructure (tech, ops) • Architectural planning (tech, all)
- 72. A Toolset
- 73. A Toolset Jenkins: • Multibranch project • Declarative pipeline • Shared libraries
- 74. A Toolset Jenkins: • Multibranch project • Declarative pipeline • Shared libraries Shared libraries are pretty much Groovy :(
- 75. A Toolset Dynamic Env: • Kubernetes on Spot • Helm
- 76. A Toolset Dynamic Env: • Kubernetes on Spot • Helm • Chart for entire environment is complex • Helm 2 vs Helm 3
- 77. Where are we going to?
- 78. Where are we going to? • Stable CICD flow for existing monolith • Survive in the Microservices world • Figure out deploy to k8s (to Helm or not to Helm) • Canary (Blue/Green) deployments • Perhaps try JenkinsX • Do something with CICD for Lambdas
- 79. Thank you! yrochnyak@gmail.com TG: @grem1in && @catops Questions?