Continuous Testing in Vegas
Download as PPTX, PDF1 like284 views
The document discusses continuous testing and its role within Agile methodologies, emphasizing the importance of fast feedback loops in software development. It outlines key concepts such as continuous integration, deployment, and testing processes, while introducing the GROWS methodology to improve the software development life cycle. The document also highlights the benefits of collaboration between development and QA teams and provides practical exercises to foster understanding and adoption of these practices.
Continuous Testing in Vegas
- 1. Continuous Testing Jared Richardson Agile Artisans @JaredRichardson GrowsMethod.com
- 2. Who am I?
- 3. 3 Cofounder of GROWS Methodology Author Ship It! and Career 2.0 Screencast editor PragProg.com 2nd public signatory of the Agile Manifesto Started AgileRTP in 2007 Coach AgileArtisans.com Jared Richardson
- 4. Who are you?
- 5. What’s up with the material? Major rewrite Introduction Slides on USB On Slideshare.com Provided to conference as well
- 6. Who’s a User? Already using continuous integration? Continuous deployment? Which tools? Test first?
- 7. What would you say you do here? Roles? Responsibilities? Why are you here? What are your tests for this session?
- 8. Objectives Understand Continuous Feedback in context of a healthy SDLC Understand the Continuous Paradigm Understand an end-to-end Continuous workflow Understand how to scale your efforts
- 9. Outline GROWS Methodology Continuous paradigm Battleship! Continuous Integration (benefits & workflow) Continuous Testing (benefits & workflow) Miscellaneous tips & tricks
- 10. Breaks Halfway (90 minutes)? Each hour ? Speak up!
- 11. SDLC
- 12. Idea Verifies Use or delete Implements Software Development Life Cycle
- 15. Requirements Write Code The Traditional Approach Demo Rewrite (60%) “Discussion” The architects generate a document and hand it off An isolated event, primarily worked by the customer proxy Arch Code like mad! We’re starting late! Not what we meant! Wrong arch Ignored my docs etc so on
- 16. Demo Write Code Requirements A More Agile Approach Before significant time and $$ is invested Discuss Include the Golden Triad Dev QA Customer Rewrite
- 17. Demo Write Code Architecture Proof of Concept Requirements Arch
- 18. Why Feedback? Feedback is good for budgets Cheaper Faster More accurate Lower risk
- 19. flickr.com/photos/alancleaver/4293345631 To Err is Human What’s your recovery plan?
- 20. Experimentation Learn about software by writing software Can’t study your way into experience Like riding a bike Create feedback loops flickr.com/photos/schwar/6801765768
- 21. Requirements Rewrite Discuss Node.js Bootstrap Angular.js Pick one SOA Monolithic Peer to peer Pick one Developers disagree on which architecture is best Continue until you can demo Developers disagree on which Javascript toolkit is best Demo Revise Rewrite Adjust Frequent demos discover communication failures early
- 23. Decide! Learn Learn Decide! Two Paths
- 27. Acceptance We’re not very good We’re going to mess up Mistakes are inevitable How do we recover?
- 29. Experimentation We’re not very good We’re going to mess up Mistakes are inevitable How do we recover?
- 31. Automated Tests Answer Questions Is this done? Does it still work? Does it work somewhere besides my machine? This is what I’m asking you to write. Everything is checked in.
- 32. flickr.com/photos/bobowen/4501875907 Product or system flickr.com/photos/30691679@N07/3808395213 Module/ Sub System flickr.com/photos/kevino/1438502858 Class
- 34. Agility Is... Tighter Feedback Loops
- 35. Which Level? 35
- 37. C*/Board 37
- 38. VP 38
- 42. flickr.com/photos/89228431@N06/11348332905 Minimize Our Dark Times Code Change Validation
- 43. Fast feedback leads to fast fixes flickr.com/photos/bogenfreund/556656621
- 44. Different Levels Manual acceptance tests Unit tests Continuous Testing
- 45. Different Levels Good at the bottom Great at the top Ignore the middle
- 46. Continuous Feedback Continuous Integration Continuous Delivery Continuous Testing Continuous Monitoring
- 47. Continuous deployment is controversial. Most people ... think I’m advocating low-quality code or an undisciplined cowboy-coding development process. Eric Ries radar.oreilly.com/2009/03/continuous-deployment-5-eas.html
- 48. Language is a Tool Use it
- 49. Exercise How long is your SDLC (idea to verification)? What are your SDLC feedback bottlenecks? Write down 3 worst bottlenecks Gather into groups of 5 to 7 Share and discuss Keep your list! Take ten minutes
- 51. Continuous Integration Watch Code Build product Run tests Publish results
- 52. Pros Fast feedback Known state Easily consumed history Shines a light 52
- 53. Cons Slows you down Nags you after every code change Exposes problems Shares issues Forces you to deal with problems Smoke alarm 53flickr.com/photos/117168856@N06/12486837493
- 54. 54 Work Time Perception of Problem Accumulation
- 57. Sidebar: How We Learn Dreyfus Model of Skills Acquisition 57
- 59. Key Points Skills don’t map 59dilbert.com/strip/1990-09-13
- 60. Key Points Beginners need steps 60
- 61. Steps 1. Use source code management (SCM) 2. Check in ~everything~ 3. Automate the build (script) 4. Acquire a build environment (isolated) 5. Install build tools (dev tools) 61
- 63. Continuous Integration Servers Many Jenkins Cruise Control Bamboo Pulse TFS 63
- 64. Jenkins Open source Established Split from Hudson Java application Any application server Self contained 64
- 65. Example Publish results javac (Oracle & IBM) message broker etc ws ejb deploy message broker tools sshexec (to scripts) JUnit HP's UFT Selenium Cucumber Web page RSS Email
- 67. Dependent Builds My build depends on another team’s build No problem We’ll trigger your build based on their results 67
- 69. Complicated Dependencies Our software is too complicated for CI 69 flickr.com/photos/chaoticmind75/16169722270/
- 70. Dependent Builds Data Layer Data Services Rich Client Common Libs Web Portal Biz Services
- 71. What’s Big? 3,000 developers 300 major projects All related Platform to UI Multiple branches 71
- 73. Chained Continuous Integration Team 1 Team 2 Team 3 Integration build
- 75. Platform Testing Team 1 Team 3Team 2 Team 4 Team 5 Integration Build Platform1 Plat 3Plat 2 Plat 4 Plat 5
- 76. Battleship!
- 77. Each player has a board flickr.com/photos/spookyamd/6333317567 flickr.com/photos/jking89/1363983490 Place your fleet Drop your bombs
- 78. Two Ways to Play flickr.com/photos/compacflt/22797421725 flickr.com/photos/bz3rk/2436165683 Use all your ammo quickly Use ammo one shot at a time
- 79. Round 1 Start a timer Place your fleet Use all your “ammo” Check with your opponent Record hits/misses/time Play twice Round 2 Start a timer Place your fleet Fire 1 round Check with your opponent Continue until out of ammo Record hits/misses/time Ships & Ammo 1 Sub (2 dot ship) 2 Destroyers (3 dots ships) 1 Battleship (4 dot ship) 1 Aircraft carrier (5 dot ship) 35 rounds to fire
- 80. Results Which way is faster? Which spends less “budget”? Which way is more effective? Which way do you prefer?
- 81. Feedback Faster with feedback? Slower with feedback? Is slower bad?
- 83. A B C D E F G H I J 1 2 3 4 5 6 7 8 9 10 Opponent’s Fleet A B C D E F G H I J 1 2 3 4 5 6 7 8 9 10 My Fleet A B C D E F G H I J 1 2 3 4 5 6 7 8 9 10 Opponent’s Fleet A B C D E F G H I J 1 2 3 4 5 6 7 8 9 10 My Fleet
- 84. A B C D E F G H I 1 2 3 4 5 6 7 8 9 A B C D E F G H I 1 2 3 4 5 6 7 8 9 A B C D E F G H I 1 2 3 4 5 6 7 8 9 A B C D E F G H I 1 2 3 4 5 6 7 8 9 A B C D E F G H I 1 2 3 4 5 6 7 8 9 A B C D E F G H I 1 2 3 4 5 6 7 8 9 A B C D E F G H I 1 2 3 4 5 6 7 8 9 A B C D E F G H I 1 2 3 4 5 6 7 8 9 Time Shots Misses Hits Sinks % hits % sinks 1st All at once 2nd All at once 3rd Interactive 4th Interactive Battleship! An exercise by Tony Brill1 2 3 4
- 86. Continuous Integration Watch Code Build product Run tests Publish results
- 87. Continuous Testing Watch CI Deploy product Integration tests Publish results
- 88. Continuous Testing CI CT Compile & unit test Deploy & integration test
- 91. SCMDevelopers Watch Build Unit Tests Deploy to Dev Integration Tests First Look Continuous Testing On Demand Deploy to QA Manual Testing AgileArtisans.com
- 92. Continuous Testing Build in CI (compile & unit tests) Deploy to dev (staging) Run integration tests
- 93. What does this tell us? If it works
- 94. How? By keeping fast feedback fast Providing slow feedback slowly Don’t tie fast feedback to slow feedback
- 95. Fast vs Slow “Fast” Integration Tests Compiles & Unit Tests Slow Integration Tests 5-10 minutes 30 to 60 4 to 8 hours
- 96. What's this look like?
- 97. Continuous Testing CI CT Compile & unit test Deploy & integration test
- 98. Multi Platform Deploys CI CD Win7 CD Linux CD OS X
- 99. Exercise Remember your SDLC bottlenecks? What could continuous integration solve? What could continuous testing solve? Regroup with your team and discuss Bring your top 3 solutions 99
- 100. As We Explore a Continuous World... 10 0
- 102. Nuke and Pave Reset (completely) Start over Completely Scripted 10 2
- 103. Binaries Nuke & Pave Delete Recompile Deploy Use 10 3 flickr.com/photos/x-ray_delta_one/4152356464
- 104. Data Nuke & Pave Delete schema Drop tables Recreate Restock Rails Liquibase 10 4 flickr.com/photos/x-ray_delta_one/4152356464
- 105. Rails ActiveRecord CREATE TABLE products ( id int(11) NOT NULL auto_increment, name varchar(255), PRIMARY KEY (id) ); 10 5
- 106. Rails ActiveRecord change_table :card_deck do |table| tablet.remove :description, :name tablet.string :first_name tablet.index :part_number tablet.rename :list_name, :last_name end 10 6
- 107. Rails ActiveRecord rake db:migrate rake db:migrate VERSION=0 rake db:reset 10 7
- 108. ActiveRecord rake db:reset Drops schema Recreates schema 10 8
- 109. Operating System Nuke & Pave Virtualize Recreate 10 9 flickr.com/photos/x-ray_delta_one/4152356464
- 110. Steps for a New OS Install VirtualBox Install Vagrant vagrant init hashicorp/precise32 vagrant up 11 0
- 111. Benefits Scriptable Reproducible environment Solid results Fast results Repeatable results
- 113. “If you don’t do it on Day One, you probably won’t” Andy Hunt 113andy.pragprog.com
- 115. Costs Move to Zero Once scripted… What changes?
- 116. How do We Improve Our Feedback Loops?
- 117. Developers Devs write code normally SCM Code is shared via a source code management tool Continuous Testing Providing immediate feedback and a shared environment for a QA “First Look” Triggered Deploy to QA QA team members trigger this deploy Manual Testing A stable environment for manual and exploratory testing
- 118. SCMDevelopers Watch Build Unit Tests Deploy to Dev Integration Tests First Look Continuous Testing On Demand Deploy to QA Manual Testing Rapid feedback Pair Dev & QA
- 119. First Look • QA can see what’s been coded immediately • Differences in opinions can be discussed quickly • Fewer problems during the larger testing cycles • Keeps QA & Dev working together as partners • Minimizes feedback for the previous iteration • Faster feedback for QA
- 120. Continuous Testing • Includes QA in the continuous feedback cycle • Retains a traditional QA stable testing environment • Dev & QA should both review all tests (unit & int.) • QA requests trigger the QA env. deployment
- 121. Alternatives
- 124. Be Lazy! You have builds scripted CI servers running Tests written Things deploy What else can you do? flickr.com/photos/ofernandezberrios/2720567252
- 126. Traditionally… Someone needs the app Notices it’s down URGENT! FIX! Fire drill flickr.com/photos/kaibara/5332191131
- 127. Detect Problems Before it’s urgent Detect Notify Fix
- 132. Steps Continuous Integration (aka compiles) flickr.com/photos/jalalspages/842122500
- 133. Steps Continuous Integration (aka compiles) Test coverage numbers (aka unit tests) flickr.com/photos/jalalspages/842122500
- 134. Steps Continuous Integration (aka compiles) Test coverage numbers (aka unit tests) Continuous deployment (aka deploys to dev/test) flickr.com/photos/jalalspages/842122500
- 135. Steps Continuous Integration (aka compiles) Test coverage numbers (aka unit tests) Continuous deployment (aka deploys to dev/test) Continuous testing (aka integration tests) flickr.com/photos/jalalspages/842122500
- 136. Steps Continuous Integration (aka compiles) Test coverage numbers (aka unit tests) Continuous deployment (aka deploys to dev/test) Continuous testing (aka integration tests) Continuous monitoring (test against environs) flickr.com/photos/jalalspages/842122500
- 137. Culture Change Fundamental change in development
- 138. Steps Automate builds Continuous Integration Add unit tests Automate deployments Continuous Delivery Add integration tests flickr.com/photos/untitlism/25474234 65
- 139. Your Move
- 140. Exercise At lunch Continuous Testing Two 3x5 cards 3 reasons for 3 barriers to adoption Groups of 5 to 7 Brainstorm on solutions flickr.com/photos/curtisperry/6119618990
- 141. Exercise Useful or not... Will you pursue or not... Continuous Integration Continuous Testing Continuous Monitoring Test automation How will these ideas help or hurt your org? Write down your answers and pass them up
- 145. An Addendum
- 146. Test Automation Unit tests Acceptance Tests
- 147. Unit Tests One class One method Fast Illuminating
- 148. Integration Tests High level Lots of coverage No detail
- 149. Continuous Feedback Large amounts of code Small amounts of change Tests run after every change Requires frequent code checkins
- 150. What's Better? Unit or Integration? 150
- 152. Test Lifecycle Set up state Use product Verify results (Clean up) 152
- 153. Set Your Teams Up Builds work Functionality works Biggest risk? Requirements
- 155. Grid Driven Development Remove technical barriers Everyone speaks “Excel”
- 156. Testing Strategies Test First is best Defect Driven Testing Blitzkrieg Testing
- 157. Tips Open standards... Junit or Junit compatible Avoid vendor lock in Glue tools Time matters.... Fast servers Incremental change
- 158. Script Tips Save scripts in source code management Modular Basic encapsulation One target => One function
- 159. Script Targets all depends_on clean, compile, test-unit clean # deletes build artifacts compile # builds the product (Ruby?) test-unit # run your tests deploy_locally # to your dev environment deploy_dev (?) # to a shared dev environment test-integration # run your integration tests
- 160. Resources JUnit JUnit.org Jenkins Jenkins-ci.org BDD Dannorth.net/introducing-bdd Cucumber Cukes.info Continuous Deployment en.wikipedia.org/wiki/Continuous_delivery
- 163. Additional Content Material for you to review later…
- 164. Release Planning Releases contain features How many are ready to ship? 164
- 166. Releases contain features Team Release Features Stories Stories Complete % Done A 7/1 25 392 17 0.23 B 7/15 13 208 104 0.5 C 7/12 96 2,400 150 0.16 D 12/30 476 ?? 234 ?
- 167. Release date Burn Up Chart 167 Time Features Features in the release Required rate of work
- 168. Will we hit our date? 168 Time Features ACTUAL rate of work
- 169. Burn Up Chart 169 Time Features ACTUAL rate of work
- 172. The Solution Reduce features DO NOT focus on increased work 172
- 173. Good or Not? 173 Time Features ACTUAL rate of work
- 174. What’s the Fix? 174 Time Features How do I make the team work harder?
- 179. What Happens to Overloaded Planes? 179 flickr.com/photos/jameski/5322886899