Building a World-Class Quality Team at eBay
- 2. ! ! !!!!! ! !!!! With more than thirty years of experience in the software industry, Steve Hares remembers keypunch machines, twenty-four hour batch turn-around time, and acoustic coupling modems. Steve has worked in multiple industries from networking and telecommunications to Internet security and eCommerce, as a developer, development manager, and QA leader. At eBay for six years running the quality engineering organization for the customer support technology group, Steve has built a world-class team from the ground up. It now tests everything from enterprise-wide products to desktop applications, from ETL files to native language IVRs. Steve Hares eBay
- 4. 5/18/15! StarEast!Conference,!Orlando,!May!2015! 2! A)Li:le)About)Me) ! • I’ve!been!in!the!soEware!industry!for!over!30!years! • The!past!6!at!eBay!in!the!Customer!Support! Technology!division! • Grew!my!QE!team!from!4!people!to!60,!one!of!the! largest!at!eBay! • Load!and!performance,!heavy!automaPon!reliance,! and!mulPple!test!methodologies! • All!in!support!of!delivering!world!class!customer! support! • Based!on!NPS!scores,!one!of!the!highest!rated!CS! experiences!in!the!market!segment!
- 5. Agenda) ! • What!is!world!class!QA! • How!do!we!get!to!be!world!class! • HypothePcal!QA!team!and!current!state! • Issues!confronted!by!this!QA!team! • RealXlife!examples!of!how!this!team!improved! • Success!!!World!class!QA! • Spreading!world!class!quality!to!other!areas!
- 7. So)how)do)we)get)there?) ! • No!team!ever!starts!out!being!world!class! • We!grow!into!it:! – Through!a!process!of!conPnuous!reflecPon!and! improvement!! – Applying!the!right!methodologies!for!the!right! problems!at!the!right!Pme! – Being!adaptable,!flexible,!and!creaPve! • We!demand!it!of!others! – Pushing!quality!upstream!
- 9. An)Imaginary)QA)Team) • Feature!TesPng,!mostly!manual! • Regression!tesPng! • Some!automaPon! • Load!and!Performance!tesPng! • ProducPon!Smoke!TesPng!or!Sanity!TesPng! • High!desire!to!get!be`er!and!restore!work/life! balance!
- 10. State)Of)Our)Imaginary)QA)Team) ! • Deals!with!late!code!deliveries!to!QA! • Poorly!defined!or!incomplete!requirements! • Compacted!or!aggressive!schedules! • Bugs!not!gebng!discovered!unPl!late! • High!producPon!bug!count! • Poor!performance!in!producPon! • Many!code!changes!during!regression! • Unwanted!high!visibility!to!leadership!
- 11. Case)Study)Issue)–)New)Product)Adop-on) ! 1. High!number!of!producPon!defects! 2. TesPng!was!broad!enough!but!lacked!depth! 3. Performance!issues!leading!to!Sev!1!incidents! 4. Business!pushing!for!rapid!turnaround!on!releases!! 5. Customers!were!unhappy!and!complaining!! ! Let’s!see!how!this!team!worked!through! these!issues!to!reach!a!world!class!level!of! quality! !
- 12. Issue)1:)High)Number)of)Produc-on)Defects) ) • Some!areas!of!the!code!worked!well,!some!areas! were!highly!problemaPc! • Many!bugs!were!things!that!QA!should!have!caught! • Through!some!introspecPon!we!decided!to!employ:!! – A!Focus!Methodology!to!reXprioriPze!tesPng! – Started!AdXHoc/Gorilla!TesPng! – White!Box!tesPng!
- 13. Focused)Tes-ng)Method) ! • ProducPon!bugs!can!be!analyzed!for! funcPonal!or!component!level!frequencies! • These!are!the!areas!that!addiPonal!tesPng!or! more!resources!should!be!applied! • The!focus!will!change!over!Pme!to!reXvisiPng! the!analysis!is!necessary! • This!can!also!be!useful!to!development!as!it! indicates!a!need!for!more!unit!tesPng!
- 14. Ad)Hoc)/)Exploratory)Method)! • Requirements!tesPng!takes!you!down!the!basic! paths! – !Mostly!sunny!day! – !Some!rainy!day!if!the!the!requirements!are!good! • Ad!Hoc,!or!Gorilla!tesPng,!is!very!key!to!success!as!it! is!where!many!criPcal!bugs!can!be!found! • It!is!basically!tesPng!outside!of!the!requirements,! beaPng!on!the!system!in!any!way!you!can,!based!on! your!knowledge!of!the!system! • Should!also!involve!your!product!managers!or! requirements!authors!
- 15. White)Box)Method)! • Use!knowledge!and!understanding!of!internal! applicaPon!code!to!develop!test!cases! • Focuses!on!internal!constructs!at!the!unit! level!more!than!funcPonality! • Usually!involves!wriPng!code!itself!to!execute! APIs!or!service!calls! • When!combined!with!other!methodologies! can!be!very!effecPve!at!finding!bugs!!
- 16. Issue)2:)Tes-ng)Was)Broad)But)Not)Deep) ! • Due!to!aggressive!schedules!QA!Pmes!were! compacted! • Late!deliveries!of!code!into!into!QA!reduced!test! Pme! • High!number!of!possible!paths!meant!only!samples! were!tested,!not!all!possible!cases! • AEer!some!analysis!we!decided!to!employ:!! – An!IteraPve!Methodology! – A!Data!Driven!Approach!
- 17. Itera-ve)Method) ! • RelaPve!of!Agile!processes! • Used!when!development!must!deliver!code! to!QA!in!phases!for!one!release! • Each!phase!is!tested!with!previous!phase! regressed!by!automaPon! • MulPple!passes!through!highest!priority!test! cases!aEer!final!delivery! • Allows!for!tesPng!to!start!while!development! conPnues!
- 18. Data)Driven)Method) ! • Highly!Automated! • Uses!a!set!of!inputs!and!expected!outputs!in!a! dynamic!test!environment! • Can!be!a!spreadsheet!of!tuples! • Can!be!represented!in!XML!or!any!common! data!representaPon! • Allows!for!high!frequency!of!test!runs!and! complete!coverage!of!high!volume!test!cases!
- 19. Data)Input) Test_Case_Name DT_Flow Run Preview_Type Decision_Trees Signed_Member Topic_ID Site TC_CB_S01 S01 N Decision(Tree 103_Cancel_Bid US TC_CB_S02_A S02 N Decision(Tree 103_Cancel_Bid US TC_CB_S02_B1 S02 N Decision(Tree 103_Cancel_Bid US TC_CB_S02_B2 S02 N Decision(Tree 103_Cancel_Bid US TC_CB_S02_B3 S02 N Decision(Tree 103_Cancel_Bid US TC_CB_S02_C1 S02_5bidders N Decision(Tree 103_Cancel_Bid US TC_CB_S02_C2 S02_5bidders N Decision(Tree 103_Cancel_Bid US TC_CB_S02_C3 S02_5bidders N Decision(Tree 103_Cancel_Bid US TC_CB_S02_D1 S02_10bidders N Decision(Tree 103_Cancel_Bid US TC_CB_S02_D2 S02_10bidders N Decision(Tree 103_Cancel_Bid US TC_CB_S02_D3 S02_10bidders N Decision(Tree 103_Cancel_Bid US TC_CB_S02_D4 S02_10bidders N Decision(Tree 103_Cancel_Bid US TC_CB_S03_A S03 N Decision(Tree 103_Cancel_Bid US TC_CB_S03_B1 S03 N Decision(Tree 103_Cancel_Bid US TC_CB_S03_B2 S03 N Decision(Tree 103_Cancel_Bid US TC_CB_S03_B3 S03 N Decision(Tree 103_Cancel_Bid US TC_CB_S03_C1 S03_5bidders N Decision(Tree 103_Cancel_Bid US TC_CB_S03_C2 S03_5bidders N Decision(Tree 103_Cancel_Bid US TC_CB_S03_C3 S03_5bidders N Decision(Tree 103_Cancel_Bid US TC_CB_S03_D1 S03_10bidders N Decision(Tree 103_Cancel_Bid US
- 20. Issue)3:)Performance)issues)leading)to)many) Sev)1)incidents)) ! • Long!response!Pmes!prevented!users!from!doing! their!job! • Volume!issues,!certainly!around!holidays! • During!complex!releases!performance!impacts! difficult!to!judge! • AEer!some!analysis!we!decided!to!leverage!load!and! performance!tesPng!to!employ:!! – An!Risk!Driven!Methodology! – A!Black!Box!Approach!to!highest!use!cases!
- 21. Risk)Based)Method) ! • CollaboraPon!with!developers!and! product!owners!! • IdenPfy!areas!of!change!in!a!release! • PrioriPze!tesPng!based!on!idenPfied!risk! • Test!prioriPes!can!change!between! releases! • New!test!cases!may!be!needed! • High!value!methodology!and!an!area!of! focus!for!each!release!
- 22. Black)Box)Method) ! • FuncPonal!tesPng!driven!by!understanding!of! new!funcPonality!or!features! • Commonly!used!when!tesPng!GUIs!or!front!ends! • Can!also!be!applied!to!endXtoXend!tesPng!when! mulPple!systems!are!involved! • PrioriPze!around!top!5X10!use!cases.!Most!end! users!will!use!only!a!small!number!of!features! • Also!good!for!gebng!product!owners!and!endX users!involved.!They!can!follow!a!test!plan!!
- 23. Issue)4:)Business)pushing)for)rapid) turnaround)on)releases)) ! • With!high!volume!of!bugs!business!wanted! fixes!quickly! • New!product,!new!features!needed!quickly! • Difficult!to!turn!around!weekly!releases! • Combined!several!methodologies!to!form!a! Rapid!Release!process,!using:!! – A!Risk!Driven!Methodology! – An!IteraPve!approach!
- 24. Rapid)Releases) ! • Employing!the!iteraPve!approach!allowed!for! small,!rapid!releases!that!were!easier!to!test! • Bug!fixes!rolled!out!weekly,!restricted!features! • Some!small!features!can!be!added! • Risk!analysis!is!key!X!just!the!change!areas!are! manually!regressed! • Frequent,!rapid!automaPon!regression!runs!
- 25. Issue)5:)Unhappy,)complaining)Customers)) ! • Customers!felt!they!had!no!input!or!input!was! ignored! • High!defect!count!and!performance!issues! affected!morale! • Customers!unaware!of!upcoming!changes! • AEer!meePng!with!customers!and!analyzing! our!process,!decided!to!employ:!! – A!Customer!Driven!Methodology!
- 26. Customer)Driven)Method) ! ! • End!users!are!involved!in!the!tesPng! • Run!proof!of!concepts!and!pilots! • Daily!feedback!that!is!reviewed!and!turned!into! defect!reports!or!new!enhancements! • Rapid!feedback!back!to!the!end!users! • Leverage!end!users!as!agents!of!change! • Controlled!deployments! – Phase!in!volume!to!build!confidence!in!quality!
- 27. Methodologies)Summary) • More!methodologies!are!available!to!us!now! than!ever!before:! • Ad!Hoc!/!Exploratory! • Focused! • Data!Driven! • Risk!Based! • IteraPve! • Customer!Driven! • White!Box! • Black!Box!
- 28. Mature)Methodologies) • As!your!processes!become!more!mature! addiPonal!methodologies!can!be!brought!in:! • Agile! • Zero!Defect!! • ConPnuous!IntegraPon!
- 29. Agile)Tes-ng) ! • Can!be!difficult!for!large!projects!with!many! different!teams! • Very!popular!right!now!but!not!the!answer!for! all!projects! • Be!careful!not!to!move!to!Agile!just!to!do!so! or!because!it!is!all!the!rage!right!now! • SoEware!that!your!business!relies!on!may!not! be!a!good!fit!for!a!short!dev/test!cycle!
- 30. Zero)Defect) • Goal!is!not!to!have!no!defects!in!a!release! • Emphasizes!high!quality!in!all!tasks!performed! by!each!person! • Involves!strict!adherence!to!requirements,! requirement!traceability! • Fix!bugs!before!any!new!code!is!developed! • It!is!a!mindset!more!than!a!process!
- 31. Con-nuous)Integra-on) • OnXdemand!code!merges!and!builds!triggered! by!code!commits! • Automated!tesPng!integrated!in!with!Jenkins! • Helps!to!prevent!integraPon!or!merge!issues! • Feedback!to!the!developer!on!any!defects! introduced!is!frequent!and!fast! • Be`er!code!deliveries!into!QA! • Requires!some!infrastructure!to!make!it!work!
- 32. Itera-ve)approach)to)improvement) ! ! • Key!is!to!conPnuously!improve!processes! through!open!and!honest!communicaPon! • PostXmortems! • Reviews!of!process!and!strategies! • AdopPng!new!methodologies!to!fill!in!the! gaps! • Apply!learning's!to!other!trouble!areas!