Taming iOS Testing at Square -- JUC West 2015
1 like1,210 views
The document discusses strategies and experiences related to improving iOS testing and continuous integration at Square. It highlights problems faced with Jenkins, the setup of build environments, and the importance of code signing and build stability. Michael Tauraso shares insights from his role as a Senior Release Engineer emphasizing automation, management, and maintaining high standards for build performance.
Taming iOS Testing at Square -- JUC West 2015
- 1. Taming iOS Testing at Square Santa Clara, CA, September 2-3 1 Michael Tauraso, Sr. Release Engineer
- 2. Footer #jenkinsconf What do I do? CI + Release automation for mobile app developers I don’t write the tests I don’t know Objective-C or Swift I’m from a Linux + Devops background 2
- 3. Footer #jenkinsconf What does Square do? Square has 4 iOS apps Dozens of iOS developers Several internal iOS apps 3
- 4. Footer #jenkinsconf What’s different about iOS? • Mobile • OSX and Xcode • Velocity of ecosystem • Closed ecosystem 4
- 5. Footer #jenkinsconf What’s different about iOS? 5 October November December January February March April May June July August September 6.1 6.2 6.3 6.4 7.0?
- 6. Footer #jenkinsconf What’s different about iOS? • Mobile • OSX and Xcode • Velocity of ecosystem • Closed ecosystem 6
- 7. Footer #jenkinsconf iOS CI @ Square 1 year ago • 30 minute builds, longer queue times • Manual upgrades, no scale plans • Mysterious build failures • Jenkins difficult to manage • Lots of developer support + firefighting 7
- 8. Footer #jenkinsconf How do you make it better? • Assume change will happen • Know exactly what’s changing • Control all the variables • Deliver repeatable builds • Keep up with Apple 8
- 9. Footer #jenkinsconf What are we going to talk about? • How Square uses Jenkins • Configuring OSX machines for builds • Code signing on iOS • Finding bugs in build systems 9
- 10. Footer #jenkinsconf What are we going to talk about? • How Square uses Jenkins • Configuring OSX machines for builds • Code signing on iOS • Finding bugs in build systems 10
- 11. Footer #jenkinsconf Mobile Jenkins @Square 11 Git (Stash) Stashkins Jenkins iOS-builder Checkout Cocoapods Xcodebuild iOS Simulator
- 12. Footer #jenkinsconf Stashkins • Uses disabled jobs as templates • Substitutes values in Jenkins job XML • Runs off a config file in the app’s repo • Triggered on git-push 12
- 13. Footer #jenkinsconf Jenkins jobs • Check out ios-builder on a build slave • Run ios-builder • Display logs, results, and artifacts 13
- 14. Footer #jenkinsconf Jenkins Plugins • Promoted builds plugin • Send packaged .ipa files to OTA site • Warnings parser plugin • JUnit test results plugin • Swarm plugin 14
- 15. Footer #jenkinsconf Release Code • Pattern: Factor release code to another repo • Tweak things without touching Jenkins • Devs don’t need/want Jenkins write access • Easier management • Devs edit configs on their branch • CI + Code Review release code! 15
- 18. Footer #jenkinsconf Xcodebuild Needs • Project/Workspace • Scheme • Configuration • Destination • Action • build, test, and archive 18
- 19. Footer #jenkinsconf Useful Xcodebuild • Archive is not build • Uses more exact header search paths • SKIP_INSTALL Project setting • Use xcodebuild -exportArchive for ipa files • Flag reference: • git.io/vGDOd 19
- 20. Footer #jenkinsconf Xcodebuild Friends • Loose coupling • Frontends other than ios-builder • facebook/xctool • fastlane/gym • Log output • supermarin/xcpretty 20
- 21. Footer #jenkinsconf What are we going to talk about? • How Square uses Jenkins • Configuring OSX machines for builds • Code signing on iOS • Finding bugs in build systems 21
- 22. Footer #jenkinsconf Build slaves • Virtualization :( • Hardware • Dual core Mac-Minis • 16 GB RAM • 256 GB SSD • $1200 each 22
- 23. Footer #jenkinsconf Build slaves - DeployStudio • Network boot • Base image with OS X 10.10 • Installs Xcode 23
- 24. Footer #jenkinsconf Build slaves - Ansible • Basic config management • Machines ansible themselves • Allows easy config changes • Multiple Xcode Versions • Swarm plugin • Jenkins remoting version 24
- 25. Footer #jenkinsconf Build slaves - OSX • Match versions with Xcode • Never sleep (caffeinate command) • Turn off: • Software update • spotlight • bonjour • ssh w/password + VNC 25
- 26. Footer #jenkinsconf Build slaves - Xcode • Always have a GUI context! • Automatic login • swarm as a LaunchAgent • TCC Database • git.io/osx-tcc • DevToolsSecurity 26
- 27. Footer #jenkinsconf Build slaves - Xcode Packaging • Xcode likes to install things on first-startup • iOS SDKs installed via Xcode • Command line tools are another package • We record installs with Composer • (JAMF software) • Side-by-side xcode versions with jenkins labels + xcode-select 27
- 28. Footer #jenkinsconf Build slaves - More detail • https://corner.squareup.com/2015/07/ios-build- infrastructure.html 28
- 29. Footer #jenkinsconf What are we going to talk about? • How Square uses Jenkins • Configuring OSX machines for builds • Code signing on iOS • Finding bugs in build systems 29
- 30. Footer #jenkinsconf Code signing • Its confusing • The error messages are unclear • Apple changes things • Apple Watch! • Swift! • Your app needs it to run 30
- 31. Footer #jenkinsconf Code signing • Provisioning Profiles • Signing Identity 31
- 32. Footer #jenkinsconf Code signing - Mobileprovision • Message from Apple to iPhone • These devices • Can run this app (app ID) • With these permissions (entitlements) • So long as ( ) signed the code 32
- 33. Footer #jenkinsconf Code signing - Mobileprovision • is a plist in CMS format • Signing identities are just x509 certs • emonti/pliney • Parse + check for developer errors 33
- 34. Footer #jenkinsconf Code signing - Xcode • Xcode “helps” • Restrict Mobileprovision directory in CI • ~/Library/MobileDevice/Provisioning Profiles • Apple Watch apps • Multiple mobileprovision files • All on the same signing identity 34
- 35. Footer #jenkinsconf Code signing - HSM • Yubikey-NEO • Acts as an HSM • Signing identity available through keychain • Key never touches a hard-drive 35
- 36. Footer #jenkinsconf Code signing - Yubikeys • Can create on-device or import existing • Needs OpenSC (+reboot) • OpenSC/OpenSC • Needs yubikey-piv-tool • https://developers.yubico.com/yubico-piv-tool/Releases/ • Tutorial • https://developers.yubico.com/yubico-piv-tool/ OS_X_code_signing.html 36
- 37. Footer #jenkinsconf What are we going to talk about? • How Square uses Jenkins • Configuring OSX machines for builds • Code signing on iOS • Finding bugs in build systems 37
- 38. Footer #jenkinsconf Build Stability • Dozens of dev make 100s of builds • 1 in 100 failures happen every day • Builds that work sometimes are frustrating • Green builds usually result in code merging • “Sometimes” errors accumulate 38
- 39. Footer #jenkinsconf Build Stability • Run passing builds at night! • Use our idle infrastructure • How green are they really? 39
- 40. Footer #jenkinsconf Build Stability — Statistics • Builds are very unfair coins • Binomial distribution • Wilson score for confidence interval 40
- 42. Footer #jenkinsconf Build Stability — Statistics • Count individual failure types • Allocate investigative resources • Keep an SLA (99.5%) 42
- 43. Footer #jenkinsconf Build Stability 43 2 failures 300 builds 0.66% 6 failures 900 builds 0.66%
- 44. Footer #jenkinsconf Build Stability — Statistics • Found Xcode bugs • Reduced failure rate to 0.5% • More detail: • https://corner.squareup.com/2015/06/build-stability.html 44
- 45. Footer #jenkinsconf Where are we now? • 3x as many tests, lower build time • 99.5% SLA. We file bugs for bad tests • Many developer requests are self-serve • Growing cluster with developer hiring • Supporting latest Apple hardware 45
- 46. #jenkinsconf Footer Thanks to our Sponsors! 46
- 47. Footer #jenkinsconf Questions? Michael Tauraso mtauraso@{gmail,squareup}.com @mtauraso mtauraso 47
- 48. #jenkinsconf Footer Thanks to our Sponsors! 48
- 49. Taming iOS Testing Santa Clara, CA, September 2-3 49