DevOps / Agile Tools Seminar 2013

© All rights reserved
Want to use slides from
this presentation?
Ask for permission:
ethanram@gmail.com

Agile Tools Seminar
- Technical Introduction Ethan Ram / 16 Oct. 2013
v1.2

Agenda: Agile Tools Seminar
• Session presents the automation model
developed in Videobet, the tools used in the
R&D, QA and operations:
–
–
–
–
–
–
–

Issue mgmt.: JIRA/Greenhopper
Build system and repository: Maven & Nexus
Build server: QuickBuild
Code quality: Sonar
Continuous Integration: Selenium Grid
Crash dump analysis: Socorro
Database versioning: Flyway DB

• Open discussion
• Refreshments & tour in Videobet office
Videobet, Paldiski Mnt. 29, Tallinn
iPad Mini – Draw on Exit
10/22/13


2

Videobet
• A Playtech division
• Innovative Gaming
Platform Provider
– Established 9 years ago
– 150 employees in Tallinn
and Kiev
– Operate in 9 countries
– 60% of UK market

• R&D
– Front-end teams (C++)
– Server teams (Java)
– Game development teams
10/22/13


3

Ethan Ram - Host
• Chief architect of Videobet
• Leading Videobet’s R&D and OPs
teams Agile Injection effort
• As VP R&D of GameGround
transformed the R&D to run in
Kanban in a Continuous
Deployment environment
• Hacker in soul and a Windows
Internals hobbyist
Find me here:
•ethnram@gmail.com
•LinkedIn
•The Null Terminator blog
10/22/13


4

 “We forgot to implement this feature…”
 “Didn’t you get that urgent build request email?!”
 “But there was a much more urgent issue to work on”
 “We lost a day in QA because the build was total crap”
 “It will take me a half a day to switch to that branch”

10/22/13


5

Automate it!

Streamlined
Operations







“We forgot to implement this feature…”
“Didn’t you get that urgent build request email?!”
“But there was a much more urgent issue to work on”
“We lost a day in QA because the build was total crap”
“It will take me a half a day to switch to that branch”

10/22/13










Transparency in all operations
Prioritize often
Enable teams collaboration
Simplify dev. environment
Reduce QA cost
Improve product quality
Go Agile; go green!
6

10/22/13


7

Target: Streamline Operations
Daily Scrum meeting around team Scrum board

Filling a bug using 7” tablet 
10/22/13


8

Agile Issue Tracking and Mgmt: JIRA
Tips
•JIRA Agile is good!
•Latest versions are fast and
intuitive
– No more than 1-2 hours a
week to manage it all

•A few plugins used for
missing functionality
•Keep all R&D in one JIRA
project
– Team backlog based on a
team label

•If you cannot model your
process into JIRA better ask:
“What do I need to change in
my process”
•Missing a good
resource/project timeline tool

10/22/13


9

JIRA Agile – Scrum Board

10/22/13


10

Scrum Board – Sprint Planning

xxx
aBC

aBsddlfhasldjhlshC

10/22/13


11

Scrum Charts: Burn-Down, Velocity,
Sprint Report

10/22/13


12

Kanban: Multi-Project Epics Mgmt.
WIP
ALERT

10/22/13


13

Kanban: Operations Teams

10/22/13


14

Continuous Integration Concept

10/22/13


15

Continuous Integration - Maven
• Mavenize it all
– Product: Java, C++, scripts, game resource
compilations etc.
– Embedded Operating System images
– Customer configurations and packages

• Nexus artifact repository
– Build output is here! Snapshots, releases
– Installer downloads directly from Nexus repo.
– New artifacts pushed to 2nd Nexus in remote site

• Build promotions
– Snapshot builds avail to Scrum team immediately
(Auto-deleted after a week)
– Promote builds to QA and customer (“gold”)
10/22/13


16

Continuous Integration – Build Server
•
•
•
•
•

Build on every commit – fail build on errors
Run unit testing – fail build on assertions
Run integration tests – alert testers
Code quality metrics – produced weekly
Distributed build agents – some in remote
sites
• Daily snapshot installed and run in QA
– Nightly scripts run on 60+ terminals

• Integration testing – run every hour
– Deploy and upgrade servers and database
– Run Selenium script on management console
– Run clients simulators
10/22/13


17

CI Server - QuickBuild
Build and test queue

Latest builds
promoted to QA

Alert when Continuous
Integration fails
10/22/13


18

Code Quality Metrics - Sonar

•
•
•
•
•

Percent of code covered by unit tests, per module
Common developer mistakes: leaks, uninitialized vars, unsafe error handling etc.
Code complexity measures based on static code analysis
Drill-down to line of code
Supports continuous improvement by developers: Java and C++
10/22/13


19

Integration Testing

• This is unsustainable!
• Goal: break this is vicious cycle!
10/22/13


20

Integration Testing – The basics
• Initial goal: main data flows should
remain working at all times to allow
QA to run full regression test suite
• Long term goal: replace most
regression testing
• Tests are run hourly on active
branches - snapshot build
• Deployment on a set of virtual
machines

10/22/13


“Keeping product
stable is more
important than
developing new
features”

21

Integration Testing – DEMO

• Using Selenium scripts to test admin console and check
that a test-setup of servers and clients is working
10/22/13


22

Here’s a New Bug CI Found Today!
 Email: CI Alerts on failed test
 Tester validates it’s a bug
Bug was resolved 2 hrs after it was introduced
CI is back to “green”

10/22/13


Integration Testing – Solution
• Infrastructure
–
–
–
–
–

Virtual Machines deployment scripts
DB deployment scripts
Server install
Clients install
Client simulators (initially created used
for load-testing sessions)
– Various libs allowing triggering of
hardware events and reading screen
output – with activated over http

• Tests
– Selenium grid – for web-testing
• nUnit tests running on build server
10/22/13


24

Integration Testing – Tips
• Find the natural topic owners: the Scrum testers
– Lots a scripting is required
– They are busy – but this can be their baby

• Plan ahead but start small
– Get one test running first

• The hard part is to set the infra. You have to
combine knowledge from
–
–
–
–
–

Devs
Testers
Build
Deployment
Infra-teams

• Education is a must

– This is a radical change that ppl do not understand

• Keep tests short
• Celebrate the first bugs found 
10/22/13


25

Crashes In a Large-Scale Install-base
• A major problem on all platforms
• Intermittent crash is very hard to
find in QA
– Hard to reproduce
– May-be overlooked by tester
– Happens only when combined with
“other” software and hardware

• Crash statistics gives clarity
– Scale of problems: focus for
urgency
– Direct access to dump for devs
10/22/13


26

Crash Stats Server - Socorro
Select Product
or Content and
version

Filter by time
and platform

Sort by first time
this crash was
reported

Trending!
Crash dumps are
grouped by crash location
in code;
Click to see graph

• Open source server by Mozilla provides large scale dump analysis
• All licensee’s terminals upload dumps of platform and games (“content”)
• Dumps redirected to a single server as Videobet data center
10/22/13


27

Crash Drill-Down- Socorro
Product build
number and
client ID

Stack trace
with symbols
directly from
build server

Create
bug

Click to go
directly to
source code

• 105 bugs opened in QA night-runs (12/’12 – 8/’13)
10/22/13


28

Database Versioning
Database

Code

•
•
•
•

• Branch
• Build version

Schema
Config data
Operational data / OLTP
Reporting data / OLAP

We need to find an answer to these:
•What state is the database in on this machine?

– Has this script already been applied or not?
– Has the quick fix in production been applied in test afterwards?

•Can this DB correspond to my code version?
•How do I roll-back this DB to previous release version?
•How do you set up a new database instance?
10/22/13


29

Database Versioning

10/22/13


30

Flyway DB: First Migration

10/22/13


31

Flyway DB: Second Migration

10/22/13


32

Flyway DB is Agile!
• Migrations kept with source code
– Applying migrations added between v1 to v2 is
easy
– Roll-back of migrations supported built-in

• Merging team-branch migrations is like (even
simpler than) merging code
• Each dev can have its own DB and manage
it’s data in a simple way
– Integrative with Maven/Ant
– Command-line or JAVA API

• Data for unit-testing can be added and
removed on the fly
• Excellent for Continues Deployment
10/22/13


33

Goal: Continuous Deployment
At SlideShare, instead of big launches, we
We have been doing continuous deployment code all the time... it reduces risk.
release
at Plex since our inception in 1995. We we are putting out small changes at a
Since
formalized the process with our own “deploy” breaks we know what it is and can
time, if it
button in 2003, and hit the 25k deployment
roll back.
mark less than two years thereafter. Flickr "10+ Deploys Per Day"
We
Rashmi
Steve Conover
currently average between 50-100
At bixbe.com, we auto-deploy the latest
deployments per day as soon as anything is
checked in code to our massive online
ERP system.
installed, so no safety net at all. Using this
One of the exciting things, from a
Jerry Foster
method, we’re doing 10-30 deploys per day.One of the exciting things, from a
development perspective, about Digg is
development
It’s a totally productive way to move quickly.
WordPress.com - we’ve averaged about 16 perspective, about Digg is
continuous deployment - when developers fix
We still maintain the conceptevery continuous deployment - when developers fix
product releases a day, of ‘beta’ for the last
day
aabug or add aanew feature, there's no need
features… four and bits ofyears… Everyone in our
optional a half code that canbug or add new feature, there's no need
be
to wait for aascheduled release. Instead, the
tobutton that
turned on or off by user account. This letswait for scheduled release. Instead, the
company has access to a deploy us
change can go live right away. This is great change
roll out new capabilities to checked in code to about live right away. This is great releases the latest limited exposure can go
the turnaround time for a change drops
the
and avoid production the corein ourflow. tier in less
400 disrupting servers user web turnaround time for a change drops
dramatically.
dramatically.
than 30 mtrifiro (across 3 data centers).
seconds
Andrew Bayer
Andrew Bayer
Toni Schneider
10/22/13


34

Continues Deployment
“Let’s Leave Something
for next time…”

10/22/13


35

Q&A
Open Discussion

10/22/13


DevOps / Agile Tools Seminar 2013

More Related Content

What's hot (20)

Viewers also liked (13)

Similar to DevOps / Agile Tools Seminar 2013 (20)

Recently uploaded (20)

DevOps / Agile Tools Seminar 2013

Editor's Notes