DockerCon EU 2015: It's in the game: the path to micro-services at Electronic Arts with Docker
Download as PPTX, PDF11 likes7,638 views
This document discusses Electronic Arts' use of Docker containers to improve the scalability and deployment of their mobile game server infrastructure. It describes how EA was facing challenges with manual and complex deployment processes. With Docker, EA was able to break their application into microservices, package them as containers, and use tools like Jenkins and Fleet for automated builds, deployment, and orchestration. This resulted in faster and more reliable deployments while improving scalability to meet the demands of live service games.
DockerCon EU 2015: It's in the game: the path to micro-services at Electronic Arts with Docker
- 1. It's in the game: The path to micro-services at Electronic Arts with Docker Andrew Hately IBM CTO Cloud Architecture Scott Porter EA Cloudcell Server Lead Melbourne Firemonkeys
- 2. GAMING MARKET GROWING ACROSS ALL PLATFORMS AND REGIONS Mobile PC/F2P Console NORTH AMERICA LATAM EUROPE ASIA/ROW GROWTH YoY 12% REVENUES 2015est. 21B GROWTH YoY 19% REVENUES 2015est. 4B GROWTH YoY 7% REVENUES 2015est. 17B GROWTH YoY 9% REVENUES 2015est. 40B 17% 46% 37%41% 16% 43% 195M 182M 553M 817M 33% 24% 43% 51% 44% 5% 2015 est. Revenues Consist of Software Gaming Market
- 3. MOBILE FUELING INDUSTRY GROWTH $3.3 $4.3 $6.1 $9.9 $17.0 $27.8 $34.0 2009 2010 2011 2012 2013 2014 2015 Mobile games industry has grown 10x in less than a decade Mobile Gaming Revenue, WW $B
- 4. Mobile Game Server architecture/requirements • Mobile Gaming Back-end is a very typical Web application • Frequent updates to game content - infrequent updates to game engine • Daily peak users 150,000 to 250,000 simultaneous users and 1-2 million daily • Need to scale-up/down around release and content pack releases • Stateless front-ends with memory caching and both SQL and no-SQL DB • 50TB per day of data - Increasing use of analytics on user experience, recorded game telemetry, replays, etc. Firewall Load Balancer NGNIX Game Server REDIS Memcache MySQL cluster PHP PHP PHP PHP PHP Object Storage
- 5. Demo – Docker for Gaming
- 7. • Our current Ops complexity • Partnership with IBM • Why we believe Docker can help • How we integrated Docker
- 8. Scott Porter Cloudcell Server Lead EA Melbourne Firemonkeys scott@ea.com @sl_porter
- 9. EA MELBOURNE FIREMONKEYS • Moble studio based in Melbourne, Australia • Formed in 2012 from two EA Mobile studios, Iron Monkey and Firemint • Focused on developing the highest quality mobile titles
- 10. • Firemonkeys in house Client/Server Tech Stack • Mobile focused connected features • Originally developed by Firemint for Flight Control
- 11. Integrated into all Firemonkeys live titles
- 12. • Live Service Games • Bi monthly client releases • No downtime What do we do?
- 13. • 13.5k Peak Requests/s • ~900M Requests/day • ~430k Peak Concurrent Players • 5M – 8M Daily Active Players • > 100 TB Binary Data • 170 Hosts (60 Live Hosts) • Team of 7 Current Scale
- 14. • Baremetal Hosts • Nginx, PHP, Memcache, MySQL, Redis • Batched client requests, short-lived, stateless • Requests can touch any datastore Architecture
- 15. Game Logic Data StorageLoad Balancing Architecture External Services StatsD / Diamond / Graphite / Graphana Object Storage Datadog
- 16. • Automated Development / Manual Live process • In house database migration and code deployment system • Bash & rsync • Rollbacks are handled via code duplication Current Deployment Methodology
- 17. • Asynchronous processes (workers) manually configured ( services, cron jobs ) • Brittle. • SaltStack is awesome! • But still slow, complex, no HA Current Orchestration Methodology
- 18. • Moved binary data to SoftLayer Object Storage (Open Stack Swift) • Currently provision for Peak • Shard live environments per game title • Interested in transitioning to microservices Current Scalability
- 19. • Improved deployments • Growing ecosystem • Scalable • Easy transition, run side by side with existing stack Why Docker?
- 20. • Focused on the Game Logic Layer • Decomposition • Nginx • Admin • Application • Workers Packaging Cloudcell
- 21. • Generic Containers • Run multiple containers, not processes • Tradeoff considerations Packaging Cloudcell
- 22. Code Building Images Jenkins Docker Registry Base Images nginx php-fpm php-worker Built Images php-fpm-nfs-0001 php-fpm-nfs-0002 php-worker-a-1 nfs-0002.tar worker-a-1.tar nfs-0001.tar Dev Env Jenkins Application Code (SVN)
- 23. • Fleet • Unit files per service • Nginx • Admin • Application • Each worker type Orchestration
- 24. Load Testing Host Baremetal nginx Application Worker A Load Generation LTG LTG LTG LTG LTG LTG Host Docker Docker nginx php-fpm-nfs-0001 php-worker-a-1 etcd Fleet
- 25. • Overall 5 – 10% overhead running Load Test with Docker Load Testing Results PLACEHOLDER NEED A REAL GRAPH HERE!
- 26. • Docker Private Registry • Jenkins • Studio Jenkins ships code to build machine • Build machine builds and commits images to Registry • Generates Fleet configuration from templates • Executes Fleet deployments Deployments
- 27. • Config Editor • Bridges Cloudcell Application deployments to environment state • Provides a simple UI for building, deployments, and state via Jenkins and Fleet Deployments
- 28. Deployments Docker Build Machine Host Docker etcd Jenkins Docker Registry Config Editor nginx php-fpm-nfs-0001 php-worker-a-1 Fleet
- 29. Docker enhanced architecture on SoftLayer CentOS 7 host Discovery/Config Fleet managed Docker services nginx- confd.service NGINX-confd container Orchestration Docker php- fpm@1.service ea/php- fpm:latest 1 php- fpm@2.service ea/php- fpm:latest 2 php-fpm-discovery @2.service php-worker- A@2.service ea/php- worker:latest 2 php-worker- B@1.service ea/php- worker:latest 1 php-worker-A-discovery @2.service php-worker-B-discovery @1.service php-worker- A@1.service ea/php- worker:latest 1 php-worker-A-discovery @1.service php-fpm-discovery @1.service Redis Externaldataservices MySQLMemcached LoadBalancer
- 30. Configuration, orchestration and deployment host servi ce servi ce servi ce servi ce host servi ce servi ce servi ce servi ce host servi ce servi ce servi ce servi ce host servi ce servi ce servi ce servi ce Configure topology, maintain state. Build images with code, deploy to or scale target environments. Run containers in staging, load test, and production. Capture metrics.
- 31. • Centralised Logging • Monitoring - processes inside containers Future Work
- 32. — Scott Porter “Using IBM and Docker container solutions, we have been able to build a gaming platform that is massively scalable, highly performant and meets our deployment velocity needs.” 33
- 33. Thank you! @cloudtroll hately@us.ibm.com Andrew Hately Scott Porter @sl_porter sporter@ea.com