Riot Games 글로벌 게임 운영을 위한 Docker 및 Amazon ECS 활용사례 - AWS Summit Seoul 2017
6 likes1,297 views
The document discusses how Riot Games standardized their application deployments using Amazon ECS. It describes how they broke their infrastructure into modular components defined through Terraform to provision resources like ECS clusters, services, load balancers and monitoring in a consistent and reproducible way. It also discusses lessons learned like breaking stacks apart but not overdoing it, being liberal with cluster provisioning, centralizing logs, and staying up to date with ECS and application releases.
Riot Games 글로벌 게임 운영을 위한 Docker 및 Amazon ECS 활용사례 - AWS Summit Seoul 2017
- 1. © 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Adam Rozumek, Systems Engineer RIOT GAMES Standardizing Application Deployments Using Amazon ECS
- 2. RIOT GAMES Standardizing Application Deployments Using Amazon ECS
- 4. AMAZON ECS & VIDEO GAMES WHAT TO EXPECT ECS CONSOLIDATION WINS & LESSONS Adapting existing deployments & infrastructure INFRASTRUCTURE ENCAPSULATION Object Oriented Development Operations MULTI-GAME MODULAR SERVICE DESIGN A different kind of scaling 1 2 3
- 5. AMAZON ECS & VIDEO GAMES WHAT TO EXPECT ECS CONSOLIDATION WINS & LESSONS Adapting existing deployments & infrastructure INFRASTRUCTURE ENCAPSULATION Object Oriented Development Operations MULTI-GAME MODULAR SERVICE DESIGN A different kind of scaling 1 2 3
- 6. AMAZON ECS & VIDEO GAMES WHAT TO EXPECT ECS CONSOLIDATION WINS & LESSONS Adapting existing deployments & infrastructure INFRASTRUCTURE ENCAPSULATION Object Oriented Development Operations MULTI-GAME MODULAR SERVICE DESIGN A different kind of scaling 1 2 3
- 7. AMAZON ECS & VIDEO GAMES WHAT TO EXPECT ECS CONSOLIDATION WINS & LESSONS Adapting existing deployments & infrastructure INFRASTRUCTURE ENCAPSULATION Object Oriented Development Operations MULTI-GAME MODULAR SERVICE DESIGN A different kind of scaling 1 2 3
- 10. 7.5MILLION PEAK CONCURRENT PLAYERS 100MILLION MONTHLY ACTIVE PLAYERS MORE THAN 27MILLION DAILY ACTIVE PLAYERS MORE THAN MORE THAN 2016 LEAGUE OF LEGENDS STATS
- 11. DATA PRODUCTS & SERVICES OUR MISSION Empower teams at Riot to make timely, data-informed products by maintaining a scalable and reliable data platform AWS re:Invent 2015 | (GAM303) Riot Games: Migrating Mountains of Big Data to AWS Sean Maloney
- 12. PROBLEM
- 13. PROBLEM SCALING TOTAL OWNERSHIP ON AWS Total ownership We want to empower developers to: • Provision their own infrastructure • Execute their own deployments • Monitor their own metrics
- 14. PROBLEM SCALING TOTAL OWNERSHIP ON AWS Resource attribution • Who owns these EBS volumes? • What applications depend on these security groups? • Can these AMIs be deleted?
- 15. PROBLEM SCALING TOTAL OWNERSHIP ON AWS Security
- 16. PROBLEM SCALING TOTAL OWNERSHIP ON AWS Security • Auditing is important, but it’s reactive • Operational time sink Security Monkey AWS Trusted Advisor
- 17. PROBLEM SCALING TOTAL OWNERSHIP ON AWS Amazon EC2 Container Service
- 18. CONTAINERS
- 19. CONTAINERS STANDARDIZED APPLICATION UNITS… ON AWS • Dockerfiles capture application dependencies
- 20. CONTAINERS STANDARDIZED APPLICATION UNITS… ON AWS • Dockerfiles capture application dependencies • Common use cases have great community support
- 21. CONTAINERS STANDARDIZED APPLICATION UNITS… ON AWS • Dockerfiles capture application dependencies • Common use cases have great community support • Profit from our own engineering community
- 22. CONTAINERS STANDARDIZED APPLICATION UNITS… ON AWS
- 23. CONTAINERS STANDARDIZED APPLICATION UNITS… ON AWS Embrace the abstraction • Plan for failure at all levels • Avoid manual intervention whenever possible • It’s ephemeral all the way down
- 24. CONTAINERS STANDARDIZED APPLICATION UNITS… ON AWS Scheduling is hard We need to: • Quickly and fairly run tasks • Prevent resource conflicts • Provide reasonable fault tolerance
- 25. CONTAINERS STANDARDIZED APPLICATION UNITS… ON AWS Provisioning AWS hardware • Enable total ownership of the AWS resources backing our containers • Avoid the security, resource attribution, and convention degradation pitfalls
- 26. AMAZON ECS
- 27. AMAZON EC2 CONTAINER SERVICE STANDARDIZED APPLICATION UNITS… ON ECS! • ECS AMI provides all necessary software • Designed with AWS integration in mind • Free!
- 28. AMAZON EC2 CONTAINER SERVICE KEY FEATURE TIMELINE Nov 2014 ECS ANNOUNCED Re:Invent 2014
- 29. AMAZON EC2 CONTAINER SERVICE KEY FEATURE TIMELINE Nov 2014 April 2015 Dec 2015 August 2016 ECS ANNOUNCED Re:Invent 2014 ECS GENERALLY AVAILABLE ECR & NEW REGIONS ECS becomes available in the final missing region we need (Frankfurt) for our globally deployed applications
- 30. AMAZON EC2 CONTAINER SERVICE KEY FEATURE TIMELINE Nov 2014 April 2015 Dec 2015 May 2016 July 2016 August 2016 ECS ANNOUNCED Re:Invent 2014 ECS GENERALLY AVAILABLE ECR & NEW REGIONS ECS becomes available in the final missing region we need (Frankfurt) for our globally deployed applications SERVICE SCALING Automatic task count scaling based on CloudWatch metrics introduced
- 31. AMAZON EC2 CONTAINER SERVICE KEY FEATURE TIMELINE Nov 2014 April 2015 Dec 2015 May 2016 July 2016 August 2016 ECS ANNOUNCED Re:Invent 2014 ECS GENERALLY AVAILABLE ECR & NEW REGIONS ECS becomes available in the final missing region we need (Frankfurt) for our globally deployed applications SERVICE SCALING Automatic task count scaling based on CloudWatch metrics introduced TASK SPECIFIC IAM ROLES Unlocked a lot of cluster sharing potential
- 32. AMAZON EC2 CONTAINER SERVICE KEY FEATURE TIMELINE Nov 2014 April 2015 Dec 2015 May 2016 July 2016 August 2016 ECS ANNOUNCED Re:Invent 2014 ECS GENERALLY AVAILABLE ECR & NEW REGIONS ECS becomes available in the final missing region we need (Frankfurt) for our globally deployed applications SERVICE SCALING Automatic task count scaling based on CloudWatch metrics introduced TASK SPECIFIC IAM ROLES Unlocked a lot of cluster sharing potential APPLICATION LOAD BALANCERS Several key improvements for ECS
- 33. BEYOND ECS INFRASTRUCTURE AS CODE • At scale, orchestrating infrastructure in a consistent, reproducible way is key
- 34. BEYOND ECS INFRASTRUCTURE AS CODE • At scale, orchestrating infrastructure in a consistent, reproducible way is key Total ownership
- 35. PROVISIONING
- 36. VPC NAT Gateway Route 53 Hosted Zone Route Tables VPN Gateway VPC Internet Gateway Application Subnets Tools Instances Instances Instances Availability Zone C Availability Zone B Availability Zone A PROVISIONING INFRASTRUCTURE AS OBJECT-ORIENTED CODE
- 37. CASE STUDIES
- 38. ECS CLUSTER TERRAFORM BUILDING BLOCKS VPC NAT Gateway Route 53 Hosted Zone Route Tables VPN Gateway VPC Internet Gateway Application Subnets Tools Instances Instances Instances Availability Zone C Availability Zone B Availability Zone A ECS Cluster Autoscaling Group Security Group Security Group Security Group Instance Instance Instance Instance Instance Instance Launch Configuration User Data IAM Role CloudWatch Alarms
- 39. ECS CLUSTER TERRAFORM BUILDING BLOCKS Module 1 Module 2 Module 3
- 40. ECS CLUSTER TERRAFORM BUILDING BLOCKS
- 41. MICROSERVICES WITHOUT SERVICE ENDPOINTS ECS Cluster Autoscaling Group Security Group Security Group Security Group Instance Instance Instance Instance Instance Instance Launch Configuration User Data IAM Role CloudWatch Alarm ECS Service Task Definition CloudWatch Alarms IAM Role
- 42. MICROSERVICES WITH SERVICE ENDPOINTS ECS Cluster ECS Service Task Definition CloudWatch Alarms IAM Role Application Load Balancer Monitoring CloudWatch Alarms SNS Topics Security Group Security Group Listeners Target Groups Route 53
- 43. PERSISTENT DATA LOSE ECS HOSTS WITHOUT LOSING DATA ECS Cluster ECS Service Application Load Balancer Monitoring CloudWatch Alarms SNS Topics Security Group Security Group Listeners Target Groups Route 53 Attachment Group EBS EBS EBS Elastic Network Interface Elastic IP
- 44. LESSONS LEARNED
- 45. LESSONS WHAT WORKED FOR US • Break apart your stacks ECS Cluster
- 46. LESSONS WHAT WORKED FOR US • Break apart your stacks, but don’t overdo it ECS Service ALB SNS Topics Route 53 IAM Role Task Definition CloudWatch Alarms
- 47. LESSONS WHAT WORKED FOR US • Be liberal with your cluster provisioning • Don’t risk resource contention in production • With good orchestration, additional clusters != additional operational overhead
- 48. LESSONS WHAT WORKED FOR US • Tag everything all of the time • Keep your tags organized in your Terraform templates • Have top level variables that get applied to every resource • Create a tag for every dimension that is useful to your business
- 49. LESSONS WHAT WORKED FOR US • Centralize your logs or Amazon CloudWatch Logs
- 50. LESSONS WHAT WORKED FOR US • Stay up to date with release blogs and application updates • ECS updates on the AWS blog
- 51. LESSONS WHAT WORKED FOR US • Capture your AMI generation process
- 52. LESSONS WHAT WORKED FOR US • Profile your memory requirements, monitor for scheduling issues
- 53. RIOT ENGINEERING BLOG THESE PROBLEMS AND MORE https://engineering.riotgames.com/ Riot engineering How we use data http://na.leagueoflegends.com/en/tag/insights
- 54. 함께 해주셔서 감사합니다!