B 2 line game cloud - our personal ec2
7 likes9,500 views
The document discusses improvements made to a game cloud platform between 2014 and the present. It summarizes key points as follows: 1. In 2014, the platform faced issues serving games globally from a single data center. This motivated the start of the Game Cloud project. 2. Significant improvements were made including using HAProxy for load balancing instead of hardware, establishing points of presence around the world, and improving network architecture to reduce latency. 3. The platform now uses Docker Swarm and keeps its architecture simple and reliable. It also implemented an auto-scaling system called Gearbox to provide high availability at low cost.
B 2 line game cloud - our personal ec2
- 2. Currently on the Cloud
- 3. 1 Our situation in 2014 2 How we improved 3 Sweet things 4 More sweet things 5 Future
- 4. 1 Our situation in 2014 2 How we improved 3 Sweet things 4 More sweet things 5 Future
- 5. Our situation in 2014 1st gen “HSP” – (2014) from cybercafe/漫画木さ/PC방 platform 2nd gen ECDH-based key exchange Platform Billing/AAA/Monitoring, etc. & Game servers “LGC” – (cloud release) 3rd gen: “Trident” – (current)
- 7. Process Dev/QA/Sandbox/REAL… Get VM Get L4 binding ACLs/storage, etc. From hours to days…
- 8. Thus, the Game Cloud project began…
- 9. 1 Our situation in 2014 2 How we improved 3 Sweet things 4 More sweet things 5 Future
- 10. Global In our case GSLB (Global Server Load Balancing) HAProxy instead of hardware L4 Multi-team effort client, server, cloud We get more flexibility less latency
- 11. LINE Global POP New York Tokyo Seoul HK Singapole Beijing Frankfurt a a
- 12. Network layer control High latency Fit for cloud Global Testing in Thailand TH KR GSLB TH SG KR HAProxy 500 1000 1500 2 12 206 16
- 13. Process Dev Ops Government The structure of your organization affects the structure of your software. And vice-versa! DevOps Small StartupCHOOSE
- 14. Process Progressive / Easy, simple Requirements for our new platform : We have many third parties and technology stacks involved... Etc.
- 15. Process For ourselves simple reliable future proof
- 16. Process Why not… Docker Swarm or CoreOS Mesos Kubernetes
- 18. Process KEEP THINGS SIMPLE AND RELIABLE! For distributed systems, minimize coordination A good paper: https://blog.acolyer.org/2016/01/19/dcft/ • Polling • 1way dataflow • Idempotency • Commutativity • limited trust
- 19. LGC Story Games planned for release were suddenly canceled but we needed to show results! Strong “sales” efforts to release other games on the LGC platform
- 21. Putting Out Fires The release was a success,followed by a quick scaling-up, and then our first fires… TECHNICAL Riak fire : the system works with Riak down OE fire: the system works with OE down Hardware and conf fires (TDI! Soon to come!) Full container reboot improved our design through limited trust
- 23. 1 Our situation in 2014 2 How we improved 3 Sweet things 4 More sweet things 5 Future
- 26. Monitoring
- 27. 1 Our situation in 2014 2 How we improved 3 Sweet things 4 More sweet things 5 Future
- 30. High availability Low cost Why Do We Need It?
- 32. How Does It Work?
- 33. How Did We Build it? Data Collector Monitoring API Predicator Metrics Raw Metrics Scaler States Game Cloud API Excute Scaling Gearbox
- 34. Challenges Complex query Plenty of metrics records millions of records per day Scalability of the auto-scaling system itself
- 35. Solutions Data Collector Monitoring API Predicator Metrics Metrics Scaler States Game Cloud API Excute Scaling Gearbox Module Storage ElasticSearchAdmin Site Strategy Scaler Log Metrics 1.Strategy 2.Metrics { } Predicator Log 1.Strategy 2.Scaler Log
- 36. Knife Admin Site
- 38. Upgrade
- 40. Back to Jojo What’s coming next!
- 41. 1 Our situation in 2014 2 How we improved 3 Sweet things 4 More sweet things 5 Future
- 42. Future QUIC SDN ACL, IP by container, VLAN etc. Cloud storage TDI Distributed GC – link paper DCTCP Image GC
- 45. Cloud Storage
- 46. SDN- Software-Defined Networking Container specific IP ACLs VLANs
- 47. TDI- Test-Driven Infrastructure hardware OS configuration images backup/ restore Automated testing for Etc. firmware, version, etc.
- 48. Distributed GC Max Avg.GC pause http://arxiv.org/pdf/1504.02578.pdf Median Std.Dev. Mean 7.847 0.0 2.296 0.579 2.312 GC off 7.743 12.243 2.294 0.582 2.311 Blade 164.206 12.339 2.297 3.395 2.403 GC on
- 49. Added in Linux 3.18 https://kernelnewbies.org/ Linux_3.18 http://simula.stanford.edu/ ~alizade/Site/DCTCP.html DCTCP- Data Center TCP - high burst tolerance - low latency - high throughput http://simula.stanford.edu/~alizade/Site/DCTCP.html
- 50. Distributed GC Because we generate tons of Docker images
- 51. And more and more…
- 54. • AP • Optional CP • Index/search • CRDT • Multiple backends • User ACL support RIAK/Choose a Safe and Simple Friend Make a deliberate choice of consistency model SQL NoSQL But actually… With the authorization of Kingsbury Kyle (Aphyr)