![Python & Gevent
>>> import gevent
>>> from gevent import socket
>>> urls = ['www.google.com', 'www.example.com',
'www.python.org']
>>> jobs = [gevent.spawn(socket.gethostbyname, url)
for url in urls]
>>> gevent.joinall(jobs, timeout=2)
>>> [job.value for job in jobs]
['74.125.79.106', '208.77.188.166', '82.94.164.162']](https://image.slidesharecdn.com/scalingagameserver-170508130543/85/Scaling-a-Game-Server-From-500-to-100-000-Users-36-320.jpg)
Scaling a Game Server: From 500 to 100,000 Users
- 1. Scaling a Game Server From 500 to 100,000 users
- 2. Problem ● Game runs in browser ● Real time events from server to client ● Complex game logic, changing requirements ● Game lobby shows information realtime ● Transactions on accounts involved ● Need to scale to 5000 concurrent users ● Need to store all game events from players ● High availability
- 3. Agenda – Solution ● Load balancing ● Storing logs with Cassandra ● Event driven programming with Python/Gevent ● Publish/subscribe with Redis ● Transactions with MySQL ● Performance testing
- 5. How It Started ● C++ ● Runs on a single fat server ● Logs are written to files ● Each game in a separate process
- 9. Distributed Application POST /login GET /account GET /account 403 Access Denied POST /login 200 OK
- 10. Load Balancer with Sessions POST /login GET /account GET /account 200 OK POST /login 200 OK Memory Store or Database
- 11. Load Balancing: Stateful Send requests to same server Based on: - Cookies - HTTP Headers - Incoming IP address - HTTP parameters
- 12. HAProxy ● Event driven ● SSL off-loading ● WebSockets support ● Easily handle the bandwidth of 10G Ethernet with single CPU https://www.haproxy.org/10g.html
- 13. Real Time Events from Server
- 14. WebSockets HTTP Request Upgrade to WebSocket (communicate like TCP connection)
- 15. WebSockets ● Communicate like raw TCP sockets ● Works from browsers without plugins ● Starts as a HTTP connection – Works with many proxy servers, gateways etc. – Needs a capable load balancer – Using SSL makes the connect robust
- 16. Long Polling Client Server GET /events 200 OK, nothing GET /events Event 1 200 OK, Event 1 GET /events 30 secs 36 secs 6 secs
- 17. Long Polling ● Easy to implement ● Slightly delayed events ● More bandwidth used ● Good as a fallback or for less demanding applications
- 18. Storing Logs
- 19. Log Storage ● Store logs in files? ● What about easy retrival of a single game log – Store each game in a separate file?
- 20. Log Storage ● Problems with file storage: – Poor performance due to random disk writes – No redudancy – Harder to accumulate and query – Servers become stateful
- 22. Key Spaces Key A Column 1 Column 2 Column 3 Column 4 ... Value 1 Value 2 Value 3 Value 4 ... Key B
- 23. Key Distribution 0 – N/6 N/6 to 2N/6 2N/6 to 3N/6 3N/6 to 4N/6 4N/6 to 5N/6 5N/6 to N N = 2^128
- 24. Replication = 2 0 – N/6, previous node N/6 to 2N/6, previous node 2N/6 to 3N/6, previous node N = 2^128
- 25. Read = 1, Write = N
- 26. Read = N, Write = 1
- 27. Read = Quorum, Write = Quorum Quorum = ceil((n + 1) / 2)
- 28. Cassandra ● Fully distributed ● Replicate data on mutiple nodes ● Scale to hundreds of nodes ● Loose schema ● Handle millions of columns ● Good cluster management capablities
- 30. Handling Requests ● With threads or processes
- 31. Event Driven Programming ● Event loop ● Example callback while (True): event = wait_for_event(event_subscriptions) event.callback() function callback() { console.log(“Callback called, going back to event loop”); } setTimeout(callback, 3000);
- 32. Example Frameworks ● Javascript in Browser ● Most UI libraries (GTK+, Qt, etc.) ● Python (asyncio, gevent, Twisted) ● Node.js ● Java Netty ● Etc.
- 33. Advantages ● Performant for I/O driven applications ● Handling synchronization is easier
- 34. Disadvantages: Harder def process(): sub_1() io_1_sleep() sub_2() io_2_sql_query() sub_3() io_3_socket_write() def process_1(): sub_1() io_1_sleep(process_2) def process_2() sub_2() io_2_sql_query(process_3) def process_3() sub_3() io_3_socket_write() Threaded Program Event Driven Program
- 35. Disadvantages: Libraries ● Needs different set of libraries ● Written with event loop in mind
- 36. Python & Gevent >>> import gevent >>> from gevent import socket >>> urls = ['www.google.com', 'www.example.com', 'www.python.org'] >>> jobs = [gevent.spawn(socket.gethostbyname, url) for url in urls] >>> gevent.joinall(jobs, timeout=2) >>> [job.value for job in jobs] ['74.125.79.106', '208.77.188.166', '82.94.164.162']
- 37. Python & Gevent from gevent.server import StreamServer def echo(socket, address): socket.sendall(b'Welcome to the echo server!rn') handle = socket.makefile(mode='rb') while True: line = handle.readline() if line.strip().lower() == b'quit': break socket.sendall(line) handle.close() if __name__ == '__main__': server = StreamServer(('0.0.0.0', 16000), echo) server.serve_forever()
- 38. Event Driven Program with Gevent def process(): sub_1() io_1_sleep() sub_2() io_2_sql_query() sub_3() io_3_socket_write() def process(): sub_1() io_1_sleep() sub_2() io_2_sql_query() sub_3() io_3_socket_write() Threaded Program Gevent Program
- 39. Event Driven Program with AsyncIO def process(): sub_1() io_1_sleep() sub_2() io_2_sql_query() sub_3() io_3_socket_write() def process(): sub_1() yield io_1_sleep() sub_2() yield io_2_sql_query() sub_3() yield io_3_socket_write() Threaded Program AsyncIO Program
- 41. Lobby Information ● Need to collect updates from all games ● Need to send it to every user ● Thousands of games ● Changing every second ● Tens of thousands of users ● Need to see updates every second
- 42. Publish/Subscribe Publish Subscribe Event A Event A Event A Event A Game Server Lobby Server
- 43. Redis ● Key/value store ● Supports complex data structures for value ● Mainly for memory ● Also can persist data on disk ● Supports publish/subscribe
- 44. Redis High Availability Master ReplicationReplication Sentinel Sentinel Sentinel SyncSync
- 45. Sending Updates ● Send updates over WebSocket ● Send aggregates ● Send only delta of changes
- 46. Transactions
- 47. Transactions ● Multiple operations on user accounts ● Should all be atomic ● In other words, ACID compliance required
- 49. MySQL Replication Master ReplicationReplication SyncSync Read/WriteRead Only Read Only
- 50. MySQL Cluster Query Node Query Node Query Node Node 2Node 1 Partition 1 Node 4Node 3 Partition 2 MySQL Cluster - Data Layer
- 52. Performance Testing Framework ● A bot that simulates game play – With Python/Gevent ● Another framework – Locust
- 54. Cassandra MySQL Redis Lobby Servers Game Servers Load Balancers Clients Services API
- 55. License ● License: Creative Commons Attribution Share-Alike 4.0 International License ● Copyright: Sunil Mohan Adapa <sunil at medhas dot org> ● Icons from the Noun Project by: – Alena – Unlimic – Alex WaZa – Setyo Ari Wibowo – Royyan Wijaya – Under Creative Commons Attribution 3.0 United States License