![JSON
{
“x”: 3.14,
“name”: “McK”,
“isReady”: true,
“favoriteNumbers”: [ 1, 1, 2, 3, 5 ],
“visitationCounts”:
{
“Los Angeles”: 33,
“New York”: 12
}
}](https://image.slidesharecdn.com/chapter13-240923000937-05991dbe/85/Cloud-Hosting-and-Dedicated-Servers-Multiplayer-Game-Programing-9-320.jpg)
![Process Monitoring Sample Code
function checkHeartbeats()
{
var processesToKill = [], processUUID, process, heartbeatAge;
var time = getUTCSecondsSince1970();
for( processUUID in gProcesses ) {
process = gProcesses[ processUUID ];
heartbeatAge = time - process.lastHeartbeat;
if( heartbeatAge > gMaxStartingHeartbeatAge ||
( heartbeatAge > gMaxRunningHeartbeatAge
&& process.state !== 'starting' ) ) {
console.log( "Process " + processUUID + " timeout!" );
processesToKill.push( process.child );
}
}
processesToKill.forEach(function(toKill) {toKill.kill();} );
}
setInterval( checkHeartbeats, gHeartbeatCheckPeriod );](https://image.slidesharecdn.com/chapter13-240923000937-05991dbe/85/Cloud-Hosting-and-Dedicated-Servers-Multiplayer-Game-Programing-20-320.jpg)
Cloud Hosting and Dedicated Servers - Multiplayer Game Programing
- 1. Multiplayer Game Programming Chapter 13 Cloud Hosting Dedicated Servers
- 2. Chapter 13 Objectives Hosting Pros and Cons – Is hosting the right choice for your game? Tools for backend development – REST, JSON, Node.JS Terminology – Game Instances, Processes and Virtual Machines Managing Everything – Local Server Manager – Virtual Machine Manager
- 3. To Host or Not To Host Should you host dedicated servers? – Originally, only option for AAA studio with big upfront budgets Cloud removes upfront hardware costs to hosting dedicated servers Just because you can, should you?
- 4. To Host or Not To Host, Cont’d Cons – Complexity • must manage fleet of servers! – Reliance on third party • Cloud hosts go down sometimes- even Amazon, Google and Microsoft – Unexpected hardware changes • Hosts must meet min spec, but there can be lots of variety above that spec – Loss of player ownership • Evangelist players take pride in running servers
- 5. To Host or Not To Host, Cont’d Pros: – Reliable, Scalable, High bandwidth servers • Not dependent on players to host games • Cloud host IT makes sure the data keeps flowing – Cheat prevention • Players have no access to authoritative code – Reasonable copy protection • No need for intrusive DRM on client
- 6. Tools C++ and custom protocols good for fast, low level client and server, but not for high level server management – No high level memory management – No built in async request handling – No built in high level request exchange protocol – No built in human readable, debuggable data format
- 7. REST Representational State Transfer – Self contained request format – Text Base • Human readable • Easily Debuggable – Built on top of HTTP • GET – Retrieve a resource • POST – Create / Edit a new resource • DELETE – Delete a resource • PUT – Put a resource directly at a URI
- 8. JSON JavaScript Object Notation – Text based, human readable – Compilable JavaScript – Objects are hashes of primitive data types • string • number • bool • array • object • undefined • null
- 9. JSON { “x”: 3.14, “name”: “McK”, “isReady”: true, “favoriteNumbers”: [ 1, 1, 2, 3, 5 ], “visitationCounts”: { “Los Angeles”: 33, “New York”: 12 } }
- 10. Node.JS Open Source JavaScript platform Runs Googles V8 Engine – Now semi-ES6 compatible Single threaded asynchronous processing model – Non-preemptive multitasking • Disk I/O, Net requests, etc. – Event driven • Ticks, like a game! Express.JS – Popular Web / REST Server
- 11. What Does “Server” Mean? Overloaded! – Server Game Instance – Game Server Process – Game Server Machine
- 12. Server Game Instance Simulates a game world shared by its players – A 16 player FPS battle – A 5v5 LoL game in Summoner’s Rift – A 4 player Battle Mode game in Mario Kart
- 13. Game Server Process Your game, as far as the OS is concerned – Hosts one or more server game instances – Multiple games instances per process? • Pros – Share large immutable resources » Navigation grid, collision geometry, etc – Fine grain division of CPU time • Cons – No mutable singletons – No protected memory » Crash in one instance can corrupt / bring down all in process – Harder to test
- 14. Game Server Machine Each machine has one Operating System – But one or more game server processes! – Multiple processes per machine? • Pros – Processes live in protected memory- Safe! – Fewer OS resources required per Server Game Instance • Cons – Coarse control over multithreading » One process could potentially hog resources
- 15. Hardware In the cloud, machines are typically Virtual Machines – Virtual Machine Image (VMI) encapsulates OS and game process executable – Cloud host can move virtual machine to different physical machines at will • Rapidly handle hardware failure • Rapidly spin up new servers as requested • Utilize hardware to fullest extent – Multiple VMs per physical machine
- 16. Local Server Process Manager Need to administer multiple game processes per server machine – Start up new process when necessary • With desired configuration ( Team Deathmatch! ) – Monitor process health • Sometimes processes crash or freeze – Report back to overall platform Maybe could use tools built into OS – Not cross platform – Not fully customizable Custom solution built in Node.JS! – Cross platform and full customizable!
- 17. Local Server Process Manager REST REST Endpoints – GET /processes • Gets a list of currently running processes, with unique ids – POST /processes • If not running maximum process – spawn new game process using childProcess.spawn – Save unique id of process in running list » When process closes, automatically remove from list » Read configuration data from body of request and pass to process
- 18. Local Server Process Manager REST Cont’d REST Endpoints – POST /processes/:processUUID/kill • Attempt to kill the process with given id – POST /processes/:processUUID/heartbeat – Register that given process is still alive » Not frozen – What good is that? » Monitoring!
- 19. Process Monitoring Periodically run through the process list and check the last time a heartbeat was received – If not received within threshold, kill process • Why important? • Make room to run another process! • Also can notify devops to investigate Game server process should hit heartbeat endpoint periodically to let LSPM know it’s alive – Microsoft Open Source C++ REST SDK – Or just use TCP socket ( SDK usually preferable )
- 20. Process Monitoring Sample Code function checkHeartbeats() { var processesToKill = [], processUUID, process, heartbeatAge; var time = getUTCSecondsSince1970(); for( processUUID in gProcesses ) { process = gProcesses[ processUUID ]; heartbeatAge = time - process.lastHeartbeat; if( heartbeatAge > gMaxStartingHeartbeatAge || ( heartbeatAge > gMaxRunningHeartbeatAge && process.state !== 'starting' ) ) { console.log( "Process " + processUUID + " timeout!" ); processesToKill.push( process.child ); } } processesToKill.forEach(function(toKill) {toKill.kill();} ); } setInterval( checkHeartbeats, gHeartbeatCheckPeriod );
- 21. Virtual Machine Manager Need to administer multiple machines! – Find a VM with a space for a new process – If no VM with space for a process, ask cloud host to spin up a new VM – Monitor VM and LSPM health! • Don’t want to leak VMs! – LSPM could crash / freeze and you might lost track of processes that exit unexpectedly – Spin down VMs when not in use • Save money!
- 22. Virtual Machine Manager REST Endpoints – POST /processes • Find an LSPM with room for a new process and spin spawn it – Posts to LSPM’s /processes endpoint – If no LSPM with space available » Provision VM from cloud provider » LSPM starts at boot » Then request process • Tracks any new process or LSPM in global lists • Beware of race conditions from multiple incoming requests at once!
- 23. Provisioning a VM Different for each cloud provider Typically, Node.JS package for cloud provider – VMI must be stored with cloud provider already – Can store multiple VMIs • Request VMI and min system specs through cloud provider API
- 24. Virtual Machine Manager REST, Cont’d Endpoints – POST /vms/:vmUUID/heartbeat • Register VM and associated LSPM is still alive • LSPM must call periodically – After checking for heartbeats from processes? – Can also send update on number of processes running – Can send heartbeat whenever number of processes change » Crash, game over, spin up request complete • Heartbeat indicating zero processes running on a VM can trigger VMM to request cloud provider to spin down VM – Should wait a little to make sure no new process request incoming
- 25. Virtual Machine Monitoring Check for heartbeat every period ( 60 seconds? ) – Similar to LSPM – Beware race conditions now! • VM might already be shutting down – Starting up – Starting a new process • Need to track every in progress request
- 26. Matchmaking and Virtual Machine Management Keep them separate! – Virtual Machine Manager is only responsible for spinning up machines – Matchmaking is only responsible for finding matches • If Matchmaking service cannot find an available match, it asks VMM to spin up a server • Separation of Concerns! • Allows integration with any third party service – Steam, PSN, Xbox Live, Game Center, etc.