Five Cliches of Online Game Development
- 1. Five Clichés Of Online Game Development That We Wish Weren’t True (But Will Probably Ruin Your Life Some Day) Ian Dundore – TRC Family Entertainment Ltd
- 2. Who is he? THIS GUY STANDING UP
- 3. True facts. • Started making games as a pre-teen. • Game journalist, 1999 - 2006 • B.Sc. Computer Science, 2004 • First game programming job in 2006 – Gods & Heroes: Rome Rising • CCP Games, 2008 – 2012 – Several EVE Online expansions – Dust 514
- 4. This slide is an excuse for me to make jokes in grey text that I will not read out loud. For those of you who actually read this far along a slide, bravo. I salute you, literati. THIS IS THE SLIDE WHERE I TALK ABOUT WHAT I’M GOING TO SAY.
- 5. That man has a chart. Look at him. He’s probably never worked a day in his life. He’s not even wearing a suit. By the way, don’t Google Image Search for “Getting Down to Business” with SafeSearch off. LET’S GET DOWN TO BUSINESS.
- 6. Cliché the first. THE CLIENT IS IN THE HANDS OF THE ENEMY
- 7. Ah, Client, my old foe… “Never trust the client. Never put anything on the client. The client is in the hands of the enemy. Never, ever forget this.” – Apocryphal, often misattributed to Raph Koster.
- 8. The numerology is strong with these slides. TWO MEANINGS
- 9. #1: Input • Data from the client is raw input, period. • Validate or escape everything you receive. – Should be familiar to Web devs. • Thought Exercise: What if the user could call my function/class/code snippet?
- 10. Corollary: Don’t Melt Your Server • Light/vital systems: Double-check everything. – Speech, wallet • Heavy/fuzzy systems: Calculate on client, verify possibility on server. – Physics, movement. • How much error can you tolerate?
- 11. Case Study: Speedhacks • Movement based on client-supplied position updates. • Server verifies for maximum possible speed. • How to make a speedhack: figure out maximum range, scale movement vector. – This is how real speedhacks work: WoW, EQ, etc.
- 12. #2: Output • Data to the client = data to the player. • Anything you send to the client, the user will see. • Anything measurable via the client will be decoded: game systems, etc.
- 13. Case Study: BACON • EVE Online comes with debug output viewer. • Identifying info logged each time a player entered your vicinity. • This was not displayed to the player in-game. • Result: External program plays audible alerts when enemy players enter the area. – Logs introduced in 2002, BACON released 2008.
- 14. Addendum • Anything in the client, the player will see.
- 16. Nuance • Logical extreme: Gaikai. • Compare vulnerabilities of trust to advantages in user experience. – Offloading heavy work to the client = good! – Lots of bling = happy players!
- 17. Cliché the second. PREMATURE OPTIMIZATION IS THE ROOT OF ALL EVIL
- 18. The Quote “We should forget about small efficiencies, say about 97% of the time: Premature optimization is the root of all evil.” - Donald Knuth, super genius
- 19. No “Two Meanings” Slide This Time • The proper order: – Find the fun – Make it good – Make it fast • However, don’t cut corners for the sake of it. – Avoid the most obvious blunders. – Test, test, test.
- 20. The Case for Quick • EVE - Planetary Interaction • 4 month development cycle – 10 weeks of “real” development • Fluctuating requirements • Major new features injected halfway through
- 21. The Case for Quick? • Heavy overtime • Shipped first-revision architecture – ~10 major bugs discovered after release – 1 item duplication bug discovered – Two hotfixes – Memory leaks relied on daily server reboots
- 22. The Case for Quick! • Concurrency goals exceeded – 30,000+ concurrent users after launch – 25% peak CPU usage or less – Memory not an issue, EVE already rebooted daily • Post-launch rewrite: 2 + 1 weeks – Eliminated memory issues • 50% memory usage reduction by using Python Slots – Eliminated duplication & high-priority issues – ~4 new bugs filed after 2 years in the wild
- 24. Case Study: The Five Bug • Gods & Heroes – in production for > 4 years • Fully home-grown, pure C++ engine • Largest scale test: 30-50 users, ~2-3 hours uptime • Target: ~5000 users, 72 hours uptime. • No automated test tools
- 25. The Law of Five • Server occasionally crashed – Corrupted stacktrace, clear memory corruption • Usually the value 0x5, hence the name – Random code module • But usually combat or special abilities – Cause not clearly evident, debugger useless – Deprioritized until beta
- 26. Beware the Fives of March Highest beta concurrency: ~1000 users, 30 minutes uptime Average concurrency: 300-500 users, 10-15 minutes uptime
- 27. The Fives Have It • ~6 weeks spent debugging. • Deep bug in 5-year-old inter-module communication • Very rare in the wild… – As users rise, “very rare” approaches “certain” • Bug fixed October 8, 2007 • Company closed October 9, 2007
- 28. Learn From These Mistakes Too Big to Fail Rewriteable • Networking code • Low-level code • Scene layout • Individual box features • Art style • User interface • Genre • Lore & character details Technical & creative direction Stuff built upon that stuff
- 29. Cliché the third. THERE ARE TWO TYPES OF UNKNOWNS.
- 30. Obligatory quote slide. “There are known unknowns; … things that we know we don’t know. But there are also unknown unknowns … things that we don’t know we don’t know.” - Donald Rumsfeld, defense guru
- 31. The “engineer’s rule of thumb” holds • Any given project will have work injected – Technical requirements, design changes, optimization, iteration, platform upgrades… • Account for this when planning • Planning and tracking tools are invaluable – But you will hate them every step of the way
- 32. How It Works • Come up with some estimation benchmark – Homework: Look up “Complexity Points” • Estimate ALL THE THINGS • Keep track of what you finish in a constant time period (2 weeks, 1 month, etc.) – Use these to calculate Ultra Nerdy Stats • Averages, medians, standard deviations… • BAM. You have a rough estimate of how long your project will take.
- 33. More Importantly… • Keep track of everything you add. – Note when you added it. • Everything you add must be estimated too. – Use the same metric as before. • BAM. Now you have a guideline of how much unknown work to expect.
- 34. Examples • Newly-formed team, new feature, established tech framework & art style – ~50% of work completed was injected during development • Gelled team, iteration on existing feature – ~25% of work completed was injected • Your numbers will vary!
- 35. Cliché the fourth. ANY SUFFICIENTLY CREATIVE TOOL WILL BE USED TO DRAW A…
- 36. Problem Users • 1% of your playerbase will generate 90% of your support load. • Good logging, data retention plans are key • Log everything involving money, real or otherwise
- 37. Case Study: Zero-Day Exploit • Item duplication exploit due to subtle bug in a game feature’s code. • Not readily apparent without hours/days of observation. • Easily disrupted through normal play. • If manipulated, would generate perpetual stream of items for free. – Unattended!
- 38. Incidence • 136 different abusers in prior 6 months • ~200 bugged item generators • Circa $30,000 worth of in-game currency • 120 abusers were short-time offenders – Likely unnoticed, small-scale • 3 abusers generated over 90% of exploit- driven in-game currency
- 39. The Value of Logs • Excellent logs allowed us to: – Pinpoint start of item duplication – Trace duplicated items through “fence” accounts – Measure likely scale of duping operations over time – Ban them all!
- 40. Creativity vs. Safety • Any sufficiently advanced tool… – FPS sprays, Minecraft… • Carefully weigh support cost vs. user fun – Family image? Intolerant audience? • Have good support tools in place – Habbo blockade
- 41. Cliché the fifth isn’t really a cliché at all. NEVER WRITE YOUR PRESENTATION’S TITLE BEFORE YOUR PRESENTATION
- 42. Questions?