At Scale With Style (Erlang User Conference 2012)
20 likes3,407 views
The document discusses Wooga's server architecture and innovative development approach, particularly focusing on their use of Erlang and Ruby to handle high-scale gaming environments. It emphasizes the importance of stateful game servers for managing user sessions and the efficient handling of data during gameplay. Additionally, it highlights the performance implications of different payload sizes and the architecture’s design around minimizing database bottlenecks while maintaining robust application functionality.
![Example
controller
in
Ruby
app.game_action '/:actor/fruit_tree/self/shake',
:observable => true,
:affects => [:fruit_trees, :user],
:params => [:x, :y] do
x, y = params[:x], params[:y]
fruit_trees[x, y].shake
end
DSL-‐like
definiHon
of
game
acHon
Skinny
as
controllers
should
be](https://image.slidesharecdn.com/2012-05-atscalewithstyle-martin-120530035628-phpapp02/85/At-Scale-With-Style-Erlang-User-Conference-2012-28-320.jpg)
![Looking
back
at
the
game
acHon
app.game_action '/:actor/fruit_tree/self/shake',
:observable => true,
:affects => [:fruit_trees, :user],
:params => [:x, :y] do
x, y = params[:x], params[:y]
fruit_trees[x, y].shake
end
Mapping
of
state
parts
to
game
acHons
Worker
knows
mapping
Worker
pushes
mapping
to
Erlang
on
startup
Erlang
can
query
mapping
if
needed](https://image.slidesharecdn.com/2012-05-atscalewithstyle-martin-120530035628-phpapp02/85/At-Scale-With-Style-Erlang-User-Conference-2012-31-320.jpg)
![Example
controller
in
Erlang
shake(Args, State) ->
Coords = g8_map:coords(Args),
Map = g8_game:get_map(State),
Tree = g8_map:find(Coords, Map),
NewTree = g8_fruit_tree:shake(Tree),
UserEffects = [incr_xp, decr_energy],
NewMap = g8_map:place(Coords, NewTree, Map),
[observable,
{state, g8_game:set_map(NewMap,
g8_user:apply(UserEffects, State))}].](https://image.slidesharecdn.com/2012-05-atscalewithstyle-martin-120530035628-phpapp02/85/At-Scale-With-Style-Erlang-User-Conference-2012-38-320.jpg)
At Scale With Style (Erlang User Conference 2012)
- 1. AT SCALE WITH STYLE Erlang <3 <3 <3 Ruby Mar$n Rehfeld, @klickmich
- 2. At Scale With Style How we roll Server architectures How to innovate Mind the limits
- 4. Typical game architecture Client HTTP Backend API
- 5. Typical game architecture Backend State Changes ValidaHon Persistence
- 6. The scale is interesHng 14 billion requests / month >100,000 DB operaHons / second >50,000 DB updates / second
- 7. Wooga’s approach to development Small independent teams for each game Team gets to choose tools and technologies Same team also does ops a,er going live Culture of sharing Look around what is there, pick/adapt exis;ng solu;ons, but take ownership for what you include
- 8. ExisHng backends – Technology landscape
- 9. At Scale With Style How we roll Server architectures How to innovate Mind the limits
- 10. Most games use stateless applicaHon servers Server Database
- 11. And then there’s sharding lb app app app app app app app app app My My SQL SQL slave slave
- 12. More app servers, more sharding lb app app app app app app app app app app app app app app app app app app My My My My My My My My SQL SQL SQL SQL SQL SQL SQL SQL slave slave slave slave slave slave slave slave
- 13. Wait, seriously?! lb lb app app app app app app app app app app app app app app app app app app app app app app app app app app app app app app app app app app app app app app app My My My My My redis redis redis redis redis SQL SQL SQL SQL SQL slave slave slave slave slave slave slave slave slave slave
- 14. Find the flaw “Stateless applicaHon servers guarantee one thing: The data is never where you need it!” Paolo Negri, Developer @ Wooga
- 15. Strong session pa`ern User starts playing many transformaHons of the same set of data User stops playing
- 16. Stateful servers and DBs Server Database One Game Session
- 17. Stateful game server One process per acHve user gaming session ... holds the current state and is the only one that can modify it (strong encapsulaHon) ... handles all API calls for the given user one a,er the other (concurrency control through actor model) ... loads the game state from storage and writes it back periodically and on process termina;on (;meout = user stopped playing)
- 18. The DB is no longer the bo`leneck Stateless Stateful 30.000 22.500 15.000 700 7.500 0 DB operations / second
- 19. Magic Land uses Erlang Details: Awesome presentaHon on the Magic Land game server by @knuHn & @hungryblank h`p://www.slideshare.net/wooga/from-‐0-‐to-‐1000000-‐daily-‐users-‐with-‐erlang
- 20. At Scale With Style How we roll Server architectures How to innovate Mind the limits
- 22. Erlang & Ruby Erlang is great Concurrent, robust Great for opera;on Ruby is great Concise, expressive, flexible Great for development
- 24. Bringing two worlds together Server Worker session sender Worker session Worker ... Worker receiver session Worker
- 25. Do you know Mongrel2? Web Server that hooks up applicaHons via ➡ We chose the same queue setup & message format Server Worker session sender Worker session Worker ... Worker receiver session Worker
- 27. ConnecHng the dots Mongrel2 h`p://mongrel2.org/ protocol & ZeroMQ setup Erlang: h`ps://github.com/hungryblank/emongrel2 Ruby rack-‐mongrel2 fork hKps://github.com/khiltd/khi-‐rack-‐mongrel2 rack protocol hKp://rack.rubyforge.org Sinatra hKp://www.sinatrarb.com/ ➡ essenHally we are speaking HTTP over ZeroMQ and can hook up any Rack-‐based Ruby web framework
- 28. Example controller in Ruby app.game_action '/:actor/fruit_tree/self/shake', :observable => true, :affects => [:fruit_trees, :user], :params => [:x, :y] do x, y = params[:x], params[:y] fruit_trees[x, y].shake end DSL-‐like definiHon of game acHon Skinny as controllers should be
- 29. Example model in Ruby class FruitTree < Tree property :last_shake_time, :type => Integer, :default => 0 property :collectable_fruit_count, :type => Integer, :default => 0 def shake raise G8::Error::Validation, "no fruit!" unless carries_fruit? session.user.xp += 1 session.user.energy -= 1 self.last_shake_time = game_time self.collectable_fruit_count = config.fruit_count end end Easily unit testable Minimal amount of code
- 30. Game state Game state is split in mulHple parts user, map, fruit_trees etc. Erlang does not care about content Serialized Ruby objects Erlang does know mapping of state parts to URLs
- 31. Looking back at the game acHon app.game_action '/:actor/fruit_tree/self/shake', :observable => true, :affects => [:fruit_trees, :user], :params => [:x, :y] do x, y = params[:x], params[:y] fruit_trees[x, y].shake end Mapping of state parts to game acHons Worker knows mapping Worker pushes mapping to Erlang on startup Erlang can query mapping if needed
- 32. NICE! http://www.flickr.com/photos/aigle_dore/
- 33. At Scale With Style How we roll Server architectures How to innovate Mind the limits
- 34. Performance impact for large payloads 200 kB payload 6000 5000 Requests / s 4000 Pure 3000 Erlang 2000 Ruby 1000 ZMQ 400 rps 0 1 # of workers 8
- 35. NOT CPU bound 25% Fixed bandwith limit @ 300 MB/s
- 36. This has happened before 2,000,000" 1,500,000" 1,000,000" 500,000" 0" Apr*10" Jul*10" Oct*10" Jan*11" Apr*11" Jul*11" Oct*11"
- 38. Example controller in Erlang shake(Args, State) -> Coords = g8_map:coords(Args), Map = g8_game:get_map(State), Tree = g8_map:find(Coords, Map), NewTree = g8_fruit_tree:shake(Tree), UserEffects = [incr_xp, decr_energy], NewMap = g8_map:place(Coords, NewTree, Map), [observable, {state, g8_game:set_map(NewMap, g8_user:apply(UserEffects, State))}].
- 39. Example model in Erlang shake(Tree) -> validate_carries_fruit(Tree), FruitCount = g8_fruit_tree_config:fruit_count(Tree), Tree#fruit_tree{last_shake_time = g8_game:gametime(), collectable_fruit_count = FruitCount}.
- 40. ✓ Y ES
- 41. QuesHons? MarHn Rehfeld @klickmich slideshare.net/wooga wooga.com/jobs