Going Multi-Node
0 likes698 views
The document discusses the implementation of a multi-node architecture for a multi-user dungeon (MUD) game called Exventure, focusing on aspects such as clustering, leader selection using the Raft protocol, and challenges faced during the transition from a single node to a multi-node system. Key topics include maintaining data consistency across nodes, optimizing performance, and utilizing chat services for messaging. Specific performance improvements and technical solutions are also mentioned, highlighting the project's evolution and development resources available on GitHub and its documentation site.
![libcluster
config :libcluster,
topologies: [
local: [
strategy: Cluster.Strategy.Epmd,
config: [
hosts: [:"world1@host", :"world2@host"]
]
]
]](https://image.slidesharecdn.com/goingmulti-node-180910174007/85/Going-Multi-Node-21-320.jpg)
Going Multi-Node
- 2. Eric Oestrich SmartLogic - smartlogic.io GitHub: oestrich Twitter: ericoestrich
- 3. Going Multi-Node ● What is a MUD? ● Clustering ● Leader Selection ● Spanning the Cluster ● Problems Encountered
- 7. What is a M.U.D.?
- 8. What is a MUD? ● Multi-User Dungeon ● Text based multiplayer game ● Started in the late 70s ● World of Warcraft or EverQuest, but with no graphics
- 9. What is a MUD?
- 10. What is a MUD?
- 11. Supervision Tree ● ExVenture ○ Cluster.Supervisor ○ Raft ○ Data.Repo ○ Web.Supervisor ○ Game.Supervisor ○ :ranch
- 12. Supervision Tree ● Game.Supervisor ○ Game.Session.Registry ○ Game.Config ○ Game.Caches ■ Game.Items ■ … ○ Game.Session.Supervisor ○ Game.World
- 13. Supervision Tree ● Game.World ○ Game.World.Master ○ Game.World.ZoneController ○ Game.ZoneSupervisor ○ Game.ZoneSupervisor ○ ...
- 14. Supervision Tree ● Game.ZoneSupervisor ○ Game.Zone ○ Game.Room.Supervisor ■ Game.Room ■ ... ○ Game.NPC.Supervisor ■ Game.NPC ■ ….
- 15. Starting Point
- 16. Everything Assumed One Node ● Heavy geared for a single node ● Most processes were local Registry ● Data stored in local ETS tables ● Lots of in process state (a virtual world)
- 17. Paths Forward ● Umbrella App ● Single Application
- 18. Single Application that is Cluster Aware
- 19. Clustering
- 21. libcluster config :libcluster, topologies: [ local: [ strategy: Cluster.Strategy.Epmd, config: [ hosts: [:"world1@host", :"world2@host"] ] ] ]
- 22. Start the nodes iex --sname world1 -S mix iex --sname world2 -S mix
- 23. Start the nodes
- 24. Picking a leader
- 26. Raft Basics
- 27. Raft Basics
- 28. Raft Basics
- 29. Raft Basics
- 30. Raft Basics
- 31. Picking a Leader ● Each node waits a random amount of time and picks itself as leader ● Each other node votes for the first node it sees ask ● Once a majority is chosen the leader is picked and starts the world
- 32. What the leader does ● Pushes zones out across the cluster ● When a node dies ○ Looks at zones that should be online ○ Spins them up across the cluster
- 33. Squabble ● ExVenture’s leader election has been pulled out ● GitHub: oestrich/squabble
- 34. Squabble Callbacks defmodule MyApp.Leader do @behaviour Squabble.Leader @impl true def leader_selected(term) do end @impl true def node_down() do end end
- 35. Why a majority of the cluster?
- 36. Removing local Registry Switch to :global registry
- 37. Spanning the Cluster ● Cache updates need to be updated on each node ● Process groups to handle this ● Might be a nicer way to handle these
- 38. Join the pg2 group on cache start defmodule Game.Items do @ets_key :items def init(_) do :ok = :pg2.create(@ets_key) :ok = :pg2.join(@ets_key, self()) #... end
- 39. Client API def insert(item) do members = :pg2.get_members(@ets_key) Enum.map(members, fn member -> GenServer.call(member, {:insert, item}) end) end
- 41. Calls timing out ● Might be a network blip and a previously stable call is no longer stable
- 42. Circuit Breakers
- 43. Circuit Breakers
- 44. Circuit Breakers
- 45. Things that should happen at most once ● Attached to Gossip, a chat service ● Each node connects as a websocket to Gossip ● New messages posted at most once to the local channel ● Only the leader node should handle these actions
- 46. Performance Tweaks The journey from 230 to 3500
- 47. Single process being overloaded by messages ● Room processes became a bottleneck ● Create a side process that handles notifications ● PR #72 ● ~230 -> ~600
- 48. Single process being overloaded by data size ● Session registry was overloaded with size of user data ● Pushing a large preloaded Ecto struct around ● Massively simplify what is stored ● PR #73 ● ~600 -> ~1200
- 49. Too large messages ● The data being passed around was too large ● Same huge User structs ● Ran out of ram at 50GB ● Use same simplification in messages ● PR #74 ● ~1200 -> ~3500
- 51. Specs - Intel Core i7-6700K - Quad Core, with Hyperthreading - 64GB of RAM
- 52. Demo
- 55. Thanks! ● GitHub: oestrich/ex_venture ● Documentation: exventure.org