Erlang factory layered architecture - final
2 likes1,172 views
The document describes a new layered architecture for a social and casual gaming platform. The key points are: 1) The previous architecture had inconsistent designs, unclear responsibilities, technical debt, and scaling issues with 7 layers of caching. 2) The new architecture aims to be flexible, predictable, scalable, simple with clean separation and no horizontal dependencies. 3) It has four main layers - client, interface, service, and storage - each with clear responsibilities around integration, authorization, composition, and persistence. Building blocks are small, simple applications and a platform to abstract complexity.
![Interfaces
https post
/user/get/
3rd Party {“auth”:
{“token”: “UwAA_AWeff...”} Highscore
Backend } Service
list
1
get([friends]) Application
User
2 User 2 Interface
Interface
Service
list
get(me)
1
count /application/highscores/
1
Friends
Service
1
User
Service Awards
Service
16](https://image.slidesharecdn.com/erlangfactory-layeredarchitecture-final-130323000204-phpapp02/85/Erlang-factory-layered-architecture-final-18-320.jpg)
![definition
User
profile logic Service
template
+ Award
html,
Service
[properties]
18](https://image.slidesharecdn.com/erlangfactory-layeredarchitecture-final-130323000204-phpapp02/85/Erlang-factory-layered-architecture-final-21-320.jpg)
![header
definition
User
logo profile Service
logic
template
navigation search + Award
html,
Service
[properties]
18](https://image.slidesharecdn.com/erlangfactory-layeredarchitecture-final-130323000204-phpapp02/85/Erlang-factory-layered-architecture-final-22-320.jpg)
![ame.com /
htt p://www.ag
preprocess
...
widget platform
footer ...
home_page
new games
header body
...
definition
User
logo profile Service
logic
template
navigation search + Award
html,
Service
[properties]
18](https://image.slidesharecdn.com/erlangfactory-layeredarchitecture-final-130323000204-phpapp02/85/Erlang-factory-layered-architecture-final-23-320.jpg)
Erlang factory layered architecture - final
- 1. A Layered Architecture Robustness & Efficiency dennis.docter@spilgames.com March 22nd, 2013
- 2. 2
- 3. A legacy of growth
- 4. Social and casual gaming platform 50+ portals 190+ countries 4
- 5. Casual games Multi player games Social games Native games
- 6. 200+ million unique users per month 250" 200" Traffic growth 150" over 1600% 100" 50" 5/1/12" 1/1/12" 9/1/11" 0" 5/1/11" 1/1/11" 9/1/10" 5/1/10" 1/1/10" 9/1/09" 5/1/09" 1/1/09" 9/1/08" 5/1/08" 1/1/08" 9/1/07" 5/1/07" 1/1/07" 90" 80" 70" 60" Tech dept. 50" 40" growth 30" over 2000% 20" 10" 0" 1/1/08" 1/1/09" 1/1/10" 1/1/11" 1/1/12"
- 7. XML Json Interfacing Interfacing + plain GET Intefacing Intefacing Caching SQL Business Caching Authorization + Authentication Business Caching Caching logic logic Authorization Business SQL Business Business logic Caching Caching logic logic Business SQL logic Caching SQL SQL Business SQL logic Swift SQLSchema API Schema Schema Schema Mysql Write DB Mysql DB Mysql DB Mysql DB Mysql Reader Memcache Mysql Mysql DB Cluster Readers 7
- 8. Problems • Inconsistent design & interfaces • Unclear responsibili4es • Technical debt • Independent teams developing the same features. • 7 layers of caching hell • Scaling
- 9. Problems • Inconsistent design & interfaces • Unclear responsibili4es • Technical debt • Independent teams developing the same features. • 7 layers of caching hell • Scaling not a problem :)
- 10. 9
- 11. A new approach
- 12. New architecture expectations • Flexible • Predictable • Scalable TM d wor t B uzz lian p C om 11
- 13. Choices made • Simple • Clean separa4on • No horizontal dependencies • Fault tolerant • Distributed 12
- 14. Layers Responsibilities Client • Integra4on • State • Authen4ca4on Interface • Aggrega4on • Valida4on • Authoriza4on Service • Composi4on • Transforma4on • Caching Storage • Persistence • Sharding 13
- 15. Layers Responsibilities Client • Integra4on • State • Authen4ca4on Interface • Aggrega4on Stateless • Valida4on • Authoriza4on Service • Composi4on • Transforma4on • Caching Storage • Persistence • Sharding 13
- 16. Building blocks Application • Small & Simple • Limited scope • Quick to deploy Platform • Abstracts complexity • Low change rate • Long lifetime 14
- 17. Walkthrough
- 18. Interfaces https post /user/get/ 3rd Party {“auth”: {“token”: “UwAA_AWeff...”} Highscore Backend } Service list 1 get([friends]) Application User 2 User 2 Interface Interface Service list get(me) 1 count /application/highscores/ 1 Friends Service 1 User Service Awards Service 16
- 19. Widgets
- 20. Widgets logo widget profile widget search widget navigation widget recommendation widget new_games widget most_played widget recent_games widget social_games widget
- 21. definition User profile logic Service template + Award html, Service [properties] 18
- 22. header definition User logo profile Service logic template navigation search + Award html, Service [properties] 18
- 23. ame.com / htt p://www.ag preprocess ... widget platform footer ... home_page new games header body ... definition User logo profile Service logic template navigation search + Award html, Service [properties] 18
- 24. Services user {“context”:{ get interface “guid”: 62623423, “siteid” : 88, “locale” : “en_US” },”guid”: 324112535} { “gid”: “324112535”, user service “username” : “-fooba-”, “avatarId” : “http://agame.com/avatars/99142.png” } check fetch filter modify { “gid”: “324112535”, “username” : “-fooba-”, profile “firstName” : “john” storage “avatarId” : 99142 }
- 25. Storage platform • API instead of SQL • Buckets of data = Key-‐Value with Schema • Abstracts storage Bucket Bucket Bucket Bucket Bucket Bucket Storage Node 1 Storage Node 2 Cache Cache Shard 1 Shard 2 Shard .. 20
- 26. Job handlers Account interface-account.register Audit Interface {“guid”:7734} Handler Audit User service-user.register Log Service {“result”:”success”,”guid”:”73324”} Welcome Profile Handler bucket rabbitmq send email 21
- 28. interface JSON Load balancer Auth User Award Profile Game in@erface interface interface widget widget Protobuf Widget PlaGorm service User User Game Award Award service service service service service Protobuf storage User Game Profiles Profiles Games Award Award bucket bucket bucket bucket bucket Storage PlaGorm 23
- 29. How we do it • Strictly define EVERYTHING • input/output format, urls, • rpc func4ons, types, valida4on • Strict rules on changes • Everything op4onal • Unless you’re really, really sure (and have proof!) • Generate code and documenta4on • Interface derived from specifica4on • Documenta4on derived from specifica4on 24
- 30. • XML: Defini4on • XSLT: Transforma4on • Piqi: • Protobuf & Json encoding/decoding • HTTP RPC (h`p://piqi.org/) 25
- 31. What we ended up with… • Small applica4ons running on their own nodes • Scale up where necessary • Many of them on the same machine • Small cluster already contains hundreds of nodes 26
- 32. Keeping it connected • Connec4ng all nodes is a bad idea: • Not needed • Number of open connec4ons • Solu4on: use hidden nodes • Each node specifies the nodes it needs. • How does it scale? 27
- 33. Hidden nodes vs. visible nodes Hidden" Visible" 160000" Number of TCP connections 145530$ 140000" 120000" 114960$ 100000" 87990$ 80000" 64620$ 60000" 44850$ 40000" 28680$ 16110$ 20000" 7140$ 1770$ 240$ 320$ 400$ 480$ 560$ 640$ 720$ 0" 2" 4" 6" 8" 10" 12" 14" 16" 18" Number of servers (~30 nodes / server) 28
- 34. Router • Each applica4on has it’s own router • Scans & Monitors • Balances requests across nodes • Parallelizes requests as much as possible • Applica4on can only talk to what it has defined 29
- 35. But what about response times? • Use proper transports (Erlang RPC vs. HTTP) • Yes there is some transport overhead • Typically < 1ms per layer (+network latency) • But each layer has predictable response 4mes • As long as enough downward resources are available 30
- 36. The old ... 31
- 37. .. and the new. 32
- 38. Closing Words
- 39. Where are we now.. • Interface layer has been defined and largely implemented • ImplemenMng the service & storage layer step by step • Meanwhile we have adapters in place • Several of the new API’s are in use in producMon • First portals on the new widget plaGorm will be deployed in the coming months 34
- 40. ... And where we’re going • Deploying to mul4ple datacenters this year • Bring elas4city to our deployments • Decommission our “old architecture” • Erlang game plahorm 35
- 41. Lessons Learned • An architectural vision is crucial • Integra4ng into a mess is a mess • Layers with clear responsibili4es • Strict communica4on • Package small • Change is the only constant • Op4mize for predictability first 36
- 42. Questions 37
- 43. Thank you! 38