Turning the web stack upside down rethinking how data flows through systems
1 like231 views
The document discusses rethinking data flow architecture in online gaming and web development, presenting an 'upside-down stack' that focuses on in-memory state management and event-driven systems instead of traditional request-driven models. It highlights the advantages of reduced database interactions, improved scalability, and the need for careful architectural design to avoid complexity. Additionally, it covers the use of static site generators and event-driven approaches to enhance performance and integration of third-party services.
Turning the web stack upside down rethinking how data flows through systems
- 1. Turning the web stack upside down Rethinking how data flows through systems By Paolo Negri
- 2. Paolo Negri CTO & co-founder, Contentful @hungryblank linkedin.com/in/panegri
- 3. Before we talk about data flows...
- 4. Let’s talk about online gaming
- 5. Traditional architecture Everything is driven by a request Game App server Persistent datastore @hungryblank
- 6. Advantages ● Simple to reason about ● Stateless app servers are very easy to operate and scale ● Data is always up to date in database Disadvantages ● Difficult to scale horizontally ● Heavy load on the database server ● Heavy load on app servers continuously loading and writing data to DB Trade-offs
- 7. We turned it upside down
- 8. ● On startup, load all game state from the database ● All state kept in memory ● Events modify the in-memory state ● Periodically snapshot the current state into the database ● If player is inactive for a while, snapshot the state and take it out of memory Gaming upside- down stack
- 9. Wooga’s upside-down stack Game Data layer App server Persistent datastore @hungryblank
- 10. Advantages ● Easier to scale ● 2-3 orders of magnitude fewer interactions with database ● Game logic entirely testable without DB attached Disadvantages ● Must use a runtime that works well with long lived state (we used erlang) ● Might experience data loss by design (one snapshot time unit) Trade-offs
- 11. ● State can live out of Database ● There’s a time and place for the runtime ● There might be a third way: Normal form, denormalized and data artifacts ● You can reduce your computation needs by a factor of 1000 if you are mindful of your computation ○ What data you process ○ Why ○ When ○ Where What we learned
- 12. But you’re probably not building games... ...and these days neither am I
- 13. Don’t let a CMS get in the way of shipping software Contentful provides a content infrastructure that enables teams to power content in any digital product @hungryblank
- 14. Contentful allows customers to create all kinds of experiences @hungryblank
- 15. More often than not, it’s a website @hungryblank
- 16. More often than not, it’s a website @hungryblank
- 17. More often than not, it’s a website @hungryblank
- 18. More often than not, it’s a website @hungryblank
- 19. More often than not, it’s a website @hungryblank
- 20. More often than not, it’s a website @hungryblank
- 21. Traditional web architecture Everything is driven by a request Browser App server Database @hungryblank
- 22. Is there an upside-down stack for the web?
- 24. Static site generators Browser Static site host Build server SCM Developers Editors @hungryblank
- 25. Events in a static site generator Browser Static site host Build server SCM Developers Editors Request Build completed Code changed @hungryblank
- 26. Advantages ● Trivial to scale for high traffic ● Very reliable: operations are simple ● Failure results in stale content, not downtime ● Small security surface Disadvantages ● Less intuitive editing experience for editors/marketeers ● Every update similar to a software release ● Latency until new content appears depends on build process Trade-offs
- 27. Static site generators with services Browser Static site host Build server SCM Developers Editors AWS Lambda Algolia SendGrid Content infrastructure StripeCommerce layer @hungryblank
- 28. Events in a static site generator Browser Static site host Build server SCM Developers Editors AWS Lambda Algolia Commerce layer SendGrid Content infrastructure Stripe User request Build complete Content changed Code changed @hungryblank
- 29. Static sites are an upside-down stack for the web
- 30. Events drive the system Requests are just another event
- 31. Most of the work happens when code or data changes
- 32. Work in a static site generator Browser Static site host Build server SCM Developers Editors AWS Lambda Algolia SendGrid Content infrastructure Stripe Most work here Commerce layer @hungryblank
- 33. User requests are cheap
- 34. Events in a static site generator Browser Static site host Build server SCM Developers Editors AWS Lambda Algolia SendGrid Content infrastructure StripeCommerce layer @hungryblank
- 35. Advantages ● Easy to scale ● Easy to integrate third party services ● Easy to gracefully degrade ● Better general performance Disadvantages ● Requires thinking out of the request/response box ● If not architected carefully at risk of exploding complexity ● It’s a distributed system and requires understanding of the topic Trade-offs
- 37. O N E Build driven or reactive? @hungryblank
- 38. ● Must classify how you want to handle events ○ Events that build artifacts ○ Events that trigger instant reaction (login, ecommerce purchase) ● Writing applications in both paradigms might be needed, but not a must O N E @hungryblank
- 39. T W O Do work only on meaningful events @hungryblank
- 40. ● Take advantage of pre-built artifacts ● No more “current state” rebuild ● Events build their new state into artifacts ● Artifacts are used in place of what is currently computation T W O @hungryblank
- 41. T H R E E Data = code (They move through the system in the same way) @hungryblank
- 42. T H R E E ● In this pattern, data and code follow the same lifecycle ● Every update in data or code is very much like a deployment ● Continuous delivery becomes quite literally continuous @hungryblank
- 43. F O U R Your domain model needs strong tooling @hungryblank
- 44. F O U R ● In this pattern, the architecture is fundamentally distributed, so you need strong tooling tomode events and data. ● Events and data model can be directly mapped to domain-driven design. @hungryblank
- 45. F O U R ● The GraphQL ecosystem is a very promising solution to all this, as it helps in defining clear contracts across services (own or third party). @hungryblank
- 46. 1. Reframe: Request/Response as events 2. Investigate: What are events that happen outside of request/response? Example: Deployments are events 3. Document all the above 4. Extract: Move computation to meaningful events Example: Re-build data during an external event — easiest during deployments How do I even start?