Data Patterns- Andrew Jones (By ThoughtWorks)
0 likes766 views
The document discusses migrating from monolithic data architectures to more modern architectures. It outlines some benefits of modern architectures like evolvability, scalability, and reliability. It then describes characteristics of a modern architecture like using asynchronous events, immutable log-based streaming, and microservices. It also discusses patterns for a modern data architecture like unbundling databases, using event seams, and designing for auditability and enrichment.
Data Patterns- Andrew Jones (By ThoughtWorks)
- 1. DATA PATTERNS: MIGRATING FROM MONOLITH TO MODERN ARCHITECTURES Andrew Jones
- 2. Why would we want to move away from monoliths?
- 3. Slow to change Cross team dependencies Complex integration Legacy patterns Slow to new produce insights Entangled data
- 4. Data is the lifeblood of your organisation - systems work with it, analysts work with it, and everyone needed it yesterday
- 5. What do we want out of a modern data architecture?
- 6. Evolvability
- 10. So, what does a modern data architecture actually look like?
- 13. Microservices
- 14. Asynchronous events + Immutable log-based streaming + Microservices
- 16. The Data Patterns Catalogue
- 25. Using event seams to extract capabilities
- 26. Batch to event adapter
- 33. Event to batch adapter
- 41. CREATE STREAM pageviews_enriched AS SELECT pv.viewtime, pv.userid AS userid, pv.pageid, pv.timestring, u.gender, u.regionid, u.interests, u.contactinfo FROM pageviews_transformed pv LEFT JOIN users_5part u ON pv.userid = u.userid;
- 42. No single tool to rule them all
- 43. Trust but verify
- 44. Reprocessing
- 45. Data patterns catalogue - written version coming soon
- 46. Data Engineering Melbourne Meetup Wednesday 14th November, 6pm ThoughtWorks office https://www.meetup.com/Data-Engineering-Melbourne Want to talk more about data architecture and engineering?