Beyond The Data Grid: Coherence, Normalisation, Joins and Linear Scalability
11 likes8,513 views
The document introduces the ODC, a highly distributed, normalized in-memory data store designed to enhance scalability and performance by adapting the snowflake schema for efficient data access. It critiques traditional RDBMS architectures as legacy technologies, advocating for a new approach that emphasizes normalizing data to avoid the overhead of denormalization and connect only necessary dimensions through a 'connected replication' pattern. The ODC aims to resolve the distributed join problem, enabling fast queries without the typical complexities associated with traditional database structures.
![Stage 1: Get the right keys to query
the Facts
Select Transaction, MTM, ReferenceData From
MTM, Transaction, Ref Where Cost Centre = ‘CC1’
LBs[]=getLedgerBooksFor(CC1)
SBs[]=getSourceBooksFor(LBs[])
So we have all the bottom level
dimensions needed to query facts
Transactions
Mtms
Cashflows
Partitioned](https://image.slidesharecdn.com/odc-goingbeyondthedatagrid-qcon-110409030404-phpapp01/85/Beyond-The-Data-Grid-Coherence-Normalisation-Joins-and-Linear-Scalability-65-320.jpg)
![Stage 2: Cluster Join to get Facts
Select Transaction, MTM, ReferenceData From
MTM, Transaction, Ref Where Cost Centre = ‘CC1’
LBs[]=getLedgerBooksFor(CC1)
SBs[]=getSourceBooksFor(LBs[])
So we have all the bottom level
dimensions needed to query facts
Transactions
Get all Transactions and
Mtms
MTMs (cluster side join) for
the passed Source Books
Cashflows
Partitioned](https://image.slidesharecdn.com/odc-goingbeyondthedatagrid-qcon-110409030404-phpapp01/85/Beyond-The-Data-Grid-Coherence-Normalisation-Joins-and-Linear-Scalability-66-320.jpg)
![Stage 3: Augment raw Facts with
relevant Dimensions
Select Transaction, MTM, ReferenceData From
MTM, Transaction, Ref Where Cost Centre = ‘CC1’
Populate raw facts LBs[]=getLedgerBooksFor(CC1)
(Transactions) with SBs[]=getSourceBooksFor(LBs[])
dimension data
So we have all the bottom level
before returning to
dimensions needed to query facts
client.
Transactions
Get all Transactions and
Mtms
MTMs (cluster side join) for
the passed Source Books
Cashflows
Partitioned](https://image.slidesharecdn.com/odc-goingbeyondthedatagrid-qcon-110409030404-phpapp01/85/Beyond-The-Data-Grid-Coherence-Normalisation-Joins-and-Linear-Scalability-68-320.jpg)
Beyond The Data Grid: Coherence, Normalisation, Joins and Linear Scalability
- 1. ODC Beyond The Data Grid: Coherence, Normalisation, Joins and Linear Scalability Ben Stopford : RBS
- 2. The Story… Industry and academia We introduce ODC a have responded with a The internet era has NoSQL store with a variety of solutions that moved us away from unique mechanism leverage distribution, use traditional database for efficiently of a simpler contract and architecture, now a managing normalised RAM storage. quarter of a century data. old. We show how we The result is a highly Finally we introduce the adapt the concept of scalable, in-memory ‘Connected Replication’ a Snowflake Schema data store that can pattern as mechanism to aid the application support both millisecond for making the star of replication and queries and high schema practical for in partitioning and avoid bandwidth exports over memory architectures. problems with a normalised object distributed joins. model.
- 3. Database Architecture is Old Most modern databases still follow a 1970s architecture (for example IBM’s System R)
- 4. “Because RDBMSs can be beaten by more than an order of magnitude on the standard OLTP benchmark, then there is no market where they are competitive. As such, they should be considered as legacy technology more than a quarter of a century in age, for which a complete redesign and re-architecting is the appropriate next step.” Michael Stonebraker (Creator of Ingres and Postgres)
- 5. What steps have we taken to improve the performance of this original architecture?
- 6. Improving Database Performance (1)" Shared Disk Architecture Shared Disk
- 7. Improving Database Performance (2)" Shared Nothing Architecture Shared Nothing
- 8. Improving Database Performance (3)" In Memory Databases
- 9. Improving Database Performance (4)" Distributed In Memory (Shared Nothing)
- 10. Improving Database Performance (5) " Distributed Caching Distributed Cache
- 11. These approaches are converging Shared Nothing Shared Nothing Regular Teradata, Vertica, NoSQL… (memory) Database VoltDB, Hstore Oracle, Sybase, MySql ODC Distributed Caching Coherence, Gemfire, Gigaspaces
- 12. So how can we make a data store go even faster? Distributed Architecture Drop ACID: Simplify the Contract. Drop disk
- 13. (1) Distribution for Scalability: The Shared Nothing Architecture • Originated in 1990 (Gamma DB) but popularised by Teradata / BigTable / NoSQL • Massive storage potential • Massive scalability of processing • Commodity hardware • Limited by cross partition Autonomous processing unit joins for a data subset
- 14. (2) Simplifying the Contract • For many users ACID is overkill. • Implementing ACID in a distributed architecture has a significant affect on performance. • NoSQL Movement: CouchDB, MongoDB,10gen, Basho, CouchOne, Cloudant, Cloudera, GoGrid, InfiniteGraph, Membase, Riptano, Scality….
- 15. Databases have huge operational overheads Taken from “OLTP Through the Looking Glass, and What We Found There” Harizopoulos et al
- 16. (3) Memory is 100x faster than disk ms μs ns ps 1MB Disk/Network 1MB Main Memory 0.000,000,000,000 Cross Continental Main Memory L1 Cache Ref Round Trip Ref Cross Network L2 Cache Ref Round Trip * L1 ref is about 2 clock cycles or 0.7ns. This is the time it takes light to travel 20cm
- 17. Avoid all that overhead RAM means: • No IO • Single Threaded ⇒ No locking / latching • Rapid aggregation etc • Query plans become less important
- 18. We were keen to leverage these three factors in building the ODC Simplify the Memory Distribution contract Only
- 19. What is the ODC? Highly distributed, in memory, normalised data store designed for scalable data access and processing.
- 20. The Concept Originating from Scott Marcar’s concept of a central brain within the bank: “The copying of data lies at the route of many of the bank’s problems. By supplying a single real-time view that all systems can interface with we remove the need for reconciliation and promote the concept of truly shared services” - Scott Marcar (Head of Risk and Finance Technology)
- 21. This is quite tricky problem High Bandwidth Access to Lots of Data Low Latency Access to Scalability to small lots of users amounts of data
- 22. ODC Data Grid: Highly Distributed Physical Architecture In-memory Lots of parallel storage processing Oracle Coherence Messaging (Topic Based) as a system of record (persistence)
- 23. The Layers" Access Layer Java " Java client client API API Query Layer Transactions Data Layer Mtms Cashflows Persistence Layer
- 24. But unlike most caches the ODC is Normalised
- 25. Three Tools of Distributed Data Architecture Indexing Partitioning Replication
- 26. For speed, replication is best Wherever you go the data will be there But your storage is limited by the memory on a node
- 27. For scalability, partitioning is best Keys Aa-Ap Keys Fs-Fz Keys Xa-Yd Scalable storage, bandwidth and processing
- 28. Traditional Distributed Caching Approach Keys Aa-Ap Keys Fs-Fz Keys Xa-Yd Trader Big Denormliased Party Objects are spread Trade across a distributed cache
- 29. But we believe a data store needs to be more than this: it needs to be normalised!
- 30. So why is that? Surely denormalisation is going to be faster?
- 31. Denormalisation means replicating parts of your object model
- 32. …and that means managing consistency over lots of copies
- 33. … as parts of the object graph will be copied multiple times Periphery objects that are Trader Party denormalised onto core objects Trade will be duplicated multiple times across the data grid. Party A
- 34. …and all the duplication means you run out of space really quickly
- 35. Spaces issues are exaggerated further when data is versioned Trader Party Version 1 Trade Trader Party Version 2 Trade Trader Party Version 3 Trade Trader Party Version 4 Trade …and you need versioning to do MVCC
- 36. And reconstituting a previous time slice becomes very difficult. Trade Trader Party Party Trader Trade Party Trader Trade Party
- 37. Why Normalisation? Easy to change data (no distributed locks / transactions) Better use of memory. Facilitates Versioning And MVCC/Bi-temporal
- 38. OK, OK, lets normalise our data then. What does that mean?
- 39. We decompose our domain model and hold each object separately
- 40. This means the object graph will be split across multiple machines. Trader Party Trade Trade Trader Party
- 41. Binding them back together involves a “distributed join” => Lots of network hops Trader Party Trade Trade Trader Party
- 42. It’s going to be slow…
- 43. Whereas the denormalised model the join is already done
- 44. Hence Denormalisation is FAST!" (for reads)
- 45. So what we want is the advantages of a normalised store at the speed of a denormalised one! " " " This is what the ODC is all about!
- 46. Looking more closely: Why does normalisation mean we have to be spread data around the cluster. Why can’t we hold it all together?
- 47. It’s all about the keys
- 48. We can collocate data with common keys but if they crosscut the only way to collocate is to replicate Crosscutting Keys Common Keys
- 49. We tackle this problem with a hybrid model: Denormalised Trader Party Trade Normalised
- 50. We adapt the concept of a Snowflake Schema.
- 51. Taking the concept of Facts and Dimensions
- 52. Everything starts from a Core Fact (Trades for us)
- 53. Facts are Big, dimensions are small
- 54. Facts have one key
- 55. Dimensions have many" (crosscutting) keys
- 56. Looking at the data: Valuation Legs Valuations art Transaction Mapping Cashflow Mapping Facts: Party Alias Transaction =>Big, Cashflows common Legs Parties keys Ledger Book Source Book Dimensions Cost Centre Product =>Small, Risk Organisation Unit Business Unit crosscutting HCS Entity Keys Set of Books 0 37,500,000 75,000,000 112,500,000 150,000,000
- 57. We remember we are a grid. We should avoid the distributed join.
- 58. … so we only want to ‘join’ data that is in the same process Coherence’s KeyAssociation gives us this Trades MTMs Common Key
- 59. So we prescribe different physical storage for Facts and Dimensions Replicated Trader Party Trade Partitioned (Key association ensures joins are in process)
- 60. Facts are held distributed, Dimensions are replicated Valuation Legs Valuations art Transaction Mapping Cashflow Mapping Facts: Party Alias Transaction =>Big Cashflows =>Distribute Legs Parties Ledger Book Source Book Dimensions Cost Centre Product =>Small Risk Organisation Unit Business Unit => Replicate HCS Entity Set of Books 0 37,500,000 75,000,000 112,500,000 150,000,000
- 61. - Facts are partitioned across the data layer" - Dimensions are replicated across the Query Layer Query Layer Trader Party Trade Transactions Data Layer Mtms Cashflows Fact Storage (Partitioned)
- 62. Key Point We use a variant on a Snowflake Schema to partition big stuff, that has the same key and replicate small stuff that has crosscutting keys.
- 63. So how does they help us to run queries without distributed joins? Select Transaction, MTM, ReferenceData From MTM, Transaction, Ref Where Cost Centre = ‘CC1’ This query involves: • Joins between Dimensions: to evaluate where clause • Joins between Facts: Transaction joins to MTM • Joins between all facts and dimensions needed to construct return result
- 64. Stage 1: Focus on the where clause:" Where Cost Centre = ‘CC1’
- 65. Stage 1: Get the right keys to query the Facts Select Transaction, MTM, ReferenceData From MTM, Transaction, Ref Where Cost Centre = ‘CC1’ LBs[]=getLedgerBooksFor(CC1) SBs[]=getSourceBooksFor(LBs[]) So we have all the bottom level dimensions needed to query facts Transactions Mtms Cashflows Partitioned
- 66. Stage 2: Cluster Join to get Facts Select Transaction, MTM, ReferenceData From MTM, Transaction, Ref Where Cost Centre = ‘CC1’ LBs[]=getLedgerBooksFor(CC1) SBs[]=getSourceBooksFor(LBs[]) So we have all the bottom level dimensions needed to query facts Transactions Get all Transactions and Mtms MTMs (cluster side join) for the passed Source Books Cashflows Partitioned
- 67. Stage 2: Join the facts together efficiently as we know they are collocated
- 68. Stage 3: Augment raw Facts with relevant Dimensions Select Transaction, MTM, ReferenceData From MTM, Transaction, Ref Where Cost Centre = ‘CC1’ Populate raw facts LBs[]=getLedgerBooksFor(CC1) (Transactions) with SBs[]=getSourceBooksFor(LBs[]) dimension data So we have all the bottom level before returning to dimensions needed to query facts client. Transactions Get all Transactions and Mtms MTMs (cluster side join) for the passed Source Books Cashflows Partitioned
- 69. Stage 3: Bind relevant dimensions to the result
- 70. Bringing it together: Replicated Partitioned Java client Dimensions Facts API We never have to do a distributed join!
- 71. Coherence Voodoo: Joining Distributed Facts across the Cluster Related Trades and MTMs (Facts) are collocated on the same machine with Key Affinity. Trades MTMs Aggregator Direct backing map access must be used due to threading http://www.benstopford.com/ issues in Coherence 2009/11/20/how-to-perform-efficient- cross-cache-joins-in-coherence/
- 72. So we are normalised And we can join without extra network hops
- 73. We get to do this… Trader Party Trade Trade Trader Party
- 74. …and this… Trader Party Version 1 Trade Trader Party Version 2 Trade Trader Party Version 3 Trade Trader Party Version 4 Trade
- 75. ..and this.. Trade Trader Party Party Trader Trade Party Trader Trade Party
- 76. …without the problems of this…
- 77. …or this..
- 78. ..all at the speed of this… well almost!
- 80. But there is a fly in the ointment…
- 81. I lied earlier. These aren’t all Facts. Valuation Legs Valuations rt Transaction Mapping Cashflow Mapping Party Alias Facts Transaction Cashflows Legs Parties This is a dimension Ledger Book • It has a different Source Book Cost Centre key to the Facts. Dimensions Product • And it’s BIG Risk Organisation Unit Business Unit HCS Entity Set of Books 0 125,000,000
- 82. We can’t replicate really big stuff… we’ll run out of space" => Big Dimensions are a problem.
- 83. Fortunately we found a simple solution!
- 84. We noticed that whilst there are lots of these big dimensions, we didn’t actually use a lot of them. They are not all “connected”.
- 85. If there are no Trades for Barclays in the data store then a Trade Query will never need the Barclays Counterparty
- 86. Looking at the All Dimension Data some are quite large Party Alias Parties Ledger Book Source Book Cost Centre Product Risk Organisation Unit Business Unit HCS Entity Set of Books 0 1,250,000 2,500,000 3,750,000 5,000,000
- 87. But Connected Dimension Data is tiny by comparison Party Alias Parties Ledger Book Source Book Cost Centre Product Risk Organisation Unit Business Unit HCS Entity Set of Books 20 1,250,015 2,500,010 3,750,005 5,000,000
- 88. So we only replicate ‘Connected’ or ‘Used’ dimensions
- 89. As data is written to the data store we keep our ‘Connected Caches’ up to date Processing Layer Dimension Caches (Replicated) Transactions Data Layer As new Facts are added Mtms relevant Dimensions that they reference are moved Cashflows to processing layer caches Fact Storage (Partitioned)
- 90. Coherence Voodoo: ‘Connected Replication’
- 91. The Replicated Layer is updated by recursing through the arcs on the domain model when facts change
- 92. Saving a trade causes all it’s 1st level references to be triggered Query Layer Save Trade (With connected dimension Caches) Data Layer Cache Trade (All Normalised) Store Partitioned Trigger Source Cache Party Ccy Alias Book
- 93. This updates the connected caches Query Layer (With connected dimension Caches) Data Layer Trade (All Normalised) Party Source Ccy Alias Book
- 94. The process recurses through the object graph Query Layer (With connected dimension Caches) Data Layer Trade (All Normalised) Party Source Ccy Alias Book Party Ledger Book
- 95. ‘Connected Replication’ A simple pattern which recurses through the foreign keys in the domain model, ensuring only ‘Connected’ dimensions are replicated
- 96. Limitations of this approach • Data set size. Size of connected dimensions limits scalability. • Joins are only supported between “Facts” that can share a partitioning key (But any dimension join can be supported)
- 97. Performance is very sensitive to serialisation costs: Avoid with POF Deserialise just one field from the object stream Binary Value Integer ID
- 98. Other cool stuff (very briefly)
- 99. Everything is Java Java client Java schema API Java ‘Stored Procedures’ and ‘Triggers’
- 100. Messaging as a System of Record" " ODC provides a realtime Access Layer Java Java client! API! view over any part of the client! dataset as messaging is the API! used as the system of record. Processing Layer Transactions Messaging provides a more Data Layer Mtms scalable system of record Cashflows than a database would. Persistence Layer
- 101. Being event based changes the programming model. The system provides both real time and query based views on the data. The two are linked using versioning Replication to DR, DB, fact aggregation
- 102. API – Queries utilise a fluent interface
- 103. Performance Query with more than twenty joins conditions: 2GB per min / 250Mb/s 3ms latency (per client)
- 104. Conclusion Data warehousing, OLTP and Distributed caching fields are all converging on in- memory architectures to get away from disk induced latencies.
- 105. Conclusion Shared nothing architectures are always subject to the distributed join problem if they are to retain a degree of normalisation.
- 106. Conclusion We present a novel mechanism for avoiding the distributed join problem by using a Star Schema to define whether data should be replicated or partitioned. Partitioned Storage
- 107. Conclusion We make the pattern applicable to ‘real’ data models by only replicating objects that are actually used: the Connected Replication pattern.
- 108. The End • Further details online http://www.benstopford.com (linked from my Qcon bio) • A big thanks to the team in both India and the UK who built this thing. • Questions?
Editor's Notes
- #38: This avoids distributed transactions that would be needed when updating dimensions in a denormalised model. It also reduces memory consumption which can become prohibitive in MVCC systems. This then facilitates a storage topology that eradicates cross-machine joins (joins between data sets held on different machines which require that entire key sets are shipped to perform the join operation).
- #50: Big data sets are held distributed and only joined on the grid to collocated objects.Small data sets are held in replicated caches so they can be joined in process (only ‘active’ data is held)
- #60: Big data sets are held distributed and only joined on the grid to collocated objects.Small data sets are held in replicated caches so they can be joined in process (only ‘active’ data is held)