Real World Big Data, 100,000 transactions a second in a NoSQL Database

Achieving 100,000
Transactions Per Second
with a NoSQL Database

Eric David Bloch
@eedeebee

23 May 2012

A bit about me

• I’ve written software used by millions of people.

Apps, libraries, compilers, device drivers, operating systems

• This is my third Gluecon and my first talk

• I’m the Community Director at MarkLogic, last 2 years.

• I survived having 3 kids in less than 2 years.

Storyline

• Why?
• How?
• Whirl-wind tour of database architecture
• Techniques to get to 100K

 It’s about money.

 Top 5 bank needed to manage trades
 Trades look more like documents than tables
 Schemas for trades change all the time
 Transactions
 Scale and velocity (“Big Data”)

Trades and Positions

 1 million trades per day
 Followed by 1 million position reports at end of day
 Roll up trades of current date for each “book, instrument” pair
 Group-by, with key = “date, book, instrument”

1M positions
1M trades

Day Day Day Day Day ...
1 2 3 4 5


<trade>
<quantity>8540882</quantity>
<quantity2>1193.71</quantity2>
<instrument>WASAX</instrument>
<book>679</book>
<trade-date>2011-03-13-07:00</trade-date>
<settle-date>2011-03-17-07:00</settle-date>
</trade>

<position>
<instrument>EAAFX</instrument>
<book>679</book>
<quantity>3</quantity>
<business-date>2011-03-25Z</business-date>
<position-date>2011-03-24Z</position-date>
</position>

Requirements

NoSQL flexibility,
performance & scale

Enterprise-grade
transactional guarantees

Now show us

• 15K inserts per second
• Linear scalability

What NoSQL Database out there can we use?

A quote from the bank
“We threw everything we had at MarkLogic and it didn’t break a sweat”

We weren’t content with 15K

So we showed them…

Slide 13 Copyright © 2012 MarkLogic® Corporation. All rights reserved.

What is MarkLogic?

 Non-relational, document-oriented, distributed
database
 Shared nothing clustering, linear scale out
 Multi-Version Concurrency Control (MVCC)
 Transactions (ACID)

 Search Engine
 Web scale (Big Data)
 Inverted indexes (term lists)
 Real-time updates
 Compose-able queries


Architecture

 Data model
 Indexing
 Clustering
 Query execution
 Transactions


Key (URI) Value (Document)

/trade/153748994 <trade>
<id>8</id>
<time>2012-02-20T14:00:00</time>
<instrument>BYME AAA</instrument>
<price cur=“usd”>600.27</price>
</trade>

/user/eedeebee {
“name” : “Eric Bloch”,
“age” : 47,
“hair” : “gray”,
“kids” : [ “Grace”, “Ryan”, “Owen” ]
}

/book5293 It was the best of times, it was the worst of times, it
was the age of wisdom, ...

/2012-02-20T14:47:53/01445 .mp3
.avi
[your favorite binary format]

Inverted Index

“which” 123, 127, 129, 152, 344, 791 . . .

“uniquely” 122, 125, 126, 129, 130, 167 . . .

“identify” 123, 126, 130, 142, 143, 167 . . .

“each” 123, 130, 131, 135, 162, 177 . . . Document
“uniquely identify”
126, 130, 167, 212, 219, 377 . . . References
<article> ...
126, 130, 167, 212, 219, 377 . . .
article/abstract/@author ...

<product>IMS</product>


Range Index

<trade>
<trader_id>8</trader_id> Value Docid
<time>2012-02-20T14:00:00</time>
<instrument>IBM</instrument> 0 287
…
</trade> 8 1129
13 531 Docid Value
<trade>
<trader_id>13</trader_id> … … 287 0
<time>2012-02-20T14:30:00</time>
Rows <instrument>AAPL</instrument> … … 531 13
… 1129 8
</trade>
… …
<trade>
<trader_id>0</trader_id> … …
<time>2012-02-20T15:30:00</time>
<instrument>GOOG</instrument> • Column Oriented
… • Memory Mapped
</trade>


Shared-Nothing Clustering

Host D1 Host D2 Host D3 … Host Dj


Query Evaluation – “Map”

Q
Host E1 Host E2 Host E3 … Host Ei

Host D1 Host D2 Host D3 Host Dj
…

Q Q Q Q
F1 … Fn F1 … Fn F1 … Fn F1 … Fn

Query Evaluation – “Reduce”

Top
10
Host E1 Host E2 Host E3 … Host Ei

Host D1 Host D2 Host D3 Host Dj
…
Top Top Top Top
10 10 10 10
F1 … Fn F1 … Fn F1 … Fn F1 … Fn

Queries/Updates with MVCC

 Every query has a timestamp
 Documents do not change

 Reads are lock-free
 Inserts – see next slide
 Deletes – mark as deleted
 Edits –
 copy
 edit
 insert the copy
 mark the original as deleted


Insert Mechanics

1) New URI+Document arrive at E-node
2) URI Probe – determine whether URI exists in any forest
3) URI Lock – write locks is taken on D node(s)
4) Forest Assignment – URI is deterministically placed in Forest
5) Indexing
6) Journaling
7) Commit – transaction complete
8) Release URI Locks – D node(s) are notified to release lock


Save-and-merge
(Log Stuctured Tree Merge)

Database
Insert/
Update

Forest 1 Forest 2
S0
Memory Save

Disk
Journaled
S1 S2 S1 S2
S3 S3

Merge
J



 1 million trades per day
 followed by 1 million position reports at end of day
 Roll up trades of the current “date” for each “book:instrument” pair
 Group-by, with key = “book:date:instrument”

1M positions
1M trades

Day Day Day Day Day ...
1 2 3 4 5


Naive Query Pseudocode

for each book
for each instrument in that book
position = position(yesterday, book, instrument)
for each trade of that instrument in this book
position += trade(today, book, instrument).quantity
insert(today, book, instrument, position)


Initial results

Single node – 19,000 inserts per second


Initial results with cluster

70000

60000

50000

40000
doc/sec

30000 Report Query 2

20000

10000

0
1DE 2DE 3DE
# of nodes


Techniques to get to 100K

 Query for Computing New Positions
 Materialized compound key, Co-Occurrence Query and Aggregation
 Insert of New Positions
 Batching
 In-Forest Eval


Materializing a compound key

<trade>
<book>679</book>
</trade>


Materializing a compound key

<trade>
<roll-up book-date-instrument=“151333445566782303”/>
<book>679</book>
</trade>


Co-Occurrence and
Distributed Aggregation
<trade>
<roll-up
book-date-instrument=“151373445566703”/> V D
D V
<quantity>8540882</quantity> … …
<instrument>WASAX</instrument> … …
… 1129
<book>679</book> … …
<trade-date>2011-03-13-07:00</trade-date> … …
15137… …
… …
</trade> … …
… …
… …

• Co-occurrences:
V D
Find pairings of range indexed values D V
… …
• Aggregate on the D nodes (Map/Reduce):
8540882
… 1129 …
Sum up the quantities above
… …
… …
… 8540882
• Similar to a Group-by, … …
• in a column-oriented, in-memory … … … …
database … …


Initial results with new query

> 30K inserts/second on a single node


Co-Occurrence + Aggregate versus
Naïve approach

70000

60000

50000

40000
doc/sec

Report Query 3
30000
Report Query 2

20000

10000

0
1DE 2DE 3DE
# of nodes


Techniques

 Computing Positions
 Materialized compound key, Co-Occurrence Query and Aggregation
 Updates
 Batching
 In-Forest Eval


Transaction Size and
Throughput

35000

30000

25000
#docs per sec.

20000

15000 insert throughput

10000

5000

0
1 2 4 8 16 32 100 500 1000 10000
# doc inserts per transaction


Techniques

 Computing Positions (Data Warehouse)
 Hash Key Decoration
 Co-Occurrence
 Updates (Transaction)
 Batching
 In-Forest Eval


Insert Mechanics

2) *URI Probe – determine whether URI exists in any forest
3) *URI Lock – write locks are created
5) Indexing
6) Journaling
8) *Release URI Locks – D nodes are notified to release lock

* Overhead of these operations increases with cluster size


Deterministic Placement


Hash 64-bit Bucketed Fi
URI
function number into a Forest

• Done in C++ within server
• But…
• Can also be done in the client
• Server allows queries to be evaluated against only one
forest…


In-Forest Eval

Q
E

D1 D2

Q
F1 F2 F3 F4


In-Forest Eval Insert Mechanics

a) Compute Fi using
b) Ask server to evaluated the insert query only against Fi
2) URI Probe – Fi Only
3) URI Lock – Fi Only
4) Forest Assignment – Fi Only
5) Indexing
6) Journaling
8) Lock Release - Fi Only


Regular Insert Vs. In-Forest Eval

70000

60000

50000
docs/second

40000

regular insert
30000
in-forest-eval
In-Forest Eval

20000

10000

0
1DE 2DE 3DE
# of nodes


Tada! Achieving 100K Updates

In-Forest Eval


NoSQL document-oriented database
with real-time full-text search and transactions
Doing 100K transactions per second

(Yes that’s marker felt)

Thank You!

@eedeebee
eric.bloch@marklogic.com


Real World Big Data, 100,000 transactions a second in a NoSQL Database

More Related Content

Viewers also liked (20)

Similar to Real World Big Data, 100,000 transactions a second in a NoSQL Database (20)

Recently uploaded (20)

Real World Big Data, 100,000 transactions a second in a NoSQL Database

Editor's Notes