Webinar: How Banks Use MongoDB as a Tick Database

How Capital Markets Firms Use
MongoDB as a Tick Database

Matt Kalan, Sr. Solution Architect
Email: Matt.kalan@10gen.com
Twitter: @matthewkalan

Agenda

• MongoDB Introduction
• FS Use Cases
• Writing/Capturing Market Data
• Reading/Analyzing Market Data
• Performance, Scalability, & High Availability
• Q&A

2

Introduction

10gen is the company behind MongoDB –
the leading next generation database

Document- General Open-
Oriented Purpose Source

3

10gen Overview

200+ employees 500+ customers

Offices in New York, Palo Alto, Washington
Over $81 million in funding DC, London, Dublin, Barcelona and Sydney

4

Database Landscape
• No Automatic Joins
• Document Transactions
• Fast, Scalable Read/Writes

5

MongoDB Business Benefits

Increased Developer Productivity Better Customer Experience

Faster Time to Market Lower TCO

6

MongoDB Technical Benefits

Application
Agile &
Flexible
{ author: “roger”,
High Highly date: new Date(),
text: “Spirited Away”,
Performance Available tags: [“Tezuka”, “Manga”]}
-Indexes -Replica Sets
-RAM

Horizontally Scalable
-Sharding

7

Most Common FS Use Cases

1. Tick Data Capture & Analysis
2. Reference Data Management
3. Risk Analysis & Reporting
4. Trade Repository
5. Portfolio Reporting

8

Tick Data Capture & Analysis -
Requirements
• Capture real-time market data (multi-asset, top of
book, depth of book, even news)
• Load historical data
• Aggregate data into bars, daily, monthly intervals
• Enable queries & analysis on raw ticks or
aggregates
• Drive backtesting or automated signals

9

Tick Data Capture & Analysis –
Why MongoDB?
• High throughput => can capture real-time feeds for all
products/asset classes needed
• High scalability => all data and depth for all historical time periods
can be captured
• Flexible & Range-based indexing => fast querying on time ranges
and any fields
• Aggregation Framework => can shape raw data into aggregates
(e.g. ticks to bars)
• Map-reduce capability (Native MR or Hadoop Connector) => batch
analysis looking for patterns and opportunities
• Easy to use => native language drivers and JSON expressions that
you can apply for most operational database needs as well
• Low TCO => Low software license cost and commodity hardware
10

High Level Trading Architecture

Market Data Capturing
Feed Handler
Application

News & social
networking
sources
Cached Static &
Orders Aggregated Data
Exchanges/Mark Low Latency
ets/Brokers Applications

Trades/metrics

Orders Higher Latency Backtesting and
Trading Analysis
Applications Applications

12

High Level Trading Architecture

Market Data Capturing
Feed Handler
Application

News & social
networking
Data Types sources
• Top of book Cached Static &
Orders • Depth of book Aggregated Data
Exchanges/Mark • Low Latency
Multi-asset
ets/Brokers Applications
• Derivatives (e.g. strips)
• News (text, video)
Trades/metrics
• Social Networking

Orders Higher Latency Backtesting and
Trading Analysis
Applications Applications

13

Top of book [e.g. equities]
{
_id : ObjectId("4e2e3f92268cdda473b628f6"),
symbol : "DIS",
timestamp: ISODate("2013-02-15 10:00"),
bidPrice: 55.37,
offerPrice: 55.58,
bidQuantity: 500,
offerQuantity: 700
}

> db.ticks.find( {symbol: "DIS",
bidPrice: {$gt: 55.36} } )

14

Depth of book
{
symbol : "DIS",
bidPrices: [55.37, 55.36, 55.35],
offerPrices: [55.58, 55.59, 55.60],
bidQuantities: [500, 1000, 2000],
offerQuantities: [1000, 2000, 3000]
}

> db.ticks.find( {bidPrices: {$gt: 55.36} } )

15

or any way your app uses it
{
symbol : "DIS",
bids: [
{price: 55.37, amount: 500},
{price: 55.37, amount: 1000},
{price: 55.37, amount: 2000} ],
offers: [
{price: 55.58, amount: 1000},
{price: 55.58, amount: 2000},
{price: 55.59, amount: 3000} ]
}
> db.ticks.find( {"bids.price": {$gt: 55.36} } )

16

Synthetic spreads
{
symbol : "DIS",
spreadPrice: 0.58
leg1: {symbol: “CLM13, price: 97.34}
leg2: {symbol: “CLK13, price: 96.92}
}
db.ticks.find( { “leg1” : “CLM13” },
{ “leg2” : “CLK13” },
{ “spreadPrice” : {$gt: 0.50 } } )

17

News
{
symbol : "DIS",
title: “Disney Earnings…”
body: “Walt Disney Company reported…”,
tags: [“earnings”, “media”, “walt disney”]
}

18

Social networking
{
twitterHandle: “jdoe”,
tweet: “Heard @DisneyPictures is releasing…”,
usernamesIncluded: [“DisneyPictures”],
hashTags: [“movierumors”, “disney”]
}

19

Aggregates (bars, daily, etc.)
{
symbol : "DIS”,
openTS: Date("2013-02-15 10:00"),
closeTS: Date("2013-02-15 10:05"),
open: 55.36,
high: 55.80,
low: 55.20,
close: 55.70
}

20

Architecture for Querying Data

Research &
Analysis
• Ticks Applications

• Bars
• Other analysis

Backtesting
Applications

Higher Latency
Trading
Applications

22

Index any fields: arrays, nested, etc

// Compound indexes
> db.ticks.ensureIndex({symbol: 1, timestamp:1})

// Index on arrays
>db.ticks.ensureIndex( {bidPrices: -1})

// Index on any depth
> db.ticks.ensureIndex( {“bids.price”: 1} )

// Full text search
> db.ticks.ensureIndex ( {tweet: “text”} )

23

Query for ticks by time; price
threshold
// Ticks for last month for media companies
> db.ticks.find({
symbol: {$in: ["DIS", “VIA“, “CBS"]},
timestamp: {$gt: new ISODate("2013-01-01")},
timestamp: {$lte: new ISODate("2013-01-31")}})

// Ticks when Disney’s bid breached 55.50 this month
> db.ticks.find({
symbol: "DIS",
bidPrice: {$gt: 55.50},
timestamp: {$gt: new ISODate("2013-02-01")}})

24

Analyzing/Aggregating Options

• Custom application code
– Run your queries, compute your results
• Aggregation framework
– Declarative, pipeline-based approach
• Native Map/Reduce in MongoDB
– Javascript functions distributed across cluster
• Hadoop Connector
– Offline batch processing/computation

25

Aggregate into min bars
//Aggregate minute bars for Disney for this month

db.ticks.aggregate(
{ $match: {symbol: "DIS”, timestamp: {$gt: new ISODate("2013-02-01")}}},
{ $project: {
year: {$year: "$timestamp"},
month: {$month: "$timestamp"},
day: {$dayOfMonth: "$timestamp"},
hour: {$hour: "$timestamp"},
minute: {$minute: "$timestamp"},
second: {$second: "$timestamp"},
timestamp: 1,
price: 1}},
{ $sort: { timestamp: 1}},
{ $group :
{ _id : {year: "$year", month: "$month", day: "$day", hour: "$hour", minute: "$minute"},
open: {$first: "$price"},
high: {$max: "$price"},
low: {$min: "$price"},
close: {$last: "$price"} }} )

26

Add analysis on the bars

…
//then count the number of down bars
{ $project: {
downBar: {$lt: [“$close”, “$open”] },
timestamp: 1,
open: 1, high: 1, low: 1, close: 1}},
{ $group: {
_id: “$downBar”,
sum: {$sum: 1}}} })

27

Map-Reduce Example: Sum

var mapFunction = function () {
emit(this.symbol, this.bidPrice);
}
var reduceFunction = function (symbol, priceList) {
return Array.sum(priceList);
}
> db.ticks.mapReduce(
map, reduceFunction, {out: ”tickSums"})

28

Process Data on Hadoop

• MongoDB’s Hadoop Connector
• Supports Map/Reduce, Streaming, Pig
• MongoDB as input/output storage for Hadoop
jobs
– No need to go through HDFS
• Leverage power of Hadoop ecosystem against
operational data in MongoDB

29

Performance, Scalability, and
High Availability

Why MongoDB is fast and scalable

Better data locality In-Memory Auto-Sharding
Caching

Read/write scaling

Relational MongoDB

31

Auto-Sharding for Horizontal Scale

Key Range
Symbol: A…Z

mongod

Read/Write Scalability

32

Auto-Sharding for Horizontal Scale

Key Range Key Range
Symbol: A…J Symbol: K…Z

mongod mongod


33

Sharding

Key Range Key Range Key Range Key Range
Symbol: A…F Symbol: G…J Symbol: K…O Symbol: P…Z

mongod mongod
mongod mongod


34

Application

MongoS MongoS MongoS

Key Range Key Range Key Range Key Range
Symbol: A…F, Symbol: G…J, Symbol: K…O, Symbol: P…Z,
Time Time Time Time

Primary Primary Primary Primary

Secondary Secondary Secondary Secondary

Secondary Secondary Secondary Secondary

35

10gen Products and Services

Subscriptions
Professional Support, Enterprise Edition and Commercial License

Consulting
Expert Resources for All Phases of MongoDB Implementations

Training
Online and In-Person, for Developers and Administrators

36

Summary

• MongoDB is high performance for tick data
• Scales horizontally automatically by auto-
sharding
• Fast, flexible querying, analysis, & aggregation
• Dynamic schema can handle any data types
• MongoDB has all these features with low TCO
• 10gen can support you with anything discussed

37

For More Information

Resource User Data Management
Location

MongoDB Downloads www.mongodb.org/download

Free Online Training education.10gen.com

Webinars and Events www.10gen.com/events

White Papers www.10gen.com/white-papers

Customer Case Studies www.10gen.com/customers

Presentations www.10gen.com/presentations

Documentation docs.mongodb.org

Additional Info info@10gen.com

38

Webinar: How Banks Use MongoDB as a Tick Database

Webinar: How Banks Use MongoDB as a Tick Database

More Related Content

What's hot (20)

Viewers also liked (7)

Similar to Webinar: How Banks Use MongoDB as a Tick Database (20)

More from MongoDB (20)

Webinar: How Banks Use MongoDB as a Tick Database

Editor's Notes