SlideShare a Scribd company logo

Mythbusting: Understanding How We Measure Performance at MongoDB

Download as PPTX, PDF
MongoDB, profile picture
MongoDB

The document discusses benchmarking methods for measuring MongoDB performance, focusing on practical applications and reproducibility. It highlights common pitfalls and provides solutions to improve measurement accuracy, such as pre-creating objects, minimizing contention, and optimizing timing methods. Key goals include enhancing understanding of bottlenecks, proving competitive performance, and ensuring results can be reproduced.

Technology
1 of 70
Downloaded 32 times
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
#MongoDBDays Mythbusting: Understanding How We Measure the Performance of MongoDB Andrew Erlichson VP, Engineering, MongoDB
Before we start… • We are going to look a lot at – C++ kernel code – Java benchmarks – JavaScript tests • And lots of charts • And its going to be awesome!
Goals of Benchmarking – You have an idea – It must be tested – Others must be able to reproduce your work. – Explanations that contribute to knowledge are key. – Should have practical applications Academia
Industry Benchmarketing • Prove you are faster than the competition • Emphasize your best attributes • Repeatable • Explanations
Industry Benchmarketing • Prove you are faster than the competition • Emphasize your best attributes • Repeatable* • Explanations* *OPTIONAL
Goals of Internal Benchmarking • Always be improving • Understand our bottlenecks • Main customer is engineering • Explanations are somewhat important.
Overview – Internal Benchmarking • Some common traps • Performance measurement & diagnosis • What's next
Part One Some Common Traps
#1 Time taken to Insert x Documents long startTime = System.currentTimeMillis(); for (int roundNum = 0; roundNum < numRounds; roundNum++) { for (int i = 0; i < documentsPerInsert; i++) { id++; BasicDBObject doc = new BasicDBObject(); doc.put("_id",id); doc.put("k",rand.nextInt(numMaxInserts)+1); String cVal = "…" doc.put("c",cVal); String padVal = "…"; doc.put("pad",padVal); aDocs[i]=doc; } coll.insert(aDocs); numInserts += documentsPerInsert; globalInserts.addAndGet(documentsPerInsert); } long endTime = System.currentTimeMillis();
#1 Time taken to Insert x Documents long startTime = System.currentTimeMillis(); for (int roundNum = 0; roundNum < numRounds; roundNum++) { for (int i = 0; i < documentsPerInsert; i++) { id++; BasicDBObject doc = new BasicDBObject(); doc.put("_id",id); doc.put("k",rand.nextInt(numMaxInserts)+1); String cVal = "…" doc.put("c",cVal); String padVal = "…"; doc.put("pad",padVal); aDocs[i]=doc; } coll.insert(aDocs); numInserts += documentsPerInsert; globalInserts.addAndGet(documentsPerInsert); } long endTime = System.currentTimeMillis();
#1 Time taken to Insert x Documents long startTime = System.currentTimeMillis(); for (int roundNum = 0; roundNum < numRounds; roundNum++) { for (int i = 0; i < documentsPerInsert; i++) { id++; BasicDBObject doc = new BasicDBObject(); doc.put("_id",id); doc.put("k",rand.nextInt(numMaxInserts)+1); String cVal = "…" doc.put("c",cVal); String padVal = "…"; doc.put("pad",padVal); aDocs[i]=doc; } coll.insert(aDocs); numInserts += documentsPerInsert; globalInserts.addAndGet(documentsPerInsert); } long endTime = System.currentTimeMillis();
#1 Time taken to Insert x Documents long startTime = System.currentTimeMillis(); for (int roundNum = 0; roundNum < numRounds; roundNum++) { for (int i = 0; i < documentsPerInsert; i++) { id++; BasicDBObject doc = new BasicDBObject(); doc.put("_id",id); doc.put("k",rand.nextInt(numMaxInserts)+1); String cVal = "…" doc.put("c",cVal); String padVal = "…"; doc.put("pad",padVal); aDocs[i]=doc; } coll.insert(aDocs); numInserts += documentsPerInsert; globalInserts.addAndGet(documentsPerInsert); } long endTime = System.currentTimeMillis();
#1 Time taken to Insert x Documents long startTime = System.currentTimeMillis(); for (int roundNum = 0; roundNum < numRounds; roundNum++) { for (int i = 0; i < documentsPerInsert; i++) { id++; BasicDBObject doc = new BasicDBObject(); doc.put("_id",id); doc.put("k",rand.nextInt(numMaxInserts)+1); String cVal = "…" doc.put("c",cVal); String padVal = "…"; doc.put("pad",padVal); aDocs[i]=doc; } coll.insert(aDocs); numInserts += documentsPerInsert; globalInserts.addAndGet(documentsPerInsert); } long endTime = System.currentTimeMillis();
So that looks ok, right? long startTime = System.currentTimeMillis(); for (int roundNum = 0; roundNum < numRounds; roundNum++) { for (int i = 0; i < documentsPerInsert; i++) { id++; BasicDBObject doc = new BasicDBObject(); doc.put("_id",id); doc.put("k",rand.nextInt(numMaxInserts)+1); String cVal = "…" doc.put("c",cVal); String padVal = "…"; doc.put("pad",padVal); aDocs[i]=doc; } coll.insert(aDocs); numInserts += documentsPerInsert; globalInserts.addAndGet(documentsPerInsert); } long endTime = System.currentTimeMillis();
What are else you measuring? long startTime = System.currentTimeMillis(); for (int roundNum = 0; roundNum < numRounds; roundNum++) { for (int i = 0; i < documentsPerInsert; i++) { id++; BasicDBObject doc = new BasicDBObject(); doc.put("_id",id); doc.put("k",rand.nextInt(numMaxInserts)+1); String cVal = "…" doc.put("c",cVal); String padVal = "…"; doc.put("pad",padVal); aDocs[i]=doc; } coll.insert(aDocs); numInserts += documentsPerInsert; globalInserts.addAndGet(documentsPerInsert); } long endTime = System.currentTimeMillis(); Object creation and GC management?
What are else you measuring? long startTime = System.currentTimeMillis(); for (int roundNum = 0; roundNum < numRounds; roundNum++) { for (int i = 0; i < documentsPerInsert; i++) { id++; BasicDBObject doc = new BasicDBObject(); doc.put("_id",id); doc.put("k",rand.nextInt(numMaxInserts)+1); String cVal = "…" doc.put("c",cVal); String padVal = "…"; doc.put("pad",padVal); aDocs[i]=doc; } coll.insert(aDocs); numInserts += documentsPerInsert; globalInserts.addAndGet(documentsPerInsert); } long endTime = System.currentTimeMillis(); Object creation and GC management? Thread contention on nextInt()?
What are else you measuring? long startTime = System.currentTimeMillis(); for (int roundNum = 0; roundNum < numRounds; roundNum++) { for (int i = 0; i < documentsPerInsert; i++) { id++; BasicDBObject doc = new BasicDBObject(); doc.put("_id",id); doc.put("k",rand.nextInt(numMaxInserts)+1); String cVal = "…" doc.put("c",cVal); String padVal = "…"; doc.put("pad",padVal); aDocs[i]=doc; } coll.insert(aDocs); numInserts += documentsPerInsert; globalInserts.addAndGet(documentsPerInsert); } long endTime = System.currentTimeMillis(); Object creation and GC management? Thread contention on nextInt()? Time to synthesize data?
What are else you measuring? long startTime = System.currentTimeMillis(); for (int roundNum = 0; roundNum < numRounds; roundNum++) { for (int i = 0; i < documentsPerInsert; i++) { id++; BasicDBObject doc = new BasicDBObject(); doc.put("_id",id); doc.put("k",rand.nextInt(numMaxInserts)+1); String cVal = "…" doc.put("c",cVal); String padVal = "…"; doc.put("pad",padVal); aDocs[i]=doc; } coll.insert(aDocs); numInserts += documentsPerInsert; globalInserts.addAndGet(documentsPerInsert); } long endTime = System.currentTimeMillis(); Object creation and GC management? Thread contention on nextInt()? Time to synthesize data? Thread contention on addAndGet()?
What are else you measuring? long startTime = System.currentTimeMillis(); for (int roundNum = 0; roundNum < numRounds; roundNum++) { for (int i = 0; i < documentsPerInsert; i++) { id++; BasicDBObject doc = new BasicDBObject(); doc.put("_id",id); doc.put("k",rand.nextInt(numMaxInserts)+1); String cVal = "…" doc.put("c",cVal); String padVal = "…"; doc.put("pad",padVal); aDocs[i]=doc; } coll.insert(aDocs); numInserts += documentsPerInsert; globalInserts.addAndGet(documentsPerInsert); } long endTime = System.currentTimeMillis(); Object creation and GC management? Thread contention on nextInt()? Time to synthesize data? Thread contention on addAndGet()? Clock resolution?
Solution: Pre-Create the objects // Pre Create the Object outside the Loop BasicDBObject[] aDocs = new BasicDBObject[documentsPerInsert]; for (int i=0; i < documentsPerInsert; i++) { BasicDBObject doc = new BasicDBObject(); String cVal = "…"; doc.put("c",cVal); String padVal = "…"; doc.put("pad",padVal); aDocs[i] = doc; } Pre-create non varying data outside the timing loop Alternative • Pre-create the data in a file; load from file
Solution: Remove contention // Use ThreadLocalRandom generator or an instance of java.util.Random per thread java.util.concurrent.ThreadLocalRandom rand; for (long roundNum = 0; roundNum < numRounds; roundNum++) { for (int i = 0; i < documentsPerInsert; i++) { id++; doc = aDocs[i]; doc.put("_id",id); doc.put("k", nextInt(rand, numMaxInserts)+1); } coll.insert(aDocs); numInserts += documentsPerInsert; } // Maintain count outside the loop globalInserts.addAndGet(documentsPerInsert * roundNum); Remove contention nextInt() by making Thread local
Solution: Remove contention // Use ThreadLocalRandom generator or an instance of java.util.Random per thread java.util.concurrent.ThreadLocalRandom rand; Remove contention nextInt() by making Thread local for (long roundNum = 0; roundNum < numRounds; roundNum++) { for (int i = 0; i < documentsPerInsert; i++) { id++; doc = aDocs[i]; doc.put("_id",id); doc.put("k", nextInt(rand, numMaxInserts)+1); } coll.insert(aDocs); numInserts += documentsPerInsert; } // Maintain count outside the loop globalInserts.addAndGet(documentsPerInsert * roundNum); Remove contention on addAndGet()
Solution: Timer resolution long startTime = System.currentTimeMillis(); … long endTime = System.currentTimeMillis(); long startTime = System.nanoTime(); … long endTime = System.nanoTime() - startTime; "granularity of the value depends on the underlying operating system and may be larger" "resolution is at least as good as that of currentTimeMillis()" Source • http://docs.oracle.com/javase/7/docs/api/java/lang/System.html
General Principal #1 Know what you are measuring
#2 Response time to return all results BasicDBObject doc = new BasicDBObject(); doc.put("v", str); // str is a 2k string for (int i=0; i < 1000; i++) { doc.put("_id",i); coll.insert(doc); } BasicDBObject predicate = new BasicDBObject(); long startTime = System.currentTimeMillis(); DBCursor cur = coll.find(predicate); DBObject foundObj; while (cur.hasNext()) { foundObj = cur.next(); } long endTime = System.currentTimeMillis();
#2 Response time to return all results BasicDBObject doc = new BasicDBObject(); doc.put("v", str); // str is a 2k string for (int i=0; i < 1000; i++) { doc.put("_id",i); coll.insert(doc); } BasicDBObject predicate = new BasicDBObject(); long startTime = System.currentTimeMillis(); DBCursor cur = coll.find(predicate); DBObject foundObj; while (cur.hasNext()) { foundObj = cur.next(); } long endTime = System.currentTimeMillis();
#2 Response time to return all results BasicDBObject doc = new BasicDBObject(); doc.put("v", str); // str is a 2k string for (int i=0; i < 1000; i++) { doc.put("_id",i); coll.insert(doc); } BasicDBObject predicate = new BasicDBObject(); long startTime = System.currentTimeMillis(); DBCursor cur = coll.find(predicate); DBObject foundObj; while (cur.hasNext()) { foundObj = cur.next(); } long endTime = System.currentTimeMillis();
#2 Response time to return all results BasicDBObject doc = new BasicDBObject(); doc.put("v", str); // str is a 2k string for (int i=0; i < 1000; i++) { doc.put("_id",i); coll.insert(doc); } BasicDBObject predicate = new BasicDBObject(); long startTime = System.currentTimeMillis(); DBCursor cur = coll.find(predicate); DBObject foundObj; while (cur.hasNext()) { foundObj = cur.next(); } long endTime = System.currentTimeMillis();
So that looks ok, right? BasicDBObject doc = new BasicDBObject(); doc.put("v", str); // str is a 2k string for (int i=0; i < 1000; i++) { doc.put("_id",i); coll.insert(doc); } BasicDBObject predicate = new BasicDBObject(); long startTime = System.currentTimeMillis(); DBCursor cur = coll.find(predicate); DBObject foundObj; while (cur.hasNext()) { foundObj = cur.next(); } long endTime = System.currentTimeMillis();
What else are you measuring? BasicDBObject doc = new BasicDBObject(); doc.put("v", str); // str is a 2k string for (int i=0; i < 1000; i++) { doc.put("_id",i); coll.insert(doc); } BasicDBObject predicate = new BasicDBObject(); long startTime = System.currentTimeMillis(); DBCursor cur = coll.find(predicate); DBObject foundObj; while (cur.hasNext()) { foundObj = cur.next(); } long endTime = System.currentTimeMillis(); Each doc is is 4080 bytes on disk with powerOf2Sizes
What else are you measuring? BasicDBObject doc = new BasicDBObject(); doc.put("v", str); // str is a 2k string for (int i=0; i < 1000; i++) { doc.put("_id",i); coll.insert(doc); } BasicDBObject predicate = new BasicDBObject(); long startTime = System.currentTimeMillis(); DBCursor cur = coll.find(predicate); DBObject foundObj; while (cur.hasNext()) { foundObj = cur.next(); } long endTime = System.currentTimeMillis(); Each doc is is 4080 bytes on disk with powerOf2Sizes Unrestricted predicate?
What else are you measuring? BasicDBObject doc = new BasicDBObject(); doc.put("v", str); // str is a 2k string for (int i=0; i < 1000; i++) { doc.put("_id",i); coll.insert(doc); } BasicDBObject predicate = new BasicDBObject(); long startTime = System.currentTimeMillis(); DBCursor cur = coll.find(predicate); DBObject foundObj; while (cur.hasNext()) { foundObj = cur.next(); } long endTime = System.currentTimeMillis(); Each doc is is 4080 bytes on disk with powerOf2Sizes Unrestricted predicate? Measuring • Time to parse & execute query • Time to retrieve all document But also • Cost of shipping ~4MB data through network stack
Solution: Limit the projection BasicDBObject predicate = new BasicDBObject(); predicate.put("_id", new BasicDBObject("$gte", 10).append("$lte", 20)); BasicDBObject projection = new BasicDBObject(); projection.put("_id", 1); long startTime = System.currentTimeMillis(); DBCursor cur = coll.find(predicate, projection ); DBObject foundObj; while (cur.hasNext()) { foundObj = cur.next(); } long endTime = System.currentTimeMillis(); Return fixed range
Solution: Limit the projection BasicDBObject predicate = new BasicDBObject(); predicate.put("_id", new BasicDBObject("$gte", 10).append("$lte", 20)); BasicDBObject projection = new BasicDBObject(); projection.put("_id", 1); long startTime = System.currentTimeMillis(); DBCursor cur = coll.find(predicate, projection ); DBObject foundObj; while (cur.hasNext()) { foundObj = cur.next(); } long endTime = System.currentTimeMillis(); Return fixed range Only project _id
Solution: Limit the projection BasicDBObject predicate = new BasicDBObject(); predicate.put("_id", new BasicDBObject("$gte", 10).append("$lte", 20)); BasicDBObject projection = new BasicDBObject(); projection.put("_id", 1); long startTime = System.currentTimeMillis(); DBCursor cur = coll.find(predicate, projection ); DBObject foundObj; while (cur.hasNext()) { foundObj = cur.next(); } long endTime = System.currentTimeMillis(); Return fixed range Only project _id Only 46k transferred through network stack
General Principal #2 Measure only what you need to measure
Part Two Performance measurement & diagnosis
Broad categories • Micro Benchmarks • Workloads
Micro benchmarks: mongo-perf
mongo-perf: goals • Measure – commands • Configure – Single mongod, ReplSet size (1 -> n), Sharding – Single vs. Multiple DB – O/S • Characterize – Throughput (ops/second) by thread count • Compare
What do you get? Better Thread Count Ops/Se c
What do you get? Measured improvement between rc0 and rc2 Better
Benchmark source code tests.push( { name: "Commands.CountsIntIDRange", pre: function( collection ) { collection.drop(); for ( var i = 0; i < 1000; i++ ) { collection.insert( { _id : i } ); } collection.getDB().getLastError(); }, ops: [ { op: "command", ns : "testdb", command : { count : "mycollection", query : { _id : { "$gt" : 10, "$lt" : 100 } } } } ] } );
Benchmark source code tests.push( { name: "Commands.CountsIntIDRange", pre: function( collection ) { collection.drop(); for ( var i = 0; i < 1000; i++ ) { collection.insert( { _id : i } ); } collection.getDB().getLastError(); }, ops: [ { op: "command", ns : "testdb", command : { count : "mycollection", query : { _id : { "$gt" : 10, "$lt" : 100 } } } } ] } );
Benchmark source code tests.push( { name: "Commands.CountsIntIDRange", pre: function( collection ) { collection.drop(); for ( var i = 0; i < 1000; i++ ) { collection.insert( { _id : i } ); } collection.getDB().getLastError(); }, ops: [ { op: "command", ns : "testdb", command : { count : "mycollection", query : { _id : { "$gt" : 10, "$lt" : 100 } } } } ] } );
Benchmark source code tests.push( { name: "Commands.CountsIntIDRange", pre: function( collection ) { collection.drop(); for ( var i = 0; i < 1000; i++ ) { collection.insert( { _id : i } ); } collection.getDB().getLastError(); }, ops: [ { op: "command", ns : "testdb", command : { count : "mycollection", query : { _id : { "$gt" : 10, "$lt" : 100 } } } } ] } );
Code Change
Workloads • "public" workloads – YCSB – Sysbench • "real world" simulations – Inbox fan in/out – Message Stores – Content Management
Example: Bulk Load Performance 16m Documents Better 55% degradation 2.6.0-rc1 vs 2.4.10
Ouch… where's the tree in the woods? • 2.4.10 -> 2.6.0 – 4495 git commits
git-bisect • Bisect between good/bad hashes • git-bisect nominates a new githash – Build against githash – Re-run test – Confirm if this githash is good/bad • Rinse and repeat
Code Change - Bad Githash
Code Change - Fix
Bulk Load Performance - Fix Better 11% improvement 2.6.1 vs 2.4.10
The problem with measurement • Observability – What can you observe on the system? • Effect – What effects does the observation cause?
mtools
mtools • MongoDB log file analysis – Filter logs for operations, events – Response time, lock durations – Plot • https://github.com/rueckstiess/mtools
Code Change – Yielding Policy
Code Change
Response Times Bulk Insert 2.6.0 vs 2.6.1 Ceiling similar, lower floor resulting in 40% improvement in throughput
Secondary effects of Yield policy change Write lock time reduced Order of magnitude reduction of write lock duration
Unexpected side effects of measurement? > db.serverStatus() Yes – will cause a read lock to be acquired > db.serverStatus({recordStats:0}) No – lock is not acquired > mongostat Yes - until SERVER-14008 resolved, uses db.serverStatus()
CPU sampling • Get an impression of – Call Graphs – CPU time spent on node and called nodes
Setup & building with google-profiler > sudo apt-get install google-perftools > sudo apt-get install libunwind7-dev > scons --use-cpu-profiler mongod
Start the profiling > mongod –dbpath <…> Note: Do not use –fork > mongo > use admin > db.runCommand({_cpuProfilerStart: {profileFilename: 'foo.prof'}}) Execute some commands that you want to profile > db.runCommand({_cpuProfilerStop: 1})
Sample start vs. end of workload
Sample start vs. end of workload
Code change
Public Benchmarks – Not all forks are the same… • YCSB – https://github.com/achille/YCSB • sysbench-mongodb – https://github.com/mdcallag/sysbench-mongodb
#MongoDBDays Thank You Andrew Erlichson andrew@mongodb.com VP, Engineering

More Related Content

PPTX
Mythbusting: Understanding How We Measure the Performance of MongoDB
MongoDB
 
PPTX
Mythbusting: Understanding How We Measure the Performance of MongoDB
MongoDB
 
TXT
Ac2
Muhammad Islahuddin
 
PPTX
Mythbusting: Understanding How We Measure the Performance of MongoDB
MongoDB
 
DOCX
WOTC_Import
Luther Quinn
 
PPTX
Using Arbor/ RGraph JS libaries for Data Visualisation
Alex Hardman
 
PDF
Green dao
彥彬 洪
 
PDF
Green dao
Droidcon Berlin
 
Mythbusting: Understanding How We Measure the Performance of MongoDB
MongoDB
 
Mythbusting: Understanding How We Measure the Performance of MongoDB
MongoDB
 
Ac2
Muhammad Islahuddin
 
Mythbusting: Understanding How We Measure the Performance of MongoDB
MongoDB
 
WOTC_Import
Luther Quinn
 
Using Arbor/ RGraph JS libaries for Data Visualisation
Alex Hardman
 
Green dao
彥彬 洪
 
Green dao
Droidcon Berlin
 

What's hot (20)

PDF
ORMLite Android
哲偉 楊
 
PPTX
GreenDao Introduction
Booch Lin
 
PDF
Look Ma, “update DB to HTML5 using C++”, no hands! 
aleks-f
 
PDF
greenDAO
Mu Chun Wang
 
PPT
Fast querying indexing for performance (4)
MongoDB
 
PPTX
Javascript Arrays
shaheenakv
 
PDF
Indexing and Query Optimizer (Mongo Austin)
MongoDB
 
PDF
Mongo indexes
Mehmet Çetin
 
PPTX
Getting Started with MongoDB and NodeJS
MongoDB
 
PPTX
A Test of Strength
Chris Oldwood
 
PPTX
MongoDB San Francisco 2013: Hash-based Sharding in MongoDB 2.4 presented by B...
MongoDB
 
PPTX
MongoDB + Java - Everything you need to know
Norberto Leite
 
PDF
MongoDB World 2016: Deciphering .explain() Output
MongoDB
 
PPS
CS101- Introduction to Computing- Lecture 26
Bilal Ahmed
 
PPT
POLITEKNIK MALAYSIA
Aiman Hud
 
PDF
Kenneth Truyers - Using Git as a NoSql database - Codemotion Milan 2018
Codemotion
 
PDF
The Ring programming language version 1.10 book - Part 79 of 212
Mahmoud Samir Fayed
 
PDF
MongoDB Performance Tuning
MongoDB
 
PPTX
Indexing and Query Optimization
MongoDB
 
PDF
The Ring programming language version 1.4.1 book - Part 13 of 31
Mahmoud Samir Fayed
 
ORMLite Android
哲偉 楊
 
GreenDao Introduction
Booch Lin
 
Look Ma, “update DB to HTML5 using C++”, no hands! 
aleks-f
 
greenDAO
Mu Chun Wang
 
Fast querying indexing for performance (4)
MongoDB
 
Javascript Arrays
shaheenakv
 
Indexing and Query Optimizer (Mongo Austin)
MongoDB
 
Mongo indexes
Mehmet Çetin
 
Getting Started with MongoDB and NodeJS
MongoDB
 
A Test of Strength
Chris Oldwood
 
MongoDB San Francisco 2013: Hash-based Sharding in MongoDB 2.4 presented by B...
MongoDB
 
MongoDB + Java - Everything you need to know
Norberto Leite
 
MongoDB World 2016: Deciphering .explain() Output
MongoDB
 
CS101- Introduction to Computing- Lecture 26
Bilal Ahmed
 
POLITEKNIK MALAYSIA
Aiman Hud
 
Kenneth Truyers - Using Git as a NoSql database - Codemotion Milan 2018
Codemotion
 
The Ring programming language version 1.10 book - Part 79 of 212
Mahmoud Samir Fayed
 
MongoDB Performance Tuning
MongoDB
 
Indexing and Query Optimization
MongoDB
 
The Ring programming language version 1.4.1 book - Part 13 of 31
Mahmoud Samir Fayed
 
Ad

Similar to Mythbusting: Understanding How We Measure Performance at MongoDB (7)

PDF
20110514 mongo dbチューニング
Yuichi Matsuo
 
PDF
10 Key MongoDB Performance Indicators
iammutex
 
PPTX
Map db
Debmalya Jash
 
PDF
MongoDB World 2019: RDBMS Versus MongoDB Aggregation Performance
MongoDB
 
PDF
Apache Cassandra - Data modelling
Alex Thompson
 
PPTX
Sizing MongoDB Clusters
MongoDB
 
PDF
Mongo db improve the performance of your application codemotion2016
Juan Antonio Roy Couto
 
20110514 mongo dbチューニング
Yuichi Matsuo
 
10 Key MongoDB Performance Indicators
iammutex
 
Map db
Debmalya Jash
 
MongoDB World 2019: RDBMS Versus MongoDB Aggregation Performance
MongoDB
 
Apache Cassandra - Data modelling
Alex Thompson
 
Sizing MongoDB Clusters
MongoDB
 
Mongo db improve the performance of your application codemotion2016
Juan Antonio Roy Couto
 
Ad

More from MongoDB (20)

PDF
MongoDB SoCal 2020: Migrate Anything* to MongoDB Atlas
MongoDB
 
PDF
MongoDB SoCal 2020: Go on a Data Safari with MongoDB Charts!
MongoDB
 
PDF
MongoDB SoCal 2020: Using MongoDB Services in Kubernetes: Any Platform, Devel...
MongoDB
 
PDF
MongoDB SoCal 2020: A Complete Methodology of Data Modeling for MongoDB
MongoDB
 
PDF
MongoDB SoCal 2020: From Pharmacist to Analyst: Leveraging MongoDB for Real-T...
MongoDB
 
PDF
MongoDB SoCal 2020: Best Practices for Working with IoT and Time-series Data
MongoDB
 
PDF
MongoDB SoCal 2020: MongoDB Atlas Jump Start
MongoDB
 
PDF
MongoDB .local San Francisco 2020: Powering the new age data demands [Infosys]
MongoDB
 
PDF
MongoDB .local San Francisco 2020: Using Client Side Encryption in MongoDB 4.2
MongoDB
 
PDF
MongoDB .local San Francisco 2020: Using MongoDB Services in Kubernetes: any ...
MongoDB
 
PDF
MongoDB .local San Francisco 2020: Go on a Data Safari with MongoDB Charts!
MongoDB
 
PDF
MongoDB .local San Francisco 2020: From SQL to NoSQL -- Changing Your Mindset
MongoDB
 
PDF
MongoDB .local San Francisco 2020: MongoDB Atlas Jumpstart
MongoDB
 
PDF
MongoDB .local San Francisco 2020: Tips and Tricks++ for Querying and Indexin...
MongoDB
 
PDF
MongoDB .local San Francisco 2020: Aggregation Pipeline Power++
MongoDB
 
PDF
MongoDB .local San Francisco 2020: A Complete Methodology of Data Modeling fo...
MongoDB
 
PDF
MongoDB .local San Francisco 2020: MongoDB Atlas Data Lake Technical Deep Dive
MongoDB
 
PDF
MongoDB .local San Francisco 2020: Developing Alexa Skills with MongoDB & Golang
MongoDB
 
PDF
MongoDB .local Paris 2020: Realm : l'ingrédient secret pour de meilleures app...
MongoDB
 
PDF
MongoDB .local Paris 2020: Upply @MongoDB : Upply : Quand le Machine Learning...
MongoDB
 
MongoDB SoCal 2020: Migrate Anything* to MongoDB Atlas
MongoDB
 
MongoDB SoCal 2020: Go on a Data Safari with MongoDB Charts!
MongoDB
 
MongoDB SoCal 2020: Using MongoDB Services in Kubernetes: Any Platform, Devel...
MongoDB
 
MongoDB SoCal 2020: A Complete Methodology of Data Modeling for MongoDB
MongoDB
 
MongoDB SoCal 2020: From Pharmacist to Analyst: Leveraging MongoDB for Real-T...
MongoDB
 
MongoDB SoCal 2020: Best Practices for Working with IoT and Time-series Data
MongoDB
 
MongoDB SoCal 2020: MongoDB Atlas Jump Start
MongoDB
 
MongoDB .local San Francisco 2020: Powering the new age data demands [Infosys]
MongoDB
 
MongoDB .local San Francisco 2020: Using Client Side Encryption in MongoDB 4.2
MongoDB
 
MongoDB .local San Francisco 2020: Using MongoDB Services in Kubernetes: any ...
MongoDB
 
MongoDB .local San Francisco 2020: Go on a Data Safari with MongoDB Charts!
MongoDB
 
MongoDB .local San Francisco 2020: From SQL to NoSQL -- Changing Your Mindset
MongoDB
 
MongoDB .local San Francisco 2020: MongoDB Atlas Jumpstart
MongoDB
 
MongoDB .local San Francisco 2020: Tips and Tricks++ for Querying and Indexin...
MongoDB
 
MongoDB .local San Francisco 2020: Aggregation Pipeline Power++
MongoDB
 
MongoDB .local San Francisco 2020: A Complete Methodology of Data Modeling fo...
MongoDB
 
MongoDB .local San Francisco 2020: MongoDB Atlas Data Lake Technical Deep Dive
MongoDB
 
MongoDB .local San Francisco 2020: Developing Alexa Skills with MongoDB & Golang
MongoDB
 
MongoDB .local Paris 2020: Realm : l'ingrédient secret pour de meilleures app...
MongoDB
 
MongoDB .local Paris 2020: Upply @MongoDB : Upply : Quand le Machine Learning...
MongoDB
 

Recently uploaded (20)

PDF
NewMind AI Weekly Chronicles - August'25 Week I
NewMind AI
 
PDF
Reach Out and Touch Someone: Haptics and Empathic Computing
Mark Billinghurst
 
PDF
Electronic commerce courselecture one. Pdf
OmerMohamed64
 
PDF
Architecting across the Boundaries of two Complex Domains - Healthcare & Tech...
Peter Tan
 
PPTX
Detection-First SIEM: Rule Types, Dashboards, and Threat-Informed Strategy
ReZa AdineH
 
PPTX
Big Data Technologies - Introduction.pptx
kuthubussaman1
 
PPTX
Digital-Transformation-Roadmap-for-Companies.pptx
David Malick Dieng
 
PDF
Build a system with the filesystem maintained by OSTree @ COSCUP 2025
Jian-Hong Pan
 
PPT
“AI and Expert System Decision Support & Business Intelligence Systems”
ZubinRadhakrishnan
 
PPTX
Spectroscopy.pptx food analysis technology
nawahassan45
 
PPTX
20250228 LYD VKU AI Blended-Learning.pptx
DatVoNgoc
 
PDF
Mobile App Security Testing_ A Comprehensive Guide.pdf
flufftailshop
 
PDF
7 ChatGPT Prompts to Help You Define Your Ideal Customer Profile.pdf
SOFTTECHHUB
 
PPTX
VMware vSphere Foundation How to Sell Presentation-Ver1.4-2-14-2024.pptx
blackmambaettijean
 
PDF
Network Security Unit 5.pdf for BCA BBA.
Serpent6
 
PDF
Blue Purple Modern Animated Computer Science Presentation.pdf.pdf
Akar16
 
PDF
Diabetes mellitus diagnosis method based random forest with bat algorithm
IAESIJAI
 
PDF
The Rise and Fall of 3GPP – Time for a Sabbatical?
3G4G
 
PDF
TokAI - TikTok AI Agent : The First AI Application That Analyzes 10,000+ Vira...
SOFTTECHHUB
 
PPTX
KOM of Painting work and Equipment Insulation REV00 update 25-dec.pptx
muhammadmuneebnaeem7
 
NewMind AI Weekly Chronicles - August'25 Week I
NewMind AI
 
Reach Out and Touch Someone: Haptics and Empathic Computing
Mark Billinghurst
 
Electronic commerce courselecture one. Pdf
OmerMohamed64
 
Architecting across the Boundaries of two Complex Domains - Healthcare & Tech...
Peter Tan
 
Detection-First SIEM: Rule Types, Dashboards, and Threat-Informed Strategy
ReZa AdineH
 
Big Data Technologies - Introduction.pptx
kuthubussaman1
 
Digital-Transformation-Roadmap-for-Companies.pptx
David Malick Dieng
 
Build a system with the filesystem maintained by OSTree @ COSCUP 2025
Jian-Hong Pan
 
“AI and Expert System Decision Support & Business Intelligence Systems”
ZubinRadhakrishnan
 
Spectroscopy.pptx food analysis technology
nawahassan45
 
20250228 LYD VKU AI Blended-Learning.pptx
DatVoNgoc
 
Mobile App Security Testing_ A Comprehensive Guide.pdf
flufftailshop
 
7 ChatGPT Prompts to Help You Define Your Ideal Customer Profile.pdf
SOFTTECHHUB
 
VMware vSphere Foundation How to Sell Presentation-Ver1.4-2-14-2024.pptx
blackmambaettijean
 
Network Security Unit 5.pdf for BCA BBA.
Serpent6
 
Blue Purple Modern Animated Computer Science Presentation.pdf.pdf
Akar16
 
Diabetes mellitus diagnosis method based random forest with bat algorithm
IAESIJAI
 
The Rise and Fall of 3GPP – Time for a Sabbatical?
3G4G
 
TokAI - TikTok AI Agent : The First AI Application That Analyzes 10,000+ Vira...
SOFTTECHHUB
 
KOM of Painting work and Equipment Insulation REV00 update 25-dec.pptx
muhammadmuneebnaeem7
 

Mythbusting: Understanding How We Measure Performance at MongoDB

  • 1. #MongoDBDays Mythbusting: Understanding How We Measure the Performance of MongoDB Andrew Erlichson VP, Engineering, MongoDB
  • 2. Before we start… • We are going to look a lot at – C++ kernel code – Java benchmarks – JavaScript tests • And lots of charts • And its going to be awesome!
  • 3. Goals of Benchmarking – You have an idea – It must be tested – Others must be able to reproduce your work. – Explanations that contribute to knowledge are key. – Should have practical applications Academia
  • 4. Industry Benchmarketing • Prove you are faster than the competition • Emphasize your best attributes • Repeatable • Explanations
  • 5. Industry Benchmarketing • Prove you are faster than the competition • Emphasize your best attributes • Repeatable* • Explanations* *OPTIONAL
  • 6. Goals of Internal Benchmarking • Always be improving • Understand our bottlenecks • Main customer is engineering • Explanations are somewhat important.
  • 7. Overview – Internal Benchmarking • Some common traps • Performance measurement & diagnosis • What's next
  • 8. Part One Some Common Traps
  • 9. #1 Time taken to Insert x Documents long startTime = System.currentTimeMillis(); for (int roundNum = 0; roundNum < numRounds; roundNum++) { for (int i = 0; i < documentsPerInsert; i++) { id++; BasicDBObject doc = new BasicDBObject(); doc.put("_id",id); doc.put("k",rand.nextInt(numMaxInserts)+1); String cVal = "…" doc.put("c",cVal); String padVal = "…"; doc.put("pad",padVal); aDocs[i]=doc; } coll.insert(aDocs); numInserts += documentsPerInsert; globalInserts.addAndGet(documentsPerInsert); } long endTime = System.currentTimeMillis();
  • 10. #1 Time taken to Insert x Documents long startTime = System.currentTimeMillis(); for (int roundNum = 0; roundNum < numRounds; roundNum++) { for (int i = 0; i < documentsPerInsert; i++) { id++; BasicDBObject doc = new BasicDBObject(); doc.put("_id",id); doc.put("k",rand.nextInt(numMaxInserts)+1); String cVal = "…" doc.put("c",cVal); String padVal = "…"; doc.put("pad",padVal); aDocs[i]=doc; } coll.insert(aDocs); numInserts += documentsPerInsert; globalInserts.addAndGet(documentsPerInsert); } long endTime = System.currentTimeMillis();
  • 11. #1 Time taken to Insert x Documents long startTime = System.currentTimeMillis(); for (int roundNum = 0; roundNum < numRounds; roundNum++) { for (int i = 0; i < documentsPerInsert; i++) { id++; BasicDBObject doc = new BasicDBObject(); doc.put("_id",id); doc.put("k",rand.nextInt(numMaxInserts)+1); String cVal = "…" doc.put("c",cVal); String padVal = "…"; doc.put("pad",padVal); aDocs[i]=doc; } coll.insert(aDocs); numInserts += documentsPerInsert; globalInserts.addAndGet(documentsPerInsert); } long endTime = System.currentTimeMillis();
  • 12. #1 Time taken to Insert x Documents long startTime = System.currentTimeMillis(); for (int roundNum = 0; roundNum < numRounds; roundNum++) { for (int i = 0; i < documentsPerInsert; i++) { id++; BasicDBObject doc = new BasicDBObject(); doc.put("_id",id); doc.put("k",rand.nextInt(numMaxInserts)+1); String cVal = "…" doc.put("c",cVal); String padVal = "…"; doc.put("pad",padVal); aDocs[i]=doc; } coll.insert(aDocs); numInserts += documentsPerInsert; globalInserts.addAndGet(documentsPerInsert); } long endTime = System.currentTimeMillis();
  • 13. #1 Time taken to Insert x Documents long startTime = System.currentTimeMillis(); for (int roundNum = 0; roundNum < numRounds; roundNum++) { for (int i = 0; i < documentsPerInsert; i++) { id++; BasicDBObject doc = new BasicDBObject(); doc.put("_id",id); doc.put("k",rand.nextInt(numMaxInserts)+1); String cVal = "…" doc.put("c",cVal); String padVal = "…"; doc.put("pad",padVal); aDocs[i]=doc; } coll.insert(aDocs); numInserts += documentsPerInsert; globalInserts.addAndGet(documentsPerInsert); } long endTime = System.currentTimeMillis();
  • 14. So that looks ok, right? long startTime = System.currentTimeMillis(); for (int roundNum = 0; roundNum < numRounds; roundNum++) { for (int i = 0; i < documentsPerInsert; i++) { id++; BasicDBObject doc = new BasicDBObject(); doc.put("_id",id); doc.put("k",rand.nextInt(numMaxInserts)+1); String cVal = "…" doc.put("c",cVal); String padVal = "…"; doc.put("pad",padVal); aDocs[i]=doc; } coll.insert(aDocs); numInserts += documentsPerInsert; globalInserts.addAndGet(documentsPerInsert); } long endTime = System.currentTimeMillis();
  • 15. What are else you measuring? long startTime = System.currentTimeMillis(); for (int roundNum = 0; roundNum < numRounds; roundNum++) { for (int i = 0; i < documentsPerInsert; i++) { id++; BasicDBObject doc = new BasicDBObject(); doc.put("_id",id); doc.put("k",rand.nextInt(numMaxInserts)+1); String cVal = "…" doc.put("c",cVal); String padVal = "…"; doc.put("pad",padVal); aDocs[i]=doc; } coll.insert(aDocs); numInserts += documentsPerInsert; globalInserts.addAndGet(documentsPerInsert); } long endTime = System.currentTimeMillis(); Object creation and GC management?
  • 16. What are else you measuring? long startTime = System.currentTimeMillis(); for (int roundNum = 0; roundNum < numRounds; roundNum++) { for (int i = 0; i < documentsPerInsert; i++) { id++; BasicDBObject doc = new BasicDBObject(); doc.put("_id",id); doc.put("k",rand.nextInt(numMaxInserts)+1); String cVal = "…" doc.put("c",cVal); String padVal = "…"; doc.put("pad",padVal); aDocs[i]=doc; } coll.insert(aDocs); numInserts += documentsPerInsert; globalInserts.addAndGet(documentsPerInsert); } long endTime = System.currentTimeMillis(); Object creation and GC management? Thread contention on nextInt()?
  • 17. What are else you measuring? long startTime = System.currentTimeMillis(); for (int roundNum = 0; roundNum < numRounds; roundNum++) { for (int i = 0; i < documentsPerInsert; i++) { id++; BasicDBObject doc = new BasicDBObject(); doc.put("_id",id); doc.put("k",rand.nextInt(numMaxInserts)+1); String cVal = "…" doc.put("c",cVal); String padVal = "…"; doc.put("pad",padVal); aDocs[i]=doc; } coll.insert(aDocs); numInserts += documentsPerInsert; globalInserts.addAndGet(documentsPerInsert); } long endTime = System.currentTimeMillis(); Object creation and GC management? Thread contention on nextInt()? Time to synthesize data?
  • 18. What are else you measuring? long startTime = System.currentTimeMillis(); for (int roundNum = 0; roundNum < numRounds; roundNum++) { for (int i = 0; i < documentsPerInsert; i++) { id++; BasicDBObject doc = new BasicDBObject(); doc.put("_id",id); doc.put("k",rand.nextInt(numMaxInserts)+1); String cVal = "…" doc.put("c",cVal); String padVal = "…"; doc.put("pad",padVal); aDocs[i]=doc; } coll.insert(aDocs); numInserts += documentsPerInsert; globalInserts.addAndGet(documentsPerInsert); } long endTime = System.currentTimeMillis(); Object creation and GC management? Thread contention on nextInt()? Time to synthesize data? Thread contention on addAndGet()?
  • 19. What are else you measuring? long startTime = System.currentTimeMillis(); for (int roundNum = 0; roundNum < numRounds; roundNum++) { for (int i = 0; i < documentsPerInsert; i++) { id++; BasicDBObject doc = new BasicDBObject(); doc.put("_id",id); doc.put("k",rand.nextInt(numMaxInserts)+1); String cVal = "…" doc.put("c",cVal); String padVal = "…"; doc.put("pad",padVal); aDocs[i]=doc; } coll.insert(aDocs); numInserts += documentsPerInsert; globalInserts.addAndGet(documentsPerInsert); } long endTime = System.currentTimeMillis(); Object creation and GC management? Thread contention on nextInt()? Time to synthesize data? Thread contention on addAndGet()? Clock resolution?
  • 20. Solution: Pre-Create the objects // Pre Create the Object outside the Loop BasicDBObject[] aDocs = new BasicDBObject[documentsPerInsert]; for (int i=0; i < documentsPerInsert; i++) { BasicDBObject doc = new BasicDBObject(); String cVal = "…"; doc.put("c",cVal); String padVal = "…"; doc.put("pad",padVal); aDocs[i] = doc; } Pre-create non varying data outside the timing loop Alternative • Pre-create the data in a file; load from file
  • 21. Solution: Remove contention // Use ThreadLocalRandom generator or an instance of java.util.Random per thread java.util.concurrent.ThreadLocalRandom rand; for (long roundNum = 0; roundNum < numRounds; roundNum++) { for (int i = 0; i < documentsPerInsert; i++) { id++; doc = aDocs[i]; doc.put("_id",id); doc.put("k", nextInt(rand, numMaxInserts)+1); } coll.insert(aDocs); numInserts += documentsPerInsert; } // Maintain count outside the loop globalInserts.addAndGet(documentsPerInsert * roundNum); Remove contention nextInt() by making Thread local
  • 22. Solution: Remove contention // Use ThreadLocalRandom generator or an instance of java.util.Random per thread java.util.concurrent.ThreadLocalRandom rand; Remove contention nextInt() by making Thread local for (long roundNum = 0; roundNum < numRounds; roundNum++) { for (int i = 0; i < documentsPerInsert; i++) { id++; doc = aDocs[i]; doc.put("_id",id); doc.put("k", nextInt(rand, numMaxInserts)+1); } coll.insert(aDocs); numInserts += documentsPerInsert; } // Maintain count outside the loop globalInserts.addAndGet(documentsPerInsert * roundNum); Remove contention on addAndGet()
  • 23. Solution: Timer resolution long startTime = System.currentTimeMillis(); … long endTime = System.currentTimeMillis(); long startTime = System.nanoTime(); … long endTime = System.nanoTime() - startTime; "granularity of the value depends on the underlying operating system and may be larger" "resolution is at least as good as that of currentTimeMillis()" Source • http://docs.oracle.com/javase/7/docs/api/java/lang/System.html
  • 24. General Principal #1 Know what you are measuring
  • 25. #2 Response time to return all results BasicDBObject doc = new BasicDBObject(); doc.put("v", str); // str is a 2k string for (int i=0; i < 1000; i++) { doc.put("_id",i); coll.insert(doc); } BasicDBObject predicate = new BasicDBObject(); long startTime = System.currentTimeMillis(); DBCursor cur = coll.find(predicate); DBObject foundObj; while (cur.hasNext()) { foundObj = cur.next(); } long endTime = System.currentTimeMillis();
  • 26. #2 Response time to return all results BasicDBObject doc = new BasicDBObject(); doc.put("v", str); // str is a 2k string for (int i=0; i < 1000; i++) { doc.put("_id",i); coll.insert(doc); } BasicDBObject predicate = new BasicDBObject(); long startTime = System.currentTimeMillis(); DBCursor cur = coll.find(predicate); DBObject foundObj; while (cur.hasNext()) { foundObj = cur.next(); } long endTime = System.currentTimeMillis();
  • 27. #2 Response time to return all results BasicDBObject doc = new BasicDBObject(); doc.put("v", str); // str is a 2k string for (int i=0; i < 1000; i++) { doc.put("_id",i); coll.insert(doc); } BasicDBObject predicate = new BasicDBObject(); long startTime = System.currentTimeMillis(); DBCursor cur = coll.find(predicate); DBObject foundObj; while (cur.hasNext()) { foundObj = cur.next(); } long endTime = System.currentTimeMillis();
  • 28. #2 Response time to return all results BasicDBObject doc = new BasicDBObject(); doc.put("v", str); // str is a 2k string for (int i=0; i < 1000; i++) { doc.put("_id",i); coll.insert(doc); } BasicDBObject predicate = new BasicDBObject(); long startTime = System.currentTimeMillis(); DBCursor cur = coll.find(predicate); DBObject foundObj; while (cur.hasNext()) { foundObj = cur.next(); } long endTime = System.currentTimeMillis();
  • 29. So that looks ok, right? BasicDBObject doc = new BasicDBObject(); doc.put("v", str); // str is a 2k string for (int i=0; i < 1000; i++) { doc.put("_id",i); coll.insert(doc); } BasicDBObject predicate = new BasicDBObject(); long startTime = System.currentTimeMillis(); DBCursor cur = coll.find(predicate); DBObject foundObj; while (cur.hasNext()) { foundObj = cur.next(); } long endTime = System.currentTimeMillis();
  • 30. What else are you measuring? BasicDBObject doc = new BasicDBObject(); doc.put("v", str); // str is a 2k string for (int i=0; i < 1000; i++) { doc.put("_id",i); coll.insert(doc); } BasicDBObject predicate = new BasicDBObject(); long startTime = System.currentTimeMillis(); DBCursor cur = coll.find(predicate); DBObject foundObj; while (cur.hasNext()) { foundObj = cur.next(); } long endTime = System.currentTimeMillis(); Each doc is is 4080 bytes on disk with powerOf2Sizes
  • 31. What else are you measuring? BasicDBObject doc = new BasicDBObject(); doc.put("v", str); // str is a 2k string for (int i=0; i < 1000; i++) { doc.put("_id",i); coll.insert(doc); } BasicDBObject predicate = new BasicDBObject(); long startTime = System.currentTimeMillis(); DBCursor cur = coll.find(predicate); DBObject foundObj; while (cur.hasNext()) { foundObj = cur.next(); } long endTime = System.currentTimeMillis(); Each doc is is 4080 bytes on disk with powerOf2Sizes Unrestricted predicate?
  • 32. What else are you measuring? BasicDBObject doc = new BasicDBObject(); doc.put("v", str); // str is a 2k string for (int i=0; i < 1000; i++) { doc.put("_id",i); coll.insert(doc); } BasicDBObject predicate = new BasicDBObject(); long startTime = System.currentTimeMillis(); DBCursor cur = coll.find(predicate); DBObject foundObj; while (cur.hasNext()) { foundObj = cur.next(); } long endTime = System.currentTimeMillis(); Each doc is is 4080 bytes on disk with powerOf2Sizes Unrestricted predicate? Measuring • Time to parse & execute query • Time to retrieve all document But also • Cost of shipping ~4MB data through network stack
  • 33. Solution: Limit the projection BasicDBObject predicate = new BasicDBObject(); predicate.put("_id", new BasicDBObject("$gte", 10).append("$lte", 20)); BasicDBObject projection = new BasicDBObject(); projection.put("_id", 1); long startTime = System.currentTimeMillis(); DBCursor cur = coll.find(predicate, projection ); DBObject foundObj; while (cur.hasNext()) { foundObj = cur.next(); } long endTime = System.currentTimeMillis(); Return fixed range
  • 34. Solution: Limit the projection BasicDBObject predicate = new BasicDBObject(); predicate.put("_id", new BasicDBObject("$gte", 10).append("$lte", 20)); BasicDBObject projection = new BasicDBObject(); projection.put("_id", 1); long startTime = System.currentTimeMillis(); DBCursor cur = coll.find(predicate, projection ); DBObject foundObj; while (cur.hasNext()) { foundObj = cur.next(); } long endTime = System.currentTimeMillis(); Return fixed range Only project _id
  • 35. Solution: Limit the projection BasicDBObject predicate = new BasicDBObject(); predicate.put("_id", new BasicDBObject("$gte", 10).append("$lte", 20)); BasicDBObject projection = new BasicDBObject(); projection.put("_id", 1); long startTime = System.currentTimeMillis(); DBCursor cur = coll.find(predicate, projection ); DBObject foundObj; while (cur.hasNext()) { foundObj = cur.next(); } long endTime = System.currentTimeMillis(); Return fixed range Only project _id Only 46k transferred through network stack
  • 36. General Principal #2 Measure only what you need to measure
  • 37. Part Two Performance measurement & diagnosis
  • 38. Broad categories • Micro Benchmarks • Workloads
  • 40. mongo-perf: goals • Measure – commands • Configure – Single mongod, ReplSet size (1 -> n), Sharding – Single vs. Multiple DB – O/S • Characterize – Throughput (ops/second) by thread count • Compare
  • 41. What do you get? Better Thread Count Ops/Se c
  • 42. What do you get? Measured improvement between rc0 and rc2 Better
  • 43. Benchmark source code tests.push( { name: "Commands.CountsIntIDRange", pre: function( collection ) { collection.drop(); for ( var i = 0; i < 1000; i++ ) { collection.insert( { _id : i } ); } collection.getDB().getLastError(); }, ops: [ { op: "command", ns : "testdb", command : { count : "mycollection", query : { _id : { "$gt" : 10, "$lt" : 100 } } } } ] } );
  • 44. Benchmark source code tests.push( { name: "Commands.CountsIntIDRange", pre: function( collection ) { collection.drop(); for ( var i = 0; i < 1000; i++ ) { collection.insert( { _id : i } ); } collection.getDB().getLastError(); }, ops: [ { op: "command", ns : "testdb", command : { count : "mycollection", query : { _id : { "$gt" : 10, "$lt" : 100 } } } } ] } );
  • 45. Benchmark source code tests.push( { name: "Commands.CountsIntIDRange", pre: function( collection ) { collection.drop(); for ( var i = 0; i < 1000; i++ ) { collection.insert( { _id : i } ); } collection.getDB().getLastError(); }, ops: [ { op: "command", ns : "testdb", command : { count : "mycollection", query : { _id : { "$gt" : 10, "$lt" : 100 } } } } ] } );
  • 46. Benchmark source code tests.push( { name: "Commands.CountsIntIDRange", pre: function( collection ) { collection.drop(); for ( var i = 0; i < 1000; i++ ) { collection.insert( { _id : i } ); } collection.getDB().getLastError(); }, ops: [ { op: "command", ns : "testdb", command : { count : "mycollection", query : { _id : { "$gt" : 10, "$lt" : 100 } } } } ] } );
  • 48. Workloads • "public" workloads – YCSB – Sysbench • "real world" simulations – Inbox fan in/out – Message Stores – Content Management
  • 49. Example: Bulk Load Performance 16m Documents Better 55% degradation 2.6.0-rc1 vs 2.4.10
  • 50. Ouch… where's the tree in the woods? • 2.4.10 -> 2.6.0 – 4495 git commits
  • 51. git-bisect • Bisect between good/bad hashes • git-bisect nominates a new githash – Build against githash – Re-run test – Confirm if this githash is good/bad • Rinse and repeat
  • 52. Code Change - Bad Githash
  • 54. Bulk Load Performance - Fix Better 11% improvement 2.6.1 vs 2.4.10
  • 55. The problem with measurement • Observability – What can you observe on the system? • Effect – What effects does the observation cause?
  • 57. mtools • MongoDB log file analysis – Filter logs for operations, events – Response time, lock durations – Plot • https://github.com/rueckstiess/mtools
  • 58. Code Change – Yielding Policy
  • 60. Response Times Bulk Insert 2.6.0 vs 2.6.1 Ceiling similar, lower floor resulting in 40% improvement in throughput
  • 61. Secondary effects of Yield policy change Write lock time reduced Order of magnitude reduction of write lock duration
  • 62. Unexpected side effects of measurement? > db.serverStatus() Yes – will cause a read lock to be acquired > db.serverStatus({recordStats:0}) No – lock is not acquired > mongostat Yes - until SERVER-14008 resolved, uses db.serverStatus()
  • 63. CPU sampling • Get an impression of – Call Graphs – CPU time spent on node and called nodes
  • 64. Setup & building with google-profiler > sudo apt-get install google-perftools > sudo apt-get install libunwind7-dev > scons --use-cpu-profiler mongod
  • 65. Start the profiling > mongod –dbpath <…> Note: Do not use –fork > mongo > use admin > db.runCommand({_cpuProfilerStart: {profileFilename: 'foo.prof'}}) Execute some commands that you want to profile > db.runCommand({_cpuProfilerStop: 1})
  • 66. Sample start vs. end of workload
  • 67. Sample start vs. end of workload
  • 69. Public Benchmarks – Not all forks are the same… • YCSB – https://github.com/achille/YCSB • sysbench-mongodb – https://github.com/mdcallag/sysbench-mongodb
  • 70. #MongoDBDays Thank You Andrew Erlichson andrew@mongodb.com VP, Engineering

Editor's Notes

  • #17: Per Java7 documentation http://docs.oracle.com/javase/7/docs/api/java/util/Random.html "Instances of java.util.Random are threadsafe. However, the concurrent use of the same java.util.Random instance across threads may encounter contention and consequent poor performance. Consider instead using ThreadLocalRandom in multithreaded designs."
  • #19: Per Java7 documentation http://docs.oracle.com/javase/7/docs/api/java/util/concurrent/atomic/package-summary.html "The specifications of these methods enable implementations to employ efficient machine-level atomic instructions that are available on contemporary processors. However on some platforms, support may entail some form of internal locking."
  • #20: Per Java7 documentation http://docs.oracle.com/javase/7/docs/api/java/lang/System.html#currentTimeMillis() "Returns the current time in milliseconds. Note that while the unit of time of the return value is a millisecond, the granularity of the value depends on the underlying operating system and may be larger. For example, many operating systems measure time in units of tens of milliseconds."
  • #22: Per Jav7 documentation http://docs.oracle.com/javase/7/docs/api/java/util/concurrent/ThreadLocalRandom.html "A random number generator isolated to the current thread…Use of ThreadLocalRandom is particularly appropriate when multiple tasks (for example, each a ForkJoinTask) use random numbers in parallel in thread pools."
  • #24: Per Java7 documentation http://docs.oracle.com/javase/7/docs/api/java/lang/System.html#nanoTime() "This method provides nanosecond precision, but not necessarily nanosecond resolution (that is, how frequently the value changes) - no guarantees are made except that the resolution is at least as good as that of currentTimeMillis()."
  • #41: Throughput by thread count.
  • #42: Threadcount on the x axis Throughput on the y axis, # ops/second.
  • #43: Client server on the same box
  • #48: Githash https://github.com/mongodb/mongo/commit/d1dc7cf2b213d77103658ccd2ea4816b33a27f6a#diff-7ba76fe024c203ca35087f3b93395acc
  • #51: Use gitbisect here.
  • #53: Githash https://github.com/mongodb/mongo/commit/00f7aeaa25f98de5e66f0759d5b102951a247526#diff-fa99d4a7f4e8efac0787f30c60814eaf
  • #54: Githash https://github.com/mongodb/mongo/commit/68d42de9a958688acbf659dfb651fb699e9d7394#diff-fa99d4a7f4e8efac0787f30c60814eaf
  • #59: Githash https://github.com/mongodb/mongo/commit/00f7aeaa25f98de5e66f0759d5b102951a247526#diff-fa99d4a7f4e8efac0787f30c60814eaf
  • #60: Githash https://github.com/mongodb/mongo/commit/68d42de9a958688acbf659dfb651fb699e9d7394#diff-fa99d4a7f4e8efac0787f30c60814eaf
  • #69: Githash https://github.com/mongodb/mongo/commit/8d43b5cb9949c16452cb8d949c89d94cab9c8bad#diff-264fb70c85a638c671570970f3752bf3