Indexing and Performance Tuning

Indexing and
Performance Tuning
Tom Schreiber
Senior Consulting Engineer, MongoDB

Indexes are the single biggest tunable
performance factor in MongoDB
Mail from a German customer after a MongoDB Performance Tuning Consulting:

{'a': 3,
'b': 20}
{'b': 30}
{'b': 40,
'a': 1 }
{'a': 9,
'b': 4,
'a': 7}
{'a': 1,
'b': 20}
{'a': 11} {'a': 17}
{'a': 5,
'b': 80}
# documents scanned:
1
db.test.find({a:5})
Full collection scan
🔍

{'a': 3,
'b': 20}
{'b': 30}
{'b': 40,
'a': 1 }
{'a': 9,
'b': 4,
'a': 7}
{'a': 1,
'b': 20}
{'a': 11} {'a': 17}
{'a': 5,
'b': 80}
🔍
db.test.find({a:5}) # documents scanned:
2

{'a': 3,
'b': 20}
{'b': 30}
{'b': 40,
'a': 1 }
{'a': 9,
'b': 4,
'a': 7}
{'a': 1,
'b': 20}
{'a': 11} {'a': 17}
{'a': 5,
'b': 80}
🔍
3

{'a': 3,
'b': 20}
{'b': 30}
{'b': 40,
'a': 1 }
{'a': 9,
'b': 4,
'a': 7}
{'a': 1,
'b': 20}
{'a': 11} {'a': 17}
{'a': 5,
'b': 80}
🔍
4

{'a': 3,
'b': 20}
{'b': 30}
{'b': 40,
'a': 1 }
{'a': 9,
'b': 4,
'a': 7}
{'a': 1,
'b': 20}
{'a': 11} {'a': 17}
{'a': 5,
'b': 80}
🔍
5

{'a': 3,
'b': 20}
{'b': 30}
{'b': 40,
'a': 1 }
{'a': 9,
'b': 4,
'a': 7}
{'a': 1,
'b': 20}
{'a': 11} {'a': 17}
{'a': 5,
'b': 80}
🔍
6

{'a': 3,
'b': 20}
{'b': 30}
{'b': 40,
'a': 1 }
{'a': 9,
'b': 4,
'a': 7}
{'a': 1,
'b': 20}
{'a': 11} {'a': 17}
{'a': 5,
'b': 80}
🔍
7

{'a': 3,
'b': 20}
{'b': 30}
{'b': 40,
'a': 1 }
{'a': 9,
'b': 4,
'a': 7}
{'a': 1,
'b': 20}
{'a': 11} {'a': 17}
{'a': 5,
'b': 80}
🔍Time Complexity for searching in a list with n entries: O(n)
8

{'a': 3,
'b': 20}
{'b': 30}
{'b': 40,
'a': 1 }
{'a': 9,
'b': 4,
'a': 7}
{'a': 1,
'b': 20}
{'a': 11} {'a': 17}
{'a': 5,
'b': 80}
1 5 179
7
113
Index on 'a'
Index scan

Indexes are the single biggest tu
{'a': 3,
'b': 20}
{'b': 30}
{'b': 40,
'a': 1 }
{'a': 9,
'b': 4,
'a': 7}
{'a': 1,
'b': 20}
{'a': 11} {'a': 17}
{'a': 5,
'b': 80}
1 5 179
7
113
Index on 'a'
Search for document(s) with 'a': 5
🔍
db.test.find({a:5})
# index entries scanned: 1
Index scan

{'a': 3,
'b': 20}
{'b': 30}
{'b': 40,
'a': 1 }
{'a': 9,
'b': 4,
'a': 7}
{'a': 1,
'b': 20}
{'a': 11} {'a': 17}
{'a': 5,
'b': 80}
1 5 179
7
113
Index on 'a'
🔍
db.test.find({a:5})
Index scan

{'a': 3,
'b': 20}
{'b': 30}
{'b': 40,
'a': 1 }
{'a': 9,
'b': 4,
'a': 7}
{'a': 1,
'b': 20}
{'a': 11} {'a': 17}
{'a': 5,
'b': 80}
1 5 179
7
113
Index on 'a'
🔍
db.test.find({a:5})
# documents scanned: 1
Time Complexity for
searching in a binary
search tree with n nodes:
O(log2 n)
log2 10.000 = 13
log2 100.000 = 16
log2 1.000.000 = 19
Index scan

MongoDB Indexes are B-Trees
[-∞, 5) [5, 10) [10, ∞)
[5, 7) [7, 9) [9, 10) [10, ∞) buckets[-∞, 5) buckets
{...} {...} {...} {...} {...} {...} {...} {...} {...}
Index on a
Searches, Insertions, and Deletions in logarithmic time.

tresult
The Query Optimizer
Chooses the most efficient query plan.
Information on query plans 
and their execution statistics:
db.col.query.explain()
full collection scan
index on x
index on y
t0
Choose and Remenber
Terminate
db.col.query(...)

stage
plan root
stage stage
stage
stage stage
Leaf nodes access the collections or the indices.
The explain results present the query plans as a tree of stages.
Each stage passes its results (i.e. documents or index keys) to the parent node.
The internal nodes manipulate
the documents or the
index keys that result
from the child nodes.
The root node is the final stage from
which MongoDB derives the result set.
db.col.query.explain()

db.test.find({a:5}).explain()
{
"queryPlanner" : {
[...],
"winningPlan" : {
"stage" : "FETCH",
"inputStage" : {
"stage" : "IXSCAN",
"keyPattern" : {
"a" : 5
},
"indexName" : "a_1",
"isMultiKey" : false,
"direction" : "forward",
"indexBounds" : {
"a" : [
"[5.0, 5.0]"
]
}
}
}
},
[...]
}
plan root
stage
FETCH
stage
IXSCAN

{
"queryPlanner" : {
[...],
"winningPlan" : {
"stage" : "FETCH",
"inputStage" : {
"stage" : "IXSCAN",
"keyPattern" : {
"a" : 5
},
"indexName" : "a_1",
"indexBounds" : {
"a" : [
"[5.0, 5.0]"
]
}
}
}
},
[...]
}
stage
FETCH
[-∞, 5) [5, 10) [10, ∞)
[5, 7) [7, 9) [9, 10) [10, ∞) buckets[-∞, 5) buckets
{...} {...} {...} {...} {...} {...} {...} {...} {...}
Index on {a:1}
plan root
stage
IXSCAN

Explain Levels
queryPlanner
"Which plan will MongoDB choose to run my query?"
executionStats
"How is my query performing?"
allPlansExecution
"I want as much information as possible to diagnose a slow query."

stage
FETCH
[-∞, 5) [5, 10) [10, ∞)
[5, 7) [7, 9) [9, 10) [10, ∞) buckets[-∞, 5) buckets
{...} {...} {...} {...} {...} {...} {...} {...} {...}
Index on {a:1}
plan root
stage
IXSCAN

{ 
[...], 
"executionStats" : { 
 
"executionSuccess" : true, 
"nReturned" : 1, 
"executionTimeMillis" : 0, 
"totalKeysExamined" : 1, 
"totalDocsExamined" : 1, 
 
"executionStages" : { 
 
"stage" : "FETCH", 
"nReturned" : 1, 
"executionTime
MillisEstimate" : 0, 
"works" : 2, 
"advanced" : 1, 
"needTime" : 0, 
"needFetch" : 0, 
"saveState" : 0, 
"restoreState" : 0, 
"isEOF" : 1, 
"invalidates" : 0, 
"docsExamined" : 1, 
"alreadyHasObj" : 0, 
 
"inputStage" : { 
 
"stage" : "IXSCAN", 
db.test.find({a:5}).explain("executionStats")
stage
FETCH
[-∞, 5) [5, 10) [10, ∞)
[5, 7) [7, 9) [9, 10) [10, ∞) buckets[-∞, 5) buckets
{...} {...} {...} {...} {...} {...} {...} {...} {...}
Index on {a:1}
plan root
stage
IXSCAN

 
 
"nReturned" : 1, 
"executionTime
"works" : 2, 
"advanced" : 1, 
"needTime" : 0, 
"needFetch" : 0, 
"saveState" : 0, 
"isEOF" : 1, 
 
"inputStage" : { 
 
"nReturned" : 1, 
"executionTime
"works" : 1, 
"advanced" : 1, 
"needTime" : 0, 
"needFetch" : 0, 
"saveState" : 0, 
"isEOF" : 1, 
stage
FETCH
[-∞, 5) [5, 10) [10, ∞)
[5, 7) [7, 9) [9, 10) [10, ∞) buckets[-∞, 5) buckets
{...} {...} {...} {...} {...} {...} {...} {...} {...}
Index on {a:1}
plan root
stage
IXSCAN

 
"inputStage" : { 
 
"nReturned" : 1, 
"executionTime
"works" : 1, 
"advanced" : 1, 
"needTime" : 0, 
"needFetch" : 0, 
"saveState" : 0, 
"isEOF" : 1, 
"keyPattern" : { 
"a" : 5 
}, 
"indexName" : "a_1", 
"isMultiKey" : false, 
"direction" : "forward", 
"indexBounds" : { 
"a" : [ 
"[5.0, 5.0]" 
] 
}, 
"keysExamined" : 1, 
"dupsTested" : 0, 
"dupsDropped" : 0, 
"seenInvalidated" : 0, 
"matchTested" : 0 
stage
FETCH
[-∞, 5) [5, 10) [10, ∞)
[5, 7) [7, 9) [9, 10) [10, ∞) buckets[-∞, 5) buckets
{...} {...} {...} {...} {...} {...} {...} {...} {...}
Index on {a:1}
plan root
stage
IXSCAN

{ 
[...], 
 
"nReturned" : 1, 
 
 
"nReturned" : 1, 
"executionTime
"works" : 2, 
"advanced" : 1, 
"needTime" : 0, 
"needFetch" : 0, 
"saveState" : 0, 
"isEOF" : 1, 
 
"inputStage" : { 
 
Explain() method key metrics
[-∞, 5) [5, 10) [10, ∞)
[5, 7) [7, 9) [9, 10) [10, ∞) buckets[-∞, 5) buckets
{...} {...} {...} {...} {...} {...} {...} {...} {...}
Index on {a:1}

{ 
[...], 
 
"nReturned" : 1, 
 
 
"nReturned" : 1, 
"executionTime
"works" : 2, 
"advanced" : 1, 
"needTime" : 0, 
"needFetch" : 0, 
"saveState" : 0, 
"isEOF" : 1, 
 
"inputStage" : { 
 
[-∞, 5) [5, 10) [10, ∞)
[5, 7) [7, 9) [9, 10) [10, ∞) buckets[-∞, 5) buckets
{...} {...} {...} {...} {...} {...} {...} {...} {...}
Index on {a:1}
"nReturned" : 1
"executionTimeMillis" : 0
"totalKeysExamined" : 1
"totalDocsExamined" : 1
# documents returned
How long did the query take
# index entries scanned
# documents scanned

Index used? Which one?
 
"inputStage" : { 
 
"nReturned" : 1, 
"executionTime
"works" : 1, 
"advanced" : 1, 
"needTime" : 0, 
"needFetch" : 0, 
"saveState" : 0, 
"isEOF" : 1, 
"keyPattern" : { 
"a" : 5 
}, 
"indexName" : "a_1", 
"isMultiKey" : false, 
"direction" : "forward", 
"indexBounds" : { 
"a" : [ 
"[5.0, 5.0]" 
] 
}, 
"keysExamined" : 1, 
"dupsTested" : 0, 
"dupsDropped" : 0, 
"seenInvalidated" : 0, 
"matchTested" : 0 
} 
"IXSCAN"
"indexName" : "a_1"
[-∞, 5) [5, 10) [10, ∞)
[5, 7) [7, 9) [9, 10) [10, ∞) buckets[-∞, 5) buckets
{...} {...} {...} {...} {...} {...} {...} {...} {...}
Index on {a:1}
# documents returned
How long did the query take
# index entries scanned
# documents scanned

Performance Tuning
with Indexes

Sample Data
db.zips.find() zips.json
{ "zip" : "01001", "city" : "AGAWAM", "loc" : [ -72.622739, 42.070206 ], "pop" : 15338, "state" : "MA" }
{ "zip" : "01002", "city" : "CUSHMAN", "loc" : [ -72.51564999999999, 42.377017 ], "pop" : 36963, "state" : "MA" }
{ "zip" : "01005", "city" : "BARRE", "loc" : [ -72.10835400000001, 42.409698 ], "pop" : 4546, "state" : "MA" }
{ "zip" : "01007", "city" : "BELCHERTOWN", "loc" : [ -72.41095300000001, 42.275103 ], "pop" : 10579, "state" : "MA" }
{ "zip" : "01008", "city" : "BLANDFORD", "loc" : [ -72.936114, 42.182949 ], "pop" : 1240, "state" : "MA" }
{ "zip" : "01010", "city" : "BRIMFIELD", "loc" : [ -72.188455, 42.116543 ], "pop" : 3706, "state" : "MA" }
{ "zip" : "01011", "city" : "CHESTER", "loc" : [ -72.988761, 42.279421 ], "pop" : 1688, "state" : "MA" }
{ "zip" : "01012", "city" : "CHESTERFIELD", "loc" : [ -72.833309, 42.38167 ], "pop" : 177, "state" : "MA" }
{ "zip" : "01013", "city" : "CHICOPEE", "loc" : [ -72.607962, 42.162046 ], "pop" : 23396, "state" : "MA" }
{ "zip" : "01020", "city" : "CHICOPEE", "loc" : [ -72.576142, 42.176443 ], "pop" : 31495, "state" : "MA" }
{ "zip" : "01022", "city" : "WESTOVER AFB", "loc" : [ -72.558657, 42.196672 ], "pop" : 1764, "state" : "MA" }
{ "zip" : "01026", "city" : "CUMMINGTON", "loc" : [ -72.905767, 42.435296 ], "pop" : 1484, "state" : "MA" }
{ "zip" : "01027", "city" : "MOUNT TOM", "loc" : [ -72.67992099999999, 42.264319 ], "pop" : 16864, "state" : "MA" }
{ "zip" : "01028", "city" : "EAST LONGMEADOW", "loc" : [ -72.505565, 42.067203 ], "pop" : 13367, "state" : "MA" }
{ "zip" : "01030", "city" : "FEEDING HILLS", "loc" : [ -72.675077, 42.07182 ], "pop" : 11985, "state" : "MA" }
{ "zip" : "01031", "city" : "GILBERTVILLE", "loc" : [ -72.19858499999999, 42.332194 ], "pop" : 2385, "state" : "MA" }
{ "zip" : "01032", "city" : "GOSHEN", "loc" : [ -72.844092, 42.466234 ], "pop" : 122, "state" : "MA" }
{ "zip" : "01033", "city" : "GRANBY", "loc" : [ -72.52000099999999, 42.255704 ], "pop" : 5526, "state" : "MA" }
{ "zip" : "01034", "city" : "TOLLAND", "loc" : [ -72.908793, 42.070234 ], "pop" : 1652, "state" : "MA" }
{ "zip" : "01035", "city" : "HADLEY", "loc" : [ -72.571499, 42.36062 ], "pop" : 4231, "state" : "MA" }
{ "zip" : "01036", "city" : "HAMPDEN", "loc" : [ -72.43182299999999, 42.064756 ], "pop" : 4709, "state" : "MA" }
{ "zip" : "01038", "city" : "HATFIELD", "loc" : [ -72.61673500000001, 42.38439 ], "pop" : 3184, "state" : "MA" }
{ "zip" : "01039", "city" : "HAYDENVILLE", "loc" : [ -72.70317799999999, 42.381799 ], "pop" : 1387, "state" : "MA" }
{ "zip" : "01040", "city" : "HOLYOKE", "loc" : [ -72.626193, 42.202007 ], "pop" : 43704, "state" : "MA" }
{ "zip" : "01050", "city" : "HUNTINGTON", "loc" : [ -72.873341, 42.265301 ], "pop" : 2084, "state" : "MA" }
{ "zip" : "01053", "city" : "LEEDS", "loc" : [ -72.70340299999999, 42.354292 ], "pop" : 1350, "state" : "MA" }
{ "zip" : "01054", "city" : "LEVERETT", "loc" : [ -72.499334, 42.46823 ], "pop" : 1748, "state" : "MA" }
{ "zip" : "01056", "city" : "LUDLOW", "loc" : [ -72.471012, 42.172823 ], "pop" : 18820, "state" : "MA" }
{ "zip" : "01057", "city" : "MONSON", "loc" : [ -72.31963399999999, 42.101017 ], "pop" : 8194, "state" : "MA" }
{ "zip" : "01060", "city" : "FLORENCE", "loc" : [ -72.654245, 42.324662 ], "pop" : 27939, "state" : "MA" }
{ "zip" : "01068", "city" : "OAKHAM", "loc" : [ -72.051265, 42.348033 ], "pop" : 1503, "state" : "MA" }
{ "zip" : "01069", "city" : "PALMER", "loc" : [ -72.328785, 42.176233 ], "pop" : 9778, "state" : "MA" }
{ "zip" : "01070", "city" : "PLAINFIELD", "loc" : [ -72.918289, 42.514393 ], "pop" : 571, "state" : "MA" }
{ "zip" : "01071", "city" : "RUSSELL", "loc" : [ -72.840343, 42.147063 ], "pop" : 608, "state" : "MA" }
{ "zip" : "01072", "city" : "SHUTESBURY", "loc" : [ -72.421342, 42.481968 ], "pop" : 1533, "state" : "MA" }
{ "zip" : "01073", "city" : "SOUTHAMPTON", "loc" : [ -72.719381, 42.224697 ], "pop" : 4478, "state" : "MA" }
{ "zip" : "01075", "city" : "SOUTH HADLEY", "loc" : [ -72.581137, 42.237537 ], "pop" : 16699, "state" : "MA" }
29353 documents

Question
Zip codes in New York City with population more than
100,000 ordered by population in descending order
db.zips.find({state:'NY',city:'NEW YORK',pop:{'$gt':100000}})
.sort({pop:-1})
{"zip" : "10021", "city" : "NEW YORK", "pop" : 106564, "state" : "NY" }
{ "zip" : "01256", "city" : "SAVOY", "loc" : [ -73.023281, 42.576964 ], "pop" : 632, "state" : "MA" }
{ "zip" : "01257", "city" : "SHEFFIELD", "loc" : [ -73.361091, 42.100102 ], "pop" : 1839, "state" : "MA" }
{ "zip" : "01258", "city" : "SOUTH EGREMONT", "loc" : [ -73.456575, 42.101153 ], "pop" : 135, "state" : "MA" }
{ "zip" : "01259", "city" : "SOUTHFIELD", "loc" : [ -73.26093299999999, 42.078014 ], "pop" : 622, "state" : "MA" }
{ "zip" : "01262", "city" : "STOCKBRIDGE", "loc" : [ -73.32226300000001, 42.30104 ], "pop" : 2200, "state" : "MA" }
{ "zip" : "01266", "city" : "WEST STOCKBRIDGE", "loc" : [ -73.38251, 42.334752 ], "pop" : 1173, "state" : "MA" }
{ "zip" : "01267", "city" : "WILLIAMSTOWN", "loc" : [ -73.20363999999999, 42.708883 ], "pop" : 8220, "state" : "MA" }
{ "zip" : "01270", "city" : "WINDSOR", "loc" : [ -73.04661, 42.509494 ], "pop" : 770, "state" : "MA" }
{ "zip" : "01301", "city" : "LEYDEN", "loc" : [ -72.60184700000001, 42.601222 ], "pop" : 18968, "state" : "MA" }
{ "zip" : "01330", "city" : "ASHFIELD", "loc" : [ -72.810998, 42.523207 ], "pop" : 1535, "state" : "MA" }
{ "zip" : "01331", "city" : "NEW SALEM", "loc" : [ -72.21464400000001, 42.592065 ], "pop" : 14077, "state" : "MA" }
{ "zip" : "01337", "city" : "LEYDEN", "loc" : [ -72.563439, 42.683784 ], "pop" : 2426, "state" : "MA" }

.sort({pop:-1})
Query Plan and Execution Statistics
.explain("executionStats")
plan root
{
"queryPlanner" : {
"winningPlan" : {
"stage" : "SORT",
"sortPattern" : {
"pop" : -1
},
"inputStage" : {
"stage" : "COLLSCAN",
"filter" : {
"$and" : [
{
"city" : {
"$eq" : "NEW YORK"
}
},
{
"state" : {
"$eq" : "NY"
}

.sort({pop:-1}).explain("executionStats")
plan root
SORT
{
"queryPlanner" : {
"winningPlan" : {
"stage" : "SORT",
"sortPattern" : {
"pop" : -1
},
"inputStage" : {
"filter" : {
"$and" : [
{
"city" : {
"$eq" : "NEW YORK"
}
},
{
"state" : {
"$eq" : "NY"
}

{
"queryPlanner" : {
"winningPlan" : {
"stage" : "SORT",
"sortPattern" : {
"pop" : -1
},
"inputStage" : {
"filter" : {
"$and" : [
{
"city" : {
"$eq" : "NEW YORK"
}
},
{
"state" : {
"$eq" : "NY"
}
plan root
SORT
COLLSCAN {...} {...} {...} {...} {...} {...} {...} {...} {...}
Collection Scan!

}
},
{
"pop" : {
"$gt" : 100000
}
}
]
},
"direction" : "forward"
}
},
"rejectedPlans" : [ ]
},
"executionStats" : {
"executionSuccess" : true,
"nReturned" : 2,
"executionTimeMillis" : 15,
"totalKeysExamined" : 0,
"totalDocsExamined" : 29353,
"executionStages" : {
"stage" : "SORT",
"nReturned" : 2,
"executionTimeMillisEstimate" : 0,
"works" : 29359,
"advanced" : 2,
"needTime" : 29355,
"needFetch" : 0,
"saveState" : 229,
"restoreState" : 229,
"isEOF" : 1,
"invalidates" : 0,
"sortPattern" : {
"pop" : -1
.explain("executionStats")
{...} {...} {...} {...} {...} {...} {...} {...} {...}
2 documents returned
29353 documents scanned
plan root
SORT
COLLSCAN
In-Memory Sort!
.sort({pop:-1})
Collection Scan!

Single Field Index
db.zips.createIndex({state:1})
{
"queryPlanner": {
"winningPlan": {
"stage": "SORT",
"sortPattern": {
"pop": -1
},
"inputStage": {
"stage": "FETCH",
"filter": {
"$and": [
{
"city": {
"$eq": "NEW YORK"
}
},
{
"pop": {
"$gt": 100000
}
plan root
SORT

{
"queryPlanner": {
"winningPlan": {
"stage": "SORT",
"sortPattern": {
"pop": -1
},
"inputStage": {
"stage": "FETCH",
"filter": {
"$and": [
{
"city": {
"$eq": "NEW YORK"
}
},
{
"pop": {
"$gt": 100000
}
Single Field Index
plan root
SORT
FETCH

"pop": -1
},
"inputStage": {
"stage": "FETCH",
"filter": {
"$and": [
{
"city": {
"$eq": "NEW YORK"
}
},
{
"pop": {
"$gt": 100000
}
}
]
},
"inputStage": {
"stage": "IXSCAN",
"keyPattern": {
"state": 1
},
"indexName": "state_1",
"isMultiKey": false,
"direction": "forward",
"indexBounds": {
"state": [
"["NY", "NY"]"
]
}
}
}
},
Single Field Index
plan root
SORT
FETCH
IXSCAN
{...} {...} {...} {...} {...} {...} {...} {...} {...}

},
"indexName": "state_1",
"direction": "forward",
"indexBounds": {
"state": [
"["NY", "NY"]"
]
}
}
}
},
"rejectedPlans": []
},
"executionStats": {
"executionSuccess": true,
"nReturned": 2,
"executionTimeMillis": 6,
"totalKeysExamined": 1595,
"totalDocsExamined": 1595,
"executionStages": {
"stage": "SORT",
"nReturned": 2,
"executionTimeMillisEstimate": 0,
"works": 1600,
"advanced": 2,
"needTime": 1596,
"needFetch": 0,
"saveState": 12,
"restoreState": 12,
"isEOF": 1,
"invalidates": 0,
"sortPattern": {
"pop": -1
Single Field Index
plan root
SORT
FETCH
IXSCAN
1595 index entries scanned
In-Memory Sort!
{...} {...} {...} {...} {...} {...} {...} {...} {...}

{ 
"queryPlanner": { 
"winningPlan": { 
"stage": "SORT", 
"sortPattern": { 
"pop": -1 
}, 
"inputStage": { 
"stage": "FETCH", 
"filter": { 
"pop": { 
"$gt": 100000 
} 
}, 
"inputStage": { 
"stage": "IXSCAN", 
"keyPattern": { 
"state": 1, 
"city": 1 
}, 
db.zips.createIndex({state:1, city:1})
Compound Index on two fields
plan root
SORT
FETCH
IXSCAN
{...} {...} {...} {...} {...} {...} {...} {...} {...}

{ 
"winningPlan": { 
"stage": "SORT", 
"sortPattern": { 
"pop": -1 
}, 
"inputStage": { 
"filter": { 
"pop": { 
"$gt": 100000 
} 
}, 
"inputStage": { 
"keyPattern": { 
"state": 1, 
"city": 1 
}, 
"indexName": "state_1_city_1", 
"isMultiKey": false, 
"direction": "forward", 
"indexBounds": { 
"state": [ 
"["NY", "NY"]" 
], 
"city": [ 
"["NEW YORK", "NEW YORK
"]" 
] 
} 
} 
} 
plan root
SORT
FETCH
IXSCAN
{...} {...} {...} {...} {...} {...} {...} {...} {...}

"direction": "forward", 
"indexBounds": { 
"state": [ 
"["NY", "NY"]" 
], 
"city": [ 
"["NEW YORK", "NEW YORK
"]" 
] 
} 
} 
} 
} 
}, 
"executionStats": { 
"executionSuccess": true, 
"nReturned": 2, 
"executionTimeMillis": 4, 
"totalKeysExamined": 40, 
"totalDocsExamined": 40, 
"executionStages": { 
"stage": "SORT", 
"nReturned": 2, 
"executionTimeMillisEstimate": 4, 
"works": 45, 
"advanced": 2, 
"needTime": 41, 
"needFetch": 0, 
"saveState": 0, 
"restoreState": 0, 
"isEOF": 1, 
"invalidates": 0, 
"sortPattern": { 
"pop": -1 
plan root
SORT
FETCH
IXSCAN
{...} {...} {...} {...} {...} {...} {...} {...} {...}
In-Memory Sort!

{ 
"winningPlan": { 
"inputStage": { 
"keyPattern": { 
"state": 1, 
"city": 1, 
"pop": 1 
}, 
"indexName": "state_1_city_1_pop_1", 
"isMultiKey": false, 
"direction": "backward", 
"indexBounds": { 
"state": [ 
"["NY", "NY"]" 
], 
"city": [ 
"["NEW YORK", "NEW YORK"]" 
IXSCAN
FETCH
db.zips.createIndex({state:1,city:1,pop:1})
Compound Index on three fields
plan root
{...} {...} {...} {...} {...} {...} {...} {...} {...}

"indexBounds": { 
"state": [ 
"["NY", "NY"]" 
], 
"city": [ 
"["NEW YORK", "NEW YORK"]" 
], 
"pop": [ 
"[inf.0, 100000.0)" 
] 
} 
} 
} 
}, 
"executionStats": { 
"executionSuccess": true, 
"nReturned": 2, 
"executionTimeMillis": 2, 
"totalKeysExamined": 2, 
"totalDocsExamined": 2, 
"executionStages": { 
"nReturned": 2, 
"executionTimeMillisEstimate": 1, 
"works": 3, 
"advanced": 2, 
"needTime": 0, 
"needFetch": 0, 
"saveState": 0, 
"restoreState": 0, 
"isEOF": 1, 
"invalidates": 0, 
"docsExamined": 2, 
"alreadyHasObj": 0, 
IXSCAN
FETCH
Compound Index on three fields
plan root
{...} {...} {...} {...} {...} {...} {...} {...} {...}
2 documents scanned
Index used for Sorting!

.sort({pop:-1})
Projection on Index-only Data

.sort({pop:-1})

db.zips.find({state:'NY',city:'NEW YORK’,pop:{‘$gt':100000}},
{state:1, city:1, pop:1})
.sort({pop:-1})
{"state" : "NY", "city" : "NEW YORK", "pop" : 106564 }
{"state" : "NY", "city" : "NEW YORK", "pop" : 100027 }

{
"queryPlanner": {
"winningPlan": {
"stage": "PROJECTION",
"transformBy": {
"state": 1,
"city": 1,
"pop": 1,
"zip": 0
},
"inputStage": {
"stage": "IXSCAN",
"keyPattern": {
"state": 1,
"city": 1,
"pop": 1
},
"indexName": "state_1_city_1_pop_1",
IXSCAN
PROJECTION
plan root

"direction": "backward",
"indexBounds": {
"state": [
"["NY", "NY"]"
],
"city": [
"["NEW YORK", "NEW YORK"]"
],
"pop": [
"[inf.0, 100000.0)"
]
}
}
}
},
"executionStats": {
"executionSuccess": true,
"nReturned": 2,
"executionTimeMillis": 0,
"totalKeysExamined": 2,
"totalDocsExamined": 0,
"executionStages": {
"stage": "PROJECTION",
"nReturned": 2,
"executionTimeMillisEstimate": 0,
"works": 3,
"advanced": 2,
"needTime": 0,
"needFetch": 0,
"saveState": 0,
"restoreState": 0,
"isEOF": 1,
"invalidates": 0,
IXSCAN
PROJECTION
plan root 2 documents returned
0 documents scanned
"Covered Query "

{
"executionStats" : {
"executionSuccess" : true,
"nReturned" : 27757,
"executionTimeMillis" : 156,
"totalKeysExamined" : 27759,
"totalDocsExamined" : 27757,
"executionStages" : {
"stage" : "SORT",
"works" : 55519,
"advanced" : 27757,
"needTime" : 27760,
"needFetch" : 0,
"saveState" : 433,
Negation
db.zips.find({state:{$ne:’NY'},city:'NEW YORK’,pop:{'$gt':100000}})
.sort({pop:-1})
db.zips.find({state:{$ne:’NY'},
city:{$ne:'NEW YORK’},
pop:{$not:{‘$gt':100000}}})
plan root
SORT
FETCH
IXSCAN
{...} {...} {...} {...} {...} {...} {...} {...} {...}
In-Memory Sort!

"alreadyHasObj" : 0,
"inputStage" : {
"stage" : "IXSCAN",
"works" : 27759,
"advanced" : 27757,
"needTime" : 2,
"needFetch" : 0,
"saveState" : 433,
"restoreState" : 433,
"isEOF" : 1,
"invalidates" : 0,
"keyPattern" : {
"state" : 1,
"city" : 1,
"pop" : 1
},
"indexName" : "state_1_city_1_pop_1",
"indexBounds" : {
"state" : [
"[MinKey, "NY")",
"("NY", MaxKey]"
],
"city" : [
"[MinKey, "NEW YORK")",
"("NEW YORK", MaxKey]"
],
"pop" : [
"[MinKey, 100000.0]",
"(inf.0, MaxKey]"
]
db.zips.find({state:{$ne:’NY'},city:'NEW YORK’,pop:{'$gt':100000}})
.sort({pop:-1})
db.zips.find({state:{$ne:’NY'},
city:{$ne:'NEW YORK’},
pop:{$not:{‘$gt':100000}}})
In-Memory Sort!
Negation
plan root
SORT
FETCH
IXSCAN
{...} {...} {...} {...} {...} {...} {...} {...} {...}

Memory Contention
{...} {...} {...} {...} {...} {...} {...} {...} {...}
Main Memory (RAM)
⚙mongod process

{...} {...} {...} {...} {...} {...} {...} {...} {...}
Main Memory (RAM)
⚙mongod process
Memory Contention

{...} {...} {...} {...} {...} {...} {...} {...}
Main Memory (RAM)
⚙mongod process
Memory Contention

Reduce Memory Contention
Main Memory (RAM)
{...} {...} {...} {...} {...} {...} {...} {...} {...}
db.zips.find({state:'NY',city:'NEW YORK’,pop:{'$gt':100000}})
.sort({pop:-1})
db.zips.createIndex({state:1, city:1, pop:1})
db.zips.dropIndex({state:1, city:1})
db.zips.dropIndex({state:1})
Remove unneeded indexes

Right-Balanced Index Access
Random Index Access
The entire index is “hot”
The entire index must fit into RAM
Right-Balanced Index Access
Spatial Data Locality
Amount of Index Data in RAM
is significantly reduced

mongod --dbpath DBPATH --storageEngine wiredTiger
--wiredTigerIndexPrefixCompression
Index compression
Prefix Compression for Indexes on disk and in RAM

Reduce Data Transfer Latency
Indexes on a separate storage device
mongod --dbpath DBPATH --storageEngine wiredTiger
--wiredTigerDirectoryForIndexes
One file per Collection under DBPATH/collection
One file per Index under DBPATH/collection
Indexes can be placed on a dedicated
storage device (e.g. SSD) for higher performance

There is more
Multikey Indexes
Text Indexes
Geospatial Indexes
TTL Indexes
Unique Indexes
. . .
Hash Indexes
Sparse Indexes
Sharding
Database Profiler
mongod Log Files
mongostat
mongotop
mongoperf
Cloud Manager
Schema Design
Query Patterns

Summary
Indexes are the single biggest tunable performance factor in MongoDB
Create indexes that support your queries
Create highly selective indexes
Remove unneeded indexes
Analyse your query plans with explain()
Use covered queries for maximum read performance
Negation Queries can’t benefit from indexes
Use wiredTiger feature to place indexes on high performance volumes

Go and build something humongous
ThankYou!

#MDBDays
mongodb.com
Get your technical questions answered
Foyer Royal Ballroom, 10:00 - 17:00
By appointment only – register in person in
the Foyer Royal Ballroom

MongoDB
Performance Measurement Tools

Log files, Profiler, Query Optimizer
mongod
log file
profiler (collection)
query engine

mongod Log Files
Sun Jun 29 06:35:37.646 [conn2]
query test.docs query:
{ parent.company: "22794",
parent.employeeId: "83881" }
ntoreturn:1 ntoskip:0 nscanned:
806381 keyUpdates:0 numYields: 5
locks(micros) r:2145254
nreturned:0 reslen:20 1156ms
date and time thread
operation
namespace
n… 
counters
lock 
times
duration
number of
yields

mtools
• http://github.com/rueckstiess/mtools
• log file analysis for poorly performing queries
– Show me queries that took more than 1000 ms from 6
am to 6 pm:
$ mlogfilter mongodb.log --from 06:00 --to
18:00 --slow 1000 > mongodb-filtered.log

mtools graphs
% mplotqueries --type histogram --group namespace --bucketSize 3600

Database Profiler
• Collect actual samples from a running
MongoDB instance
• Tunable for level and slowness
• Can be controlled dynamically

• Enable to see slow queries
– (or all queries)
– Default 100ms
Using Database profiler
// Enable database profiler on the console, 0=off 1=slow 2=all
> db.setProfilingLevel(1, 50)
{ "was" : 0, "slowms" : 50, "ok" : 1 }
// View profile with
> show profile
// See the raw data
>db.system.profile.find().pretty()

Profiler
• 1MB capped collection named system.profile per database, per replica set
• One document per operation
• Examples:
> db.setProfilingLevel(1) // log all operations greater than 100ms
> db.setProfilingLevel(1, 20) // log all operations greater than 20ms
> db.setProfilingLevel(2) // log all operations regardless of duration
> db.setProfilingLevel(0) // turn off profiling
> db.getProfilingStatus() // display current profiling level
{
"slowms": 100,
"was": 2
}
• In a sharded cluster, you will need to connect to each shard's primary
mongod, not mongos

Command Line tools
• iostat
• dstat
• mongostat
• mongotop
• mongoperf

Cloud Manager
• Memory usage
• Opcounters
• Lock percentage
• Queues
• Background flush average
• Replication oplog window and lag

Indexing and Performance Tuning

More Related Content

What's hot (20)

Viewers also liked (9)

Similar to Indexing and Performance Tuning (20)

More from MongoDB (20)

Recently uploaded (20)

Indexing and Performance Tuning