MongoDB World 2019: The Sights (and Smells) of a Bad Query
1 like5,538 views
The document discusses the identification and analysis of inefficient database queries using MongoDB, focusing on code smells that indicate potential problems in the source code. It highlights the importance of using indexing to improve query performance, illustrated with practical examples and metrics from MongoDB Atlas. Additionally, it emphasizes monitoring tools and methods for analyzing query efficiency and performance over time.
![QUERYING
MULTIPLE FIELDS
AND SORTING
{ context: { $in: [“find”, “sort”] }, smell: “good?” }](https://image.slidesharecdn.com/murrayhillday1800amsightsandsmellsofbadqueries-190701161152/85/MongoDB-World-2019-The-Sights-and-Smells-of-a-Bad-Query-20-320.jpg)
![Plotting queries by execution time
... COMMAND [conn416] command data.users command: aggregate {
aggregate: "users", pipeline: [ { $match: {} }, { $skip: 0 }, { $group: { _id:
null, n: { $sum: 1 } } } ], ... planSummary: COLLSCAN keysExamined:0
docsExamined:2585100 cursorExhausted:1 numYields:20196
nreturned:1 reslen:243 ... 1892ms
mlogfilter *.log --markers none | mplotqueries --logscale](https://image.slidesharecdn.com/murrayhillday1800amsightsandsmellsofbadqueries-190701161152/85/MongoDB-World-2019-The-Sights-and-Smells-of-a-Bad-Query-35-320.jpg)
![Plotting queries by docsExamined/n
… COMMAND [conn1136] command data.users … { find: "users",
filter: { age: { $gt: 25.0 }, address.state: "UT" }, sort: { name: 1.0 }, …
planSummary: IXSCAN { age: 1 } cursorid:73909299301
keysExamined:2154391 docsExamined:2154391 hasSortStage:1
numYields:17029 nreturned:101 … 94212ms
mlogfilter *.log --markers none | mplotqueries --type docsExamined/n](https://image.slidesharecdn.com/murrayhillday1800amsightsandsmellsofbadqueries-190701161152/85/MongoDB-World-2019-The-Sights-and-Smells-of-a-Bad-Query-37-320.jpg)
MongoDB World 2019: The Sights (and Smells) of a Bad Query
- 1. Alex Bevilacqua The Sights (and Smells) of a Bad Query @alexbevi
- 2. INTRODUCTION { context: “speaker”, smell: “pleasant” }
- 3. Alex Bevilacqua Technical Services Engineer, MongoDB
- 4. IT’S ME … MARIO! ALEX! • Application Developer • Development Lead • System Architect • Technical Services Engineer • Author • MongoDB Fanboy
- 5. Sights?
- 6. What we’ll be looking at • Generic looks at some queries • Introduction to some tooling • High level discussion • Not focusing on “solutions”
- 7. Smells?
- 8. What stinks? https://en.wikipedia.org/wiki/Code_smell A code smell is any characteristic in the source code of a program that possibly indicates a deeper problem. Code smells are usually not bugs; they are not technically incorrect and do not prevent the program from functioning. Instead, they indicate weaknesses in design that may slow down development or increase the risk of bugs or failures in the future.
- 9. Scenario • Atlas M10 Cluster • 2-3M documents • Generated and imported using a template (mgeneratejs) • Ran some adhoc queries
- 10. SIMPLE QUERY { context: “find”, smell: “good?” }
- 11. Our Dataset
- 12. What’re we looking for? db.users.find({ age: 38 })
- 13. Let’s find some data! • Found 53,516 Results • Took a few seconds longer than we’d like ☹
- 14. Who’s seen this?
- 15. ALERTS =
- 16. Let’s see that again … as an Explain Plan • Query time higher than we would expect • COLLSCAN • Large number of documents scanned (entire collection!)
- 17. COLLSCAN =
- 18. Adding an Index db.users.createIndex({ age: 1 })
- 19. Explain that again please • Performance greatly improved • IXSCAN • Fewer documents scanned
- 20. QUERYING MULTIPLE FIELDS AND SORTING { context: { $in: [“find”, “sort”] }, smell: “good?” }
- 21. What’re we looking for? db.users.find({ age: { $gt: 25 }, “address.state”: “UT” }).sort({ name: 1 })
- 22. Let’s find some data – in order this time • Looking for everyone older than 25 from Utah, sorted by name • IXSCAN …. But slow • Far more keys and documents examined than returned (we’ll come back to this one) • In memory sort!
- 23. SORT =
- 24. We know we need an index!
- 25. Let’s try that again …
- 26. What’s the explain telling us? • IXSCAN using the expected index • Faster execution • Same number of documents scanned and returned … • In memory sort???
- 27. (E)quality – (S)ort – (R)ange db.users.find({ age: { $gt: 25 }, ‘address.state’: ‘UT’ }).sort({ name: 1 }) Rang e Equality Sort
- 29. METRICS AND LOGS AND CHARTS … OH MY! { context: “analysis”, smell: “good?” }
- 30. What can the Atlas Metrics tell us? • Sort operation that couldn’t use an index • Ratio of documents scanned to the number of documents returned • Increased average read/write time per operation
- 31. Sharp rise in one or = more metrics
- 32. Remember this?
- 33. Check the Performance Advisor
- 34. Get your logs
- 35. Plotting queries by execution time ... COMMAND [conn416] command data.users command: aggregate { aggregate: "users", pipeline: [ { $match: {} }, { $skip: 0 }, { $group: { _id: null, n: { $sum: 1 } } } ], ... planSummary: COLLSCAN keysExamined:0 docsExamined:2585100 cursorExhausted:1 numYields:20196 nreturned:1 reslen:243 ... 1892ms mlogfilter *.log --markers none | mplotqueries --logscale
- 37. Plotting queries by docsExamined/n … COMMAND [conn1136] command data.users … { find: "users", filter: { age: { $gt: 25.0 }, address.state: "UT" }, sort: { name: 1.0 }, … planSummary: IXSCAN { age: 1 } cursorid:73909299301 keysExamined:2154391 docsExamined:2154391 hasSortStage:1 numYields:17029 nreturned:101 … 94212ms mlogfilter *.log --markers none | mplotqueries --type docsExamined/n
- 39. What can stink? • Alerts (from Atlas, Ops Manager, Cloud Manager) • COLLSCAN (in logs or explain plans) • SORT (in explain plans or in logs as hasSortStage) • Non-ESR Compound Indexes • Sharp rise in one or more metrics (in Atlas, Ops Manager or Cloud Manager) • Queries slower than slowms • Large docsExamined/n ratio
- 40. Links Notes/links from this talk available at: http://bit.ly/ab-mdbw19 alexbevi alexbevi @alexbevi
- 42. THANK YOU FOR ATTENDING! QUESTIONS?