MongoDB Europe 2016 - Star in a Reasonably Priced Car - Which Driver is Best?
0 likes979 views
The document discusses the role and functionality of MongoDB drivers, highlighting their importance in service discovery, authentication, and wire protocol abstraction. It includes various benchmarks of drivers across different programming languages, measuring operational speeds for create, read, and update operations. The author encourages exploring multiple languages for optimal performance and emphasizes the complexity of designing benchmarks to mitigate bias.
MongoDB Europe 2016 - Star in a Reasonably Priced Car - Which Driver is Best?
- 1. MongoDB’s best driver John Page - Principal Consulting Engineer
- 2. The nature of Drivers
- 3. #MDBE16 What does a MongoDB Driver do? • Idiomatic API • Service Discovery and High Availability • Authentication • Wire protocol version abstraction This is non trivial work – drivers are not thin wrappers.
- 4. #MDBE16 What does a MongoDB Driver do? • Idiomatic API • Service Discovery and High Availability • Authentication • Wire protocol version abstraction
- 5. #MDBE16 What Languages have Drivers Python PerlJavaScript
- 6. #MDBE16 Idiomatic API - Write Data Source Map / Object db.coll.insert(Map) BSON Bytes TCP Data
- 7. #MDBE16 Idiomatic API - Read TCP Data BSON Bytes db.coll.find() Map / Object Field Value
- 8. #MDBE16 Idiomatic API – Read Optimisations TCP Data BSON Bytes db.coll.find() Field Value Lazy BSON
- 9. #MDBE16 High Data Throughput Workloads • Data Loading • Data Conversion • External Analysis / Reporting • Sensors and Internet Things Reads/Writes from 10s to 100s MB/s
- 10. #MDBE16 Motivation for Measuring this • POC Benchmark stories • Shell Loops • YCSB • Consulting Experiences • Application Benchmarks and Sizing
- 11. #MDBE16 Quick Summary • Drivers do a number of things. • Different Languages need different things done. • Languages can vary in costs to write, maintain and run. • We want to know the relative difference for the run part.
- 13. #MDBE16 Things to be aware of • Experience. • It’s a non trivial undertaking. • Bias and Benchmarketing. • Be aware of conscious and unconscious bias. • Seek Assistance and Consensus.
- 14. #MDBE16 Our Client Benchmark CRU (!D) • Insert 1,000,000 ~8K records. • Retrieve 1 field from each from the DB. • Modify each changing each String field. Not using Aggregation, Projection or update modifiers.
- 15. #MDBE16 Architecture Client (Fixed and Smallish) • 4 Xeon 2.5GHz CPU cores • 16GB RAM • 10GB/s Network to server. • 10GB/s switch Server • MongoDB 3.2 • WiredTiger – no journal • Local SSD • 32 Xeon 2.5GHz Cores • 250GB RAM • 10GB/s Network to client.
- 16. #MDBE16 Measurables What numbers are we interested in • Straight line speed. • Single Thread • Create (records/s) • Read (records/s) • Update (records/s) • Max throughput. • Multi Thread/Process (records/s)
- 17. Results
- 18. #MDBE16 Javascript – Mongo Shell and Node.JS 0 10000 20000 30000 40000 50000 60000 70000 80000 90000 Shell Linear Shell Parallel Node.js linear Node.js Parallel Operations per Second Create Read Update
- 19. #MDBE16 Javascript – Mongo Shell and Node.JS 0 10000 20000 30000 40000 50000 60000 70000 80000 90000 Shell Linear Shell Parallel Node.js linear Node.js Parallel Operations per Second Create Read Update
- 20. #MDBE16 Javascript – Mongo Shell and Node.JS 0 5000 10000 15000 20000 25000 30000 Shell Linear Shell Parallel Node.js linear Node.js Parallel Operations per Second Create Read Update
- 21. #MDBE16 Node versus Python– the hipster dichotomy 0 5000 10000 15000 20000 25000 30000 Node.js linear Node.js Parallel Operations per Second Create Read Update
- 22. #MDBE16 Node versus Python– the hipster dichotomy 0 5000 10000 15000 20000 25000 30000 Node.js linear Node.js Parallel Python linear Python parallel Operations per Second Create Read Update
- 23. #MDBE16 Node versus Python with C extensions 0 5000 10000 15000 20000 25000 30000 Node.js linear Node.js Parallel Python w. C linear Python w. C parallel Operations per Second Create Read Update
- 24. #MDBE16 Java 8 – the corporate favourite 0 5000 10000 15000 20000 25000 30000 35000 40000 Java 8 linear Java 8 parallel Operations per Second Create Read Update
- 25. #MDBE16 Java 8 on Linux versus Java 8 on Windows 0 5000 10000 15000 20000 25000 30000 35000 40000 Java 8 linear Java 8 parallel Java 8 linear(win) Java 8 parallel (win) Operations per Second Create Read Update
- 26. #MDBE16 Java 8 on Windows versus C# 0 5000 10000 15000 20000 25000 30000 35000 40000 Java 8 linear(win) Java 8 parallel (win) C# linear C# parallel Operations per Second Create Read Update
- 27. #MDBE16 Java 8 on Windows versus C# 0 5000 10000 15000 20000 25000 30000 35000 40000 Java 8 linear(win) Java 8 parallel (win) C# linear C# parallel Operations per Second Create Read Update
- 28. #MDBE16 C# on Windows versus C# on Linux! 0 5000 10000 15000 20000 25000 30000 35000 40000 C# linear C# parallel Dotnet Core liner DotNet Core Parallel Operations per Second Create Read Update
- 29. #MDBE16 Java 8 on Linux versus Morphia ODM 0 5000 10000 15000 20000 25000 30000 35000 40000 Java 8 linear Java 8 parallel Morphia linear Morphia parallel Operations per Second Create Read Update
- 30. #MDBE16 C – the classic choice 0 20000 40000 60000 80000 100000 120000 140000 160000 180000 C linear C Parallel Operations per Second Create Read Update
- 31. #MDBE16 C – the classic choice versus Go ‘The new C’ 0 20000 40000 60000 80000 100000 120000 140000 160000 180000 C linear C Parallel GoLang Linear GoLang Parallel Operations per Second Create Read Update
- 32. #MDBE16 Java 8 on Linux versus C 0 20000 40000 60000 80000 100000 120000 140000 160000 180000 Java 8 linear Java 8 parallel C linear C Parallel Operation per Second Create Read Update
- 33. #MDBE16 Java 8 on Linux versus PHP 7 – the surprise 0 5000 10000 15000 20000 25000 30000 35000 40000 Java 8 linear Java 8 parallel PHP 7 linear PHP 7 parallel Operations per Second Create Read Update
- 34. #MDBE16 Create speed 0 5,000 10,000 15,000 20,000 25,000 30,000 35,000 40,000 45,000 C Java Threaded PHP 7 Java (on Win) GoLang Morphia Python with C Extensions Node.JS CSharp Operations per Second Create
- 35. #MDBE16 Read speed 0 20,000 40,000 60,000 80,000 100,000 120,000 140,000 160,000 180,000 C Shell PHP 7 Java Threaded Python with C Extensions Java (on Win) GoLang Node.JS Morphia Operations per Second. Read
- 36. What does that mean?
- 37. #MDBE16 Back in the real world • Database backend upgrade for logistics company • Schema conversion • Unacceptable 8 hours of downtime planned • Rewrite in C • 40 mins downtime. • Regular bulk load of JSON Ecommerce data (>500GB) • From 35 minutes using mongoimport ( written in Go ) • To 7 minutes with custom loader.
- 38. #MDBE16 Don’t be British! • If you aren’t using one ‘fast’ language, maybe you should. • If you aren’t using one dynamic language – go experience the joy. • Be a polyglot when you are working with MongoDB. • The API’s are really similar to make it easy. • A single language SLOW and LOUD isn’t always best.