Klmug presentation - Simple Analytics with MongoDB
- 1. Simple Analytics with MongoDB
- 2. About Me I’m Ross Affandy. Senior Developer Cum System Administrator at Carlist.MY MongoPress Core Developer
- 3. I will talking about: - Our stack (architecture) - Our problem - Our solution - Our lesson
- 4. Stack in cloud Platform – Linux (Amazon Distro) Database – MongoDB Language – PHP (API) Webserver – NginX (Sorry node.js – I’m not developing event-driven programming or require long pulling persistent connection) Using Amazon EC2 micro instance 600MB RAM 8GB EBS root partition 30GB EBS partition for MongoDB storage (format as xfs filesystem) Why Amazon Cloud? I want to save 70% of my time managing infrastructure and focus to writing code
- 5. Business Analytics Essential - Bank use business analytics to predict & prevent credit card fraud - Retailers use business analytics to predict the best location for store and reach target market - Even sports team use business analytics to determine game strategy and ticket price
- 6. Problem to solve Real time data collection : - Implementing pageview counter - Simple Analytics Why MongoDB? - MySQL usually blocked on file system reads - Good at saving large volume of data - Support asynchronous insert ( fire & forget ) - Fast access to large binary object - Read/write ratio is highly skewed to reads - Upsert ( simplify my code )
- 8. Data structure and how it look like?
- 9. Now the story begin!
- 10. Problem / Challenge We face many exciting challenges ( expect the unexpected ) Implementation We use map reduce to gather the information that we collect What is map reduce in MongoDB and why we use it? - Equal to count/sum/avg/group by function with MySQL. - Map reduce is easier to understand - Useful to process large dataset concurrently in large cluster of machines (sorry for this, we don’t have budget yet ) Problem Map reduce very slow and crash the server due to the javascript engine and lack of processing power (low RAM and cpu) MongoDB also has a group() function. Why not use it? Group() function only return single bson object (less than 16mb). Not useful for unique data more than 10,000 value
- 16. Moving to aggregation framework Quickly running latest version of MongoDB just to get aggregation function Changing PHP query to using aggregation instead of map reduce Good news Server not crash Bad news Aggregation is better but still need more RAM to process 2 million document. Still slow.
- 18. Experiment Test run on Amazon SSD + 64GB RAM (Virginia) - Copy 12GB data to another amazon EC2 instance - Run the map reduce and aggregation query to see what break. Nothing break. Server look happy Problem Solve? Yes, but server cost is too expensive.
- 19. Solution Denormalization - In computing, denormalization is the process of attempting to optimise the read performance of a database by adding redundant data or by grouping data.In some cases, denormalisation helps cover up the inefficiencies inherent in relational database software. A relational normalised database imposes a heavy access load over physical storage of data even if it is well tuned for high performance. - Copying of the same data into multiple documents or tables in order to simplify/optimize query processing - Be careful about duplicate data that will easier make database big When to denormalize? Query data volume or IO per query VS total data volume. Processing complexity VS total data volume. Now everytime user access the page, we run 2 query. 1) Capture the data for analytics 2) Update other collection to replace group by. Later on will be use to display to user.
- 20. Summary / Lesson learned - We learned what makes MongoDB a good analytics tool - Data modeling is important. What questions do I have? What answers do I have? - Design query before design schema - Simplified everything MapReduce is slower and is not supposed to be used in “real time.” TIPS Always run load / stress test before go live 1) capacity planning 2) capacity testing 3) performance tuning Tools 1) Dex performance tuning tool from mongolab is really helpful https://github.com/mongolab/dex
- 21. It's not about winning, It's all about taking part!
- 22. Contact Website: http://www.carlist.my Email: enquiries@carlist.my We also hiring! jobs@carlist.my
- 23. Q&A?