Silentale mongo slides
0 likes295 views
The document discusses Silentale, a service that backs up contacts and messages from multiple accounts. It indexes messages and extracts metadata to store in databases like MySQL and MongoDB. While MongoDB scales to handle large volumes of data, searches can be slow due to I/O issues. Solutions may involve separating data by intent into different databases and moving to specialized hardware.
Silentale mongo slides
- 1. keep track and backup your contacts & conversations Erwan Arzur erwan@silentale.com +33 6 09 33 76 42 http://silentale.com vendredi 18 juin 2010
- 2. Silentale in a few words • Backup and indexes messages from all accounts • Creates the user’s social address book • Freemium service - free and paid plans Silentale June 18, 2010 2 vendredi 18 juin 2010
- 3. Silentale One convenient place to backup all your contacts and messages across multiple services Silentale June 18, 2010 3 vendredi 18 juin 2010
- 4. Find what you need, faster • Retrieve contact details, messages or attachments, across all your accounts • Complete, consolidated message history • Unified address book • Wherever you need it (mobile, browser, desktop clients) Silentale June 18, 2010 4 vendredi 18 juin 2010
- 5. How does it work? Contacts & Messages from everywhere Silentale June 18, 2010 5 vendredi 18 juin 2010
- 6. How does it work? Contacts & Messages Applications from everywhere API 3rd party apps Silentale June 18, 2010 5 vendredi 18 juin 2010
- 7. Distributed platform • Crawlers retrieve messages & contacts • Store raw messages in S3 buckets • Parsers extract useful information • update databases (MySQL & MongoDB) • message meta-data • Indexers • full-text index • MongoDB • Message Queue service Silentale June 18, 2010 6 vendredi 18 juin 2010
- 8. Mongo Setup • 1 collection / user • sharding • current sharding implementation doesn’t work for us • very crude custom sharding algorithm • replica pairs • seem like a waste of resource ... until it’s maintenance time • m1.x-large instances • 15 GB RAM • 8x70 GB EBS volumes in raid0 (xfs) Silentale June 18, 2010 7 vendredi 18 juin 2010
- 9. Our history with Mongo • Our first beta in sep ’09 was using 0.9 • why did we choose mongo ? • more than a simple key-value store • easy port from MySQL (MongoMapper) • store arrays and search in them (rare feature among mongodb competition) • very fast answers from 10gen and community • a few bugs hurt us • ruby driver • replication problems • quickly fixed and has to be expected when deploying an early release ! • Now using 1.4.3 in production Silentale June 18, 2010 8 vendredi 18 juin 2010
- 10. Scales • ~100M documents • ~1.5 TB of data • >1.5 TB of indices • 15GB of RAM / shard • I/O bound • first search into a user’s collection can take several minutes (!!) • acceptable performance afterwards, but don’t try to search another user’s collection ! Silentale June 18, 2010 9 vendredi 18 juin 2010
- 11. Scales • ~100M documents • ~1.5 TB of data • >1.5 TB of indices • 15GB of RAM / shard Too Slow ! • I/O bound • first search into a user’s collection can take several minutes (!!) • acceptable performance afterwards, but don’t try to search another user’s collection ! Silentale June 18, 2010 9 vendredi 18 juin 2010
- 12. Solutions / wish-list • separate intents in different databases • meta-data needs quasi-instant retrieval • full-text doesn’t • simple searches works well thanks to indexing of array values • http://jira.mongodb.org/browse/SERVER-380 • moving to bare-metal • specialized hardware. • cache management Silentale June 18, 2010 10 vendredi 18 juin 2010
- 13. Thank you ! Questions ? Erwan Arzur / erwan@silentale.com / @zuzur Silentale June 18, 2010 11 vendredi 18 juin 2010