Scalling Solr in the Cloud - OSCON Data 2011
3 likes1,188 views
The document discusses how Nutshell, a CRM application, scaled their use of Solr for searching and deduplication. They implemented several techniques including using a separate Solr core per customer, intelligent indexing strategies, and partitioning customers across multiple Solr servers. These changes helped improve performance and allow Nutshell to scale Solr to support thousands of customers.
Scalling Solr in the Cloud - OSCON Data 2011
- 1. Scaling Solr in the Cloud By @ablyler & @LindsaySnider FROM @NutshellCRM www.nutshell.com
- 2. Agenda About the Speakers About Nutshell Nutshell and Solr Solr Resource Usage Scaling Methods Questions and Answers
- 3. About the Speakers BitLeap / Barracuda Networks Developers on the Backup Appliance / Cloud Scaled databases and storage systems Used SugarCRM & Salesforce.com
- 4. About Nutshell Web and Mobile CRM application Heavy use of OpenSource technologies: Gentoo Nginx / PHP / ZendFramework / jQuery MySQL / Solr / Gearman Jenkins / Redmine / Cacti / Nagios
- 5. Nutshell and Solr Heavy use of Solr for searching, table views, and de-duplication Used for searching / display: Accounts, Competitors, Contacts, Leads, Products, Sources, Teams, and Users Used for de-duplication: Accounts, Contacts
- 6. Reads vs Writes 7000 5250 3500 1750 0 Jul 20, 2011 3:00 AM Jul 20, 2011 6:00 AM Jul 20, 2011 9:00 AM Jul 20, 2011 12:00 PM Jul 20, 2011 3:00 PM Jul 20, 2011 6:00 PM Jul 20, 2011 9:00 PM Jul 21, 2011 12:00 AM Writes Reads Time Average Read Query Time: 2.11ms
- 8. Computer Resources CPU Disk I/O Disk RAM Storage
- 9. CPU Disk I/O Disk RAM Storage
- 10. SOLR Resources Heavy CPU Disk I/O Disk RAM Storage Light
- 11. SOLR Resources Heavy CPU Disk I/O Disk RAM Storage Light
- 12. SOLR Resources Heavy CPU Disk I/O Disk RAM Storage Light
- 13. SOLR Resources Heavy CPU Disk I/O Disk RAM Storage Light
- 14. SOLR Resources Heavy CPU Disk I/O Disk RAM Storage Light
- 15. Dark Ages Age of Enlightenment Modern Era Today Auto Provisioning Setup new Solr Jetty app Create MySQL database Populate MySQL / Solr with demo data Send welcome email
- 16. Dark Ages Age of Enlightenment Modern Era Today Separate Jetty per Customer Uses a ton of memory Separate schema / Solr for each customer Ran into upper limit morning before launch
- 17. Dark Ages Age of Enlightenment Modern Era Today Auto provisioning Separate Jetty app for each customer
- 18. Dark Ages Age of Enlightenment Modern Era Today 50 25 0 Auto provisioning Separate Jetty app for each customer
- 19. Dark Ages Age of Enlightenment Modern Era Today Solr Core per Customer Allows for management of Solr on a per customer basis: creating / stopping Contained within a single Jetty app Shared schema between all cores Easily managed via simple HTTP API
- 20. Dark Ages Age of Enlightenment Modern Era Today Fallback to MySQL Landing page of application Allows for graceful handling when Solr is down Abstracted within the application library
- 21. Dark Ages Age of Enlightenment Modern Era Today Sun JVM to IcedTea JVM IcedTea JVM uses less memory than Sun JVM
- 22. Dark Ages Age of Enlightenment Modern Era Today Separate Solr core per customer Fallback to MySQL for table data Migrated from Sun JVM to IcedTea JVM
- 23. Dark Ages Age of Enlightenment Modern Era Today 300 150 0 Separate Solr core per customer Fallback to MySQL for table data Migrated from Sun JVM to IcedTea JVM
- 24. Dark Ages Age of Enlightenment Modern Era Today Shared Schema Across Cores Decreases initialization time for each core Decreases memory usage
- 25. Dark Ages Age of Enlightenment Modern Era Today Solr Index Field Selection More indexed fields = more used memory Only index fields that are searched Store other non-indexed fields for display
- 26. Dark Ages Age of Enlightenment Modern Era Today Splitting of Reader / Writer Index building is CPU / disk intensive Writer = Solr with caching disabled Reader = Solr slave, that doesn’t build indexes
- 27. Dark Ages Age of Enlightenment Modern Era Today Web Solr MySQL Slave (master- master)
- 28. Dark Ages Age of Enlightenment Modern Era Today Web Solr MySQL Slave (master- master) Solr Master Range indexer Range indexer Range indexer Range indexer Document indexers Gearman workers
- 29. Dark Ages Age of Enlightenment Modern Era Today Web Solr MySQL Slave (master- master) Solr Master Range indexer Range indexer Range indexer Range indexer Document indexers Gearman workers
- 30. Dark Ages Age of Enlightenment Modern Era Today Web Solr MySQL Slave (master- master) Solr Master Range indexer Range indexer Range indexer Range indexer Document indexers Gearman workers
- 31. Dark Ages Age of Enlightenment Modern Era Today Web Solr MySQL Slave (master- master) Solr Master Range indexer Range indexer Range indexer Range indexer Range indexers Reindex Manager
- 32. Dark Ages Age of Enlightenment Modern Era Today Web Solr MySQL Slave (master- master) Solr Master Range indexer Range indexer Range indexer Range indexer Range indexers Reindex Manager
- 33. Dark Ages Age of Enlightenment Modern Era Today Web Solr MySQL Slave (master- master) Solr Master Range indexer Range indexer Range indexer Range indexer Range indexers Reindex Manager
- 34. Dark Ages Age of Enlightenment Modern Era Today Shared schema across cores Solr index field selection Splitting of reader / writer roles Intelligent indexing / reindexing
- 35. Dark Ages Age of Enlightenment Modern Era Today 1000 500 0 Shared schema across cores Solr index field selection Splitting of reader / writer roles Intelligent indexing / reindexing
- 36. Dark Ages Age of Enlightenment Modern Era Today Intelligent Core Management Least recently used Solr cores are spun down Solr cores started on login Reindexing is a database flag, and happens on the next login
- 37. Dark Ages Age of Enlightenment Modern Era Today Partitioning Pairs of readers and writers Partitioned based on account id Left pad the account id w/ zeros to length of two Reverse the account id Take last two digits of the account id
- 38. Dark Ages Age of Enlightenment Modern Era Today Intelligent core spin down / up Partitioning of customers to separate Solr servers
- 39. Dark Ages Age of Enlightenment Modern Era Today 8000 4000 0 Intelligent core spin down / up Partitioning of customers to separate Solr servers
- 41. SOLR Server 8000 User
- 42. SOLR SOLR SOLR Server SOLR Server SOLR Server SOLR 8000 Server UserSOLR 8000 Server User 8000 Server 8000User 8000User Server 8000User 8000User User
- 43. SOLR Server 8000 User SOLR Server 8000 User SOLR Server 8000 User SOLR Server 8000 User SOLR Server 8000 User SOLR Server 8000 User SOLR Server 8000 User
- 44. SOLR Server 8000 User SOLR Server 8000 User SOLR Server 8000 User SOLR Server 8000 User SOLR Server 8000 User SOLR Server 8000 User SOLR Server 8000 User
- 45. SOLR Server 8000 User SOLR Server 8000 User 123456 SOLR Server 8000 User SOLR Server 8000 User SOLR Server 8000 User SOLR Server 8000 User SOLR Server 8000 User
- 46. SOLR Server 8000 User SOLR Server 8000 User 123456 SOLR Server 8000 User 1 2 3 4 5 6 SOLR Server 8000 User SOLR Server 8000 User SOLR Server 8000 User SOLR Server 8000 User
- 47. SOLR Server 8000 User SOLR Server 8000 User 123456 SOLR Server 8000 User 1 2 3 4 5 6 SOLR Server 8000 User SOLR Server 8000 User SOLR Server 8000 User SOLR Server 8000 User
- 48. SOLR Server 8000 User SOLR Server 8000 User 123456 SOLR Server 8000 User 1 2 3 4 5 6 SOLR Server 8000 User SOLR 1 2 3 4 5 6 Server 8000 User SOLR Server 8000 User SOLR Server 8000 User
- 49. SOLR Server 8000 User SOLR Server 8000 User 123456 SOLR Server 8000 User 1 2 3 4 5 6 SOLR Server 8000 User SOLR 1 2 3 4 5 6 Server 8000 User 65 SOLR Server 8000 User SOLR Server 8000 User
- 50. SOLR Server 8000 User 01 SOLR 02 Server 8000 User 123456 03 SOLR 04 Server 8000 User 1 2 3 4 5 6 65 SOLR Server 66 8000 User SOLR 1 2 3 4 5 6 96 Server 8000 User 97 65 SOLR 98 Server 8000 User 99 SOLR Server 8000 User
- 51. SOLR Server 8000 User 01 SOLR 02 Server 8000 User 123456 03 SOLR 04 Server 8000 User 1 2 3 4 5 6 65 SOLR Server 66 8000 User SOLR 1 2 3 4 5 6 96 Server 8000 User 97 65 SOLR 98 Server 8000 User 99 SOLR Server 8000 User
- 52. SOLR Server 8000 User 01 SOLR 02 Server 8000 User 123456 03 SOLR 04 Server 8000 User 1 2 3 4 5 6 65 SOLR Server 66 8000 User SOLR 1 2 3 4 5 6 96 Server 8000 User 97 65 SOLR 98 Server 8000 User 99 SOLR Server 8000 User
- 53. SOLR Server 8000 User 01 SOLR 02 Server 8000 User 03 SOLR 04 Server 8000 User 65 SOLR Server 66 8000 User SOLR 96 Server 8000 User 97 SOLR 98 Server 8000 User 99 SOLR Server 8000 User
- 54. SOLR Server 8000 User 01 SOLR 02 Server 8000 User 03 SOLR 04 Server 8000 User 65 SOLR Server 66 8000 User SOLR 96 Server 8000 User 97 SOLR 98 Server 8000 User 99 SOLR Server 8000 User
- 55. SOLR Server 8000 User 01 SOLR 02 Server 8000 User 03 SOLR 04 Server 8000 User 65 SOLR Server 66 8000 User SOLR Server 8000 User SOLR 96 Server 8000 User 97 SOLR 98 Server 8000 User 99 SOLR Server 8000 User
- 56. SOLR Server 8000 User 01 SOLR 02 Server 8000 User 03 SOLR 04 Server 8000 User 65 SOLR Server 66 8000 User SOLR Server 8000 User SOLR 96 Server 8000 User SOLR 97 Server SOLR 8000 User 98 Server 8000 User 99 SOLR Server 8000 User
- 57. SOLR Server 8000 User 01 SOLR 02 Server 8000 User 03 SOLR 04 Server 8000 User 65 SOLR Server 66 8000 User SOLR Server 8000 User SOLR 96 Server 8000 User SOLR 97 Server SOLR 8000 User 98 Server 8000 User 99 SOLR Server 8000 User
- 58. SOLR Server 8000 User 01 SOLR 02 Server 8000 User 03 SOLR 04 Server 8000 User 65 SOLR Server 66 8000 User SOLR Server 8000 User SOLR 96 Server 8000 User SOLR 97 Server SOLR 8000 User 98 Server 8000 User 99 SOLR Server 8000 User
- 59. SOLR Server 8000 User 01 SOLR 02 Server 8000 User 03 SOLR 04 Server 8000 User 65 SOLR Server 66 8000 User SOLR Server 8000 User SOLR 96 Server 8000 User SOLR 97 Server SOLR 8000 User 98 Server 8000 User 99 SOLR Server 8000 User
- 60. SOLR Server 8000 User 01 SOLR 02 Server 8000 User 03 SOLR 04 Server 8000 User 65 SOLR Server 66 8000 User SOLR Server 8000 User 96 SOLR 97 Server SOLR 8000 User 98 Server 8000 User 99 SOLR Server 8000 User
- 61. Scaling Solr in the Cloud auto provisioning 50 Users separate Jetty app for each customer separate Solr core per customer fall back to MySQL for table data 300 Users migrated from Sun JVM to IcedTea JVM shared schema across cores solr index field selection 600 Users splitting reader / writer roles intelligent core spin down / up 7000 Users parititioning of customer to separate Solr servers By @ablyler & @LindsaySnider FROM @NutshellCRM