Implementing search with solr at 7digital
3 likes1,609 views
James Atherton, lead of the content discovery team at 7digital, discusses the implementation of search using Solr, detailing the company's past architecture and lessons learned. The transition from SQL to Solr has resulted in significant improvements in search efficiency as data scales increased. The document outlines ongoing challenges, future aspirations for architecture, and the tools being used for monitoring and optimization.
Implementing search with solr at 7digital
- 1. Implementing Search with Solr at 7digital James Atherton Content Discovery Team Lead
- 2. Implementing Search with Solr James Atherton Content Discovery Lead @mr_road
- 3. Who is 7digital? Online digital content provider Covering over 47 territories Online music store: www.7digital.com API: api.7digital.com We power a number of music services: Samsung Blackberry Turntable.fm Pure
- 4. Where we came from... SQL Searches SELECT * FROM <table> WHERE name LIKE '<search_term>%'; This was SLOW and BAD!!
- 5. Wrapped Solr in an API
- 7. Domain Objects Artist Documents Release Documents (e.g. album or single) Track Documents
- 8. First Attempt - 2011 • Artists and Releases • Solr 1.4 • 17 stores • ~40GB • Dropped DIH as it had issues
- 10. 2012 • Added Tracks Core • Solr 3.5 • 47 stores • ~400GB • More than 430 M docs • Didn't revisit DIH
- 11. Current Architecture HTTP API Search API Artist/Release Solrs Track Solrs Track Solrs Track Solrs Track Solrs Artist/Release Solrs
- 12. Things Learnt We should have split by <X>; for us Shops.
- 13. Beware Inflection Points Data size: 400GB != 40GB * 10 Throughput: 600 rpm IS NOT 4 * 150 rpm
- 14. What we want in our servers? RAM ? Fast Disks? CPUs? Virtual? Bare Metal?
- 17. Testing Test ingestion/data import, then test again Your data is not as clean as you think Load test early and often We need to be better at this still
- 18. Logs Logging is worth its weight in gold But don't get weighed down
- 19. Monitoring We use statsd/graphite and NewRelic:
- 20. Visualise Indexing Which territory's data has been indexed?
- 21. Instant Search
- 22. Magic Deploys We recently adopted CFEngine, it is awesome!!
- 23. The Future HTTP API Search API Artist Solrs Track Solrs Track Solrs Track Solrs Release Solrs Track Solrs Solr Cloud, in the Cloud??
- 24. Questions ?