Ubiquitous Solr - A Database's Not-So-Evil Twin: Presented by Ayon Sinha, WalmartLabs
1 like1,133 views
This document discusses how Walmart uses Apache Solr as a "not-so-evil twin" to complement their source of truth database and help scale their data infrastructure. It describes how Walmart abstracts the complexity of managing databases, caches, search queries, and messaging to provide scalable querying across database shards. The use of Solr has allowed Walmart to offload queries, recurring reads, analytics
Ubiquitous Solr - A Database's Not-So-Evil Twin: Presented by Ayon Sinha, WalmartLabs
- 1. Ubiquitous Solr - A Database’s not-so-evil Twin Ayon Sinha Data Foundation @WalmartLabs O C T O B E R 1 3 - 1 6 , 2 0 1 6 • A U S T I N , T X
- 2. 2 Text Search wow Search Suggestions Search Engine… Lucene… Solr • Internet and Intranet Search • Relevance • Search Suggestions • Faceting • Recommendations • Time series • Log search • Geo-spatial search • Analytics • Graph search • Document Store Recommendations Relevance Facets
- 3. 3 Overview • How to scale any data infrastructure with Apache Solr • Build a high performance and highly available data platform for internal and external users alike • Walmart’s commitment to open source
- 4. 4 About me • Team lead at the Data Foundation team for the largest retailer and the largest private employer in the world • Prior to Walmart, worked at startups building recommendation and analytics systems • And prior to that, was building search applications, recommendations systems and Hadoop based analytics systems for the largest online auction company, ebay, for 6 years • Have been a manuscript reviewer for Manning publications for 4 years and have helped shape the contents of “Hadoop in Practice” and “Big Data”
- 5. 5 About Walmart • 11,000+ Stores in 27 countries • 11 eCommerce sites • 250M customers weekly in stores and online • Millions of database transactions per day • Sales, Holidays and massive volume shifts
- 6. 6 It starts-up so simple An idea implemented on the LAMP stack
- 7. 7 Turns out to be a great idea! Users seem to like the new product
- 8. 8 Users REALLY like this.. Higher volume, increased use cases. Quick fix scaling alternatives add some headroom … and complexity
- 9. 9 We need more Business Intelligence Business is looking good but source-of-truth data store, not so much …
- 10. 10 Scale up (in a hurry) with hardware Least risk. Diminishing returns. What next?
- 11. 11 Design to scale out • Offload queries to Search Engines • Offload recurring reads to Cache • Offload analytics to OLAP datastores • Shard the database … and of course do something to hide the complexity. It is worth it.
- 12. 12 The Inspiration Integration tools with a Lucene based search engines are abundant
- 13. 13 The “not-so-evil” Twin to protect your Source of Truth DB • What if a copy of your source-of-truth data is available … Just about anywhere you want it? • How could you use a search engine to protect and augment your database? – Redirect queries • Helps scale by reducing demand for – database indexing – database connections – scarce database resources like memory, storage • Not-so-evil Twin – Adding multiple near real-time search adds complexity … and it comes at a cost; but done right, the benefits far outweigh the costs
- 14. 14 Our Approach • Abstract the complexity of managing – source-of-truth database – cache coherence – Search queries – message bus • Abstract Connection pool management • Provide a scalable way to query across shards with full control of Solr schema • And to analyze big data without affecting real-time systems and isolating individual data domains
- 15. 15 From a situation like..
- 16. 16 DB, Solr and Hadoop
- 17. 17 Sharded DB with Solr
- 18. 18 The Eco-system Separation of concerns
- 19. 19 The Result Scatter-gather vs Powered by Apache Solr
- 20. 20 Lessons learned A Search engine like Apache Solr is… • not limited to search-based business applications. • a first class citizen in your persistence technology stack; it complements the SoT database. • easy to adopt and has all of us as community for support.
- 21. 21 The Future • Symbiotic existence of Solr/Lucene with RDBMS, NoSQL and Big Data systems • Walmart is committed to be part of the community building it
- 22. 22 Questions? Reach us at: • You can reach me, Ayon Sinha, at: – asinha@walmartlabs.com – https://www.linkedin.com/in/ayonsinha • Jason Sardina, our Lead Persistence Architect – jsardina@walmartlabs.com • @WalmartLabs is always hiring the best