Open Source Search Evolution
2 likes1,188 views
This document summarizes the evolution of open source search tools from the early 1970s to present day. It discusses the transition from early tools like WAIS and Harvest in the 1990s to modern distributed search platforms like Elasticsearch. Key areas of advancement are highlighted, such as support for more languages through improved stemming and lemmatization, more sophisticated relevance algorithms, distributed architectures for scaling data and queries, faster indexing and real-time search, reduced memory footprints, and expanding capabilities beyond basic text search to include geospatial, classification, recommendation, key-value storage, analytics and more.
![[Open Source]
Search Evolution
Otis Gospodnetić @otisg](https://image.slidesharecdn.com/opensourcesearchevolution-140404103734-phpapp01/85/Open-Source-Search-Evolution-1-320.jpg)
![Dominance
[Open Source]
Search Evolution](https://image.slidesharecdn.com/opensourcesearchevolution-140404103734-phpapp01/85/Open-Source-Search-Evolution-9-320.jpg)
![Gobble Gobble
If software is eating the world,
then [open source] search is gobbling it.
And has been for years.](https://image.slidesharecdn.com/opensourcesearchevolution-140404103734-phpapp01/85/Open-Source-Search-Evolution-28-320.jpg)
Open Source Search Evolution
- 1. [Open Source] Search Evolution Otis Gospodnetić @otisg
- 2. Today
- 7. Then & Now 1990s 2014 WebGlimpse Swish Harvest Ht://Dig freeWAIS elasticsearch.
- 8. Still New? elasticsearch. …………………... 2000 …………………... 2004 …………………... 2010
- 10. Big Cake Big Data Beyond Text Memory Footprint Distributed Model Language Support Indexing Speed, NRT Relevance Algorithms
- 14. Language Support Lucene 2004: ~ 20 languages Lucene 2014: ~ 40 languages most are stemmers
- 15. Relevance Models: VSM TF IDF For term i in document j wi,j = tfi,j x log(N/dfi ) tfi,j = number of occurrences of i in j dfi = number of document containing i N = total number of documents
- 16. Relevance Models: Pluggable Lucene until 2011: 1 relevance model Lucene 2014: 6 relevance models got more?
- 17. Distributed Architecture 1 Master - N Slaves good for scaling queries not good for scaling data Sharded index with replication good for scaling queries good for scaling data
- 18. Indexing Speed & NRT Search
- 19. Memory Footprint
- 20. Beyond Text Geospatial Search Classifier Recommendation Engine Key Value Store NoSQL DB Analytical DB
- 22. Classifier
- 24. Key Value Store id123 ⇒ manu:Apple desc:foo bar price:$111 id234 ⇒ manu:Sony desc:baz bam price:$222
- 25. NoSQL DB Distributed Replicated Horizontally Scalable Fast Retrieval Searchable?
- 26. Slicing & Dicing
- 28. Gobble Gobble If software is eating the world, then [open source] search is gobbling it. And has been for years.