![Searching : QueryParser
• title:"apple webobjects" - Phrase Query
• title:"apple webobjects"~5 - slop of 5
• webobj* - Prefix Query
• webobjicts~ - Fuzzy Query
• lastmodified:[1/1/10 TO 1/1/11] - Range Query](https://image.slidesharecdn.com/fts-110908070657-phpapp02/85/Full-Text-Search-with-Lucene-36-320.jpg)
![Performing a Search
Query q = ... // assume exists
IndexSearcher searcher = new IndexSearcher(index, true);
TopScoreDocCollector collector = TopScoreDocCollector.create(10, true);
searcher.search(query, collector);
ScoreDoc[] hits = collector.topDocs().scoreDocs;](https://image.slidesharecdn.com/fts-110908070657-phpapp02/85/Full-Text-Search-with-Lucene-37-320.jpg)
Full Text Search with Lucene
- 1. Full Text Search David LeBer Align Software Inc.
- 2. What is full text search?
- 4. How? • Wild card database queries • Database implementations • Third party search engines • Text indexing libraries
- 5. Wild Card Queries SELECT FROM 'SOME_TABLE' WHERE 'SOME_COLUMN' LIKE '%Some String%'
- 6. Wild Card Queries • Easy
- 7. Wild Card Queries • Slow • Hard to optimize • Difficult to rank
- 8. Database Implementations • MySQL FULLTEXT index and MATCH queries • PostgreSQL tsvector & tsquery
- 9. Database Implementations • Fairly Easy
- 10. Database Implementations • Database specific SQL • May include additional limitations (i.e: MySQL - MyISAM tables only) • Functionality define by the DB engine
- 11. Third Party Search Engines • Google indexing / searching of your content
- 12. Third Party Search Engines • Easy • Matches user expectations
- 13. Third Party Search Engines • Content must be available for indexing • Loss of control • Enhances the Google hegemony
- 14. Text Indexing Library • Lucene
- 15. Text Indexing Library • Complete control • Database independent • Flexible search behaviour • Ranked results
- 16. Text Indexing Library • Adds complexity • Additional query language • Parallel index
- 17. Lucene Overview • Open Source - part of the Apache Project • Very flexible • Wickedly fast • Index based
- 18. Lucene : Installing • Add the Lucene jars to your classpath • Use ERIndexing
- 19. Lucene : Tasks • Indexing • Searching
- 20. Indexing
- 22. Indexing : Steps • Conversion (to plain text) • Analysis (clean and convert the text to tokens) • Index (save the tokens to the index)
- 23. Indexing : Parts • Index - either file or memory based • Document - represents a unique object added to the index • Field - identifies a chunk of data in the document
- 24. Indexing : Classes • IndexWriter • Directory • Analyzer • Document • Field
- 25. Creating an Index URL indexDirectoryURL = ... // assume exists File indexFile = new File(indexDirectoryURL.getPath()); FSDirectory indexDirectory = FSDirectory.open(indexFile); StandardAnalyzer analyzer = new StandardAnalyzer(Version.LUCENE_CURRENT); IndexWriter indexWriter = new IndexWriter(index, analyzer, true, IndexWriter.MaxFieldLength.UNLIMITED);
- 26. Indexing : Field Parameters • Stored or not • Analyzed or not, with and without norms • Include position, offset, and term frequency
- 27. Indexing : Analyzers • SimpleAnalyzer • StopAnalyzer • StandardAnalyzer • ...
- 28. Adding a Document String value = ... // assume exists Document doc = new Document(); Field docField = new Field("title", value, Field.Store.YES, Field.Index.ANALYZED); doc.add(docField); ... indexWriter.addDocument(doc);
- 29. Indexing : Fun with indexes • Multiple Access
- 30. Searching
- 32. Searching : Steps • Clean the user input • Create a Query • Query the Index • Return the results
- 33. Searching : Search Classes • IndexReader • IndexSearcher • Query • QueryParser • TopDocs/ScoreDocs • Document
- 34. Searching : QueryTypes • TermQuery • RangeQuery • PrefixQuery • BooleanQuery • PhraseQuery • WildCardQuery • FuzzyQuery
- 35. Searching : QueryParser • 'webobjects' - contains an exact match - TermQuery • 'webobjects apple', 'webobjects OR apple' - an OR Query • +webobjects +apple / webobjects AND apple - an AND Query • title:webobjects - Contains the term in title field • title:webobjects -subject:iTunes / title:webobjects AND NOT subject:iTunes • (webobjects OR apple) AND iTunes
- 36. Searching : QueryParser • title:"apple webobjects" - Phrase Query • title:"apple webobjects"~5 - slop of 5 • webobj* - Prefix Query • webobjicts~ - Fuzzy Query • lastmodified:[1/1/10 TO 1/1/11] - Range Query
- 37. Performing a Search Query q = ... // assume exists IndexSearcher searcher = new IndexSearcher(index, true); TopScoreDocCollector collector = TopScoreDocCollector.create(10, true); searcher.search(query, collector); ScoreDoc[] hits = collector.topDocs().scoreDocs;
- 38. Using a QueryParser QueryParser queryParser = new QueryParser(Version.LUCENE_2.9, "content", analyzer); Query query = queryParser.parse(queryString);
- 39. Demo
- 40. Scoring
- 41. “The more times a query term appears in a document relative to the number of times the term appears in all the documents in the collection, the more relevant that document is to the query”
- 42. Boost • While Indexing • Document • Field • While Searching • Query
- 43. Luke
- 44. Demo
- 45. ERIndexing
- 46. ERIndexing : Strengths • Hides some of the complexity of integrating Lucene with WO • Offers lots of utility and helper methods • Speaks WebObjects collection classes • Simplifies index creation
- 47. ERIndexing : Weaknesses • Hides some of the complexity of integrating Lucene with WO • Not fully baked • Auto indexing may be dangerous
- 48. Demo
- 49. Beyond Lucene • Solr • Compass • ElasticSearch
- 50. Q&A Lucene: http://lucene.apache.org Luke: http://code.google.com/p/luke/ Solr: http://lucene.apache.org/solr/ Compass: http://www.compass-project.org/overview.html ElasticSearch: http://www.elasticsearch.com/