Custom analyzer using lucene
- 1. Custom Analyzer in Lucene Lucene/Solr Meetup Ganesh.M http://www.linkedin.com/in/gmurugappan
- 2. âą Introduction to Analyzer âą Why we require Custom Analyzer âą Use case / Scenario âą Writing custom analyzer âą Know your analyzer
- 3. âą Analyzer : Analyzes the given text and returns tokens using Tokenizer and TokenFilter âą Tokenizer : Understands the language and breaks the text in to tokens. â WhitespaceTokenizer divides text at whitespace â LetterTokenizer divides text at non-letter â CJKTokenizer â Chinese, Japanese, Korean language tokenizer âą TokenFiler: adds / stem / deletes token â StopFilter â removes stop words â PorterStemFilter â Transforms the token
- 4. âą Lets have the text âThe quick brown fox jumps over lazy dogâ Using Standard Analyzer, it will generate following tokens Quick Brown Fox Jumps Over Lazy dog
- 5. Know Your analyzer âą It is important to choose best analyzer for your fields. âą If you choose it wrong then it may not give expected search result. âą If you ever think you are not expecting the correct result then check your Analyzer and Query parser.
- 6. Lucene 3.x: Below code will print the tokens generated from given analyzer Analyzer analyzer = new SimpleAnalyzer(); TokenStream ts = analyzer.tokenStream(âField", new StringReader(âHello world-2013 ")); ts.reset(); while (ts.incrementToken()) { System.out.println("token: " + ts.getAttribute(TermAttribute.class).term()); } ts.close();
- 7. The purpose of Custom Analyzer âą Existing analyzers not always solves our purpose, some times we need to analyze in a different way âą Custom Analyzer could use existing inbuilt filters. âą It could also be used for parsing queries
- 8. Use case âą Synonym Injection / Abbreviation Expansion â Add synonyms at the time of indexing. â In case of parsing resume, add related content for a keyword. If you find text âlucene/solrâ then you could add information retrieval, search engine. â If you are searching medical documents, chat messages etc you need to expand the abbreviation / codes at the time of indexing
- 9. âą Stripping XML / HTML tags and index only the content <Address> <Street>123, MG Road<Street> <City>Bangalore<Bangalore> <State>Karnataka<State> </Address>
- 10. âą Break Email ID / URL in to multiple tokens â Sachin Tendulkar <sachin.tendulkar123@gmail.com> â Should be analyzed as âą sachin âą tendulkar âą sachin âą tendulkar123 âą gmail âą com
- 11. HTMLAnalyzer in Lucene 4.5 public class HTMLAnalyzer extends Analyzer { @Override protected TokenStreamComponents createComponents(String arg0, Reader reader) { HTMLStripCharFilter htmlFilter = new HTMLStripCharFilter(reader); WhitespaceTokenizer tokenizer = new WhitespaceTokenizer(Version.LUCENE_45, htmlFilter); TokenStream result = new LowerCaseFilter(Version.LUCENE_45, tokenizer); return new TokenStreamComponents (tokenizer, result); } }
- 12. HTMLAnalyzer in Solr <fieldType name="text_html" class="solr.TextField" positionIncrementGap="100"> <analyzer> <charFilter class="solr.HTMLStripCharFilterFactory" escapedTags="a, title" /> <tokenizer class="solr.WhitespaceTokenizerFactory"/> </analyzer> </fieldType>
- 13. SynonymAnalyzer âą SynonymAnalyzer will inject the synonym as part of the indexed content using Lucene 3.3 âą Check out the code.. https://github.com/geekganesh/SynonymAnal yzer
- 14. PerFieldAnalyzerWrapper âą IndexWriter / IndexWriterConfig will take only one Analyzer and it will use that for all its fields. âą We may have multiple fields and each field should be indexed using specific analyzer then we need to use PerFieldAnalyzerWrapper âą PerFieldAnalyzerWrapper is used to have different analyzer per field. It will be passed to IndexWriter