Make Plone Search Act Like Google Using Solr
6 likes2,048 views
The document outlines a presentation by Clayton Parker at the Plone Conference 2011, focusing on integrating Solr with Plone to enhance search functionality similar to Google. It covers various topics including Solr configuration, schema setup, debugging tips, and character and token filters. The document also provides examples and links related to Solr resources and tools.
![Pattern Replace PLONE CONFERENCE 2011
<charFilter class="solr.PatternReplaceCharFilterFactory"
pattern="[^a-zA-z0-9_-]" replacement="" replace="all"/>
'That WAS a narrow escape!' said Alice, a good deal frightened
That WAS a narrow escape said Alice a good deal frightened](https://image.slidesharecdn.com/clayton-parker-solrlikegoogle-111106120654-phpapp02/85/Make-Plone-Search-Act-Like-Google-Using-Solr-33-320.jpg)
![Pattern Replace PLONE CONFERENCE 2011
<filter class="solr.PatternReplaceFilterFactory"
pattern="[^a-zA-z0-9_-]" replacement="" replace="all"/>
'That That
WAS WAS
a a
narrow narrow
escape!' escape
said said
Alice Alice](https://image.slidesharecdn.com/clayton-parker-solrlikegoogle-111106120654-phpapp02/85/Make-Plone-Search-Act-Like-Google-Using-Solr-45-320.jpg)
Make Plone Search Act Like Google Using Solr
- 1. Make Plone Search Act Like Google Using Solr Clayton Parker | Senior Web Developer PLONE CONFERENCE 2011
- 2. Who Am I
- 3. What will we learn? PLONE CONFERENCE 2011
- 4. What will we learn? PLONE CONFERENCE 2011 • Intro to Solr
- 5. What will we learn? PLONE CONFERENCE 2011 • Intro to Solr • Brief overview of Plone integration points
- 6. What will we learn? PLONE CONFERENCE 2011 • Intro to Solr • Brief overview of Plone integration points • Solr configuration
- 7. What will we learn? PLONE CONFERENCE 2011 • Intro to Solr • Brief overview of Plone integration points • Solr configuration • Solr schema setup
- 8. What will we learn? PLONE CONFERENCE 2011 • Intro to Solr • Brief overview of Plone integration points • Solr configuration • Solr schema setup • Debugging tips and tricks
- 9. PLONE CONFERENCE 2011 What is Solr ?
- 10. Version Madness PLONE CONFERENCE 2011 1.x (up to 1.4) 1.5 (number abandoned) 3.x (merge of Lucene and Solr)
- 11. Books PLONE CONFERENCE 2011
- 16. Query Handlers PLONE CONFERENCE 2011 • Standard • Disjunction Max (DisMax) • Extended DisMax (experimental)
- 17. DisMax PLONE CONFERENCE 2011 • Multiple index searches • Boosting • Friendlier to end users
- 18. DisMax PLONE CONFERENCE 2011 Index Name qf=SearchableText^1.0 substring^0.2 Weight
- 19. MinShouldMatch PLONE CONFERENCE 2011 mm=100% All terms required mm=50% Half of the terms required mm=-2 All but two terms required
- 20. MinShouldMatch PLONE CONFERENCE 2011 mm=2<-25% 9<-3 2 or less 3-9 terms all more than 9 terms are but 25% terms all but required required three are required
- 21. Spelling Component PLONE CONFERENCE 2011 <searchComponent name="spellcheck" class="solr.SpellCheckComponent"> <lst name="spellchecker"> <str name="name">default</str> <str name="classname">solr.IndexBasedSpellChecker</str> <str name="buildOnCommit">true</str> <str name="spellcheckIndexDir">path/to/spellcheck</str> <!-- The field that will contain the dynamic spelling data --> <str name="field">spell</str> <str name="accuracy">0.5</str> </lst> <!-- Control indexing and query of spelling data --> <str name="queryAnalyzerFieldType">spell-text</str> </searchComponent>
- 22. Spelling Schema PLONE CONFERENCE 2011 <fieldType name="spell-text" class="solr.TextField"> <analyzer> <tokenizer class="solr.ICUTokenizerFactory"/> <filter class="solr.ICUFoldingFilterFactory"/> <filter class="solr.StopFilterFactory" ignoreCase="true" words="stopwords.txt"/> </analyzer> </fieldType>
- 23. Solr Schema
- 24. Index vs Query PLONE CONFERENCE 2011 http://www.cominvent.com/2011/04/04/solr-architecture-diagram/
- 26. PLONE CONFERENCE 2011 Character Filters
- 27. PLONE CONFERENCE 2011 Character Filters Tokenizer
- 28. PLONE CONFERENCE 2011 Character Filters Tokenizer Filters
- 29. PLONE CONFERENCE 2011 Character Filters Tokenizer Filters
- 30. Complete Field PLONE CONFERENCE 2011 <fieldType name="text" class="solr.TextField" positionIncrementGap="100"> <analyzer type="index"> <charFilter class="solr.MappingCharFilterFactory" mapping="mapping.txt"/> <tokenizer class="solr.ICUTokenizerFactory"/> <filter class="solr.ICUFoldingFilterFactory"/> <filter class="solr.SynonymFilterFactory" synonyms="synonyms.txt" ignoreCase="true" expand="true"/> <filter class="solr.StopFilterFactory" ignoreCase="true" words="stopwords.txt"/> <filter class="solr.EnglishPorterFilterFactory" protected="protwords.txt"/> <filter class="solr.WordDelimiterFilterFactory" splitOnCaseChange="1" splitOnNumerics="1" stemEnglishPossessive="1" generateWordParts="1" generateNumberParts="1" catenateWords="0" catenateNumbers="0" catenateAll="0" preserveOriginal="1"/> </analyzer> <analyzer type="query"> <charFilter class="solr.MappingCharFilterFactory" mapping="mapping.txt"/> <tokenizer class="solr.ICUTokenizerFactory"/> <filter class="solr.ICUFoldingFilterFactory"/> <filter class="solr.StopFilterFactory" ignoreCase="true" words="stopwords.txt"/> <filter class="solr.EnglishPorterFilterFactory" protected="protwords.txt"/> <filter class="solr.WordDelimiterFilterFactory" splitOnCaseChange="1" splitOnNumerics="1" stemEnglishPossessive="1" generateWordParts="1" generateNumberParts="1" catenateWords="0" catenateNumbers="0" catenateAll="0" preserveOriginal="1"/> <filter class="solr.PositionFilterFactory"/> </analyzer> </fieldType>
- 31. Copy Field PLONE CONFERENCE 2011 <copyField source="SearchableText" dest="spell"/> <copyField source="SearchableText" dest="substring"/>
- 32. PLONE CONFERENCE 2011 Character Filters • Process text before tokenizing • Remove irrelevant characters
- 33. Pattern Replace PLONE CONFERENCE 2011 <charFilter class="solr.PatternReplaceCharFilterFactory" pattern="[^a-zA-z0-9_-]" replacement="" replace="all"/> 'That WAS a narrow escape!' said Alice, a good deal frightened That WAS a narrow escape said Alice a good deal frightened
- 34. Mapping PLONE CONFERENCE 2011 <charFilter class="solr.MappingCharFilterFactory" mapping="mapping.txt"/> # œ => oe "u0153" => "oe" # ß => ss "u00DF" => "ss"
- 35. HTML Strip PLONE CONFERENCE 2011 <charFilter class="solr.HTMLStripCharFilterFactory"/>
- 36. PLONE CONFERENCE 2011 Tokenizers • Split raw text into tokens / terms • Typically the first step
- 37. Whitespace Tokenizer PLONE CONFERENCE 2011 <tokenizer class="solr.WhitespaceTokenizerFactory"/> 'That WAS a narrow escape!' said Alice 'That WAS a narrow escape!' said Alice
- 38. ICU Tokenizer PLONE CONFERENCE 2011 <tokenizer class="solr.ICUTokenizerFactory"/> 'That WAS a narrow escape!' said Alice That WAS a narrow escape said Alice
- 39. Pattern Tokenizer PLONE CONFERENCE 2011 <tokenizer class="solr.PatternTokenizerFactory" pattern=";s*" /> one; two; three one two three
- 40. Path Hierarchy PLONE CONFERENCE 2011 <tokenizer class="solr.PathHierarchyTokenizerFactory"/> /usr/local/etc/nginx /usr /usr/local /usr/local/etc /usr/local/etc/nginx
- 41. PLONE CONFERENCE 2011 Token Filters • Process after tokenizing • Normalization of terms
- 42. Lower Case PLONE CONFERENCE 2011 <filter class="solr.LowerCaseFilterFactory"/> Foo bAr BAZ foo bar baz
- 43. ASCII Folding PLONE CONFERENCE 2011 <filter class="solr.ASCIIFoldingFilterFactory"/> idée bête grüßen idee bete grussen
- 44. ICU Folding PLONE CONFERENCE 2011 <filter class="solr.ICUFoldingFilterFactory"/> Idée BÊTE GrüßeN idee bete grussen
- 45. Pattern Replace PLONE CONFERENCE 2011 <filter class="solr.PatternReplaceFilterFactory" pattern="[^a-zA-z0-9_-]" replacement="" replace="all"/> 'That That WAS WAS a a narrow narrow escape!' escape said said Alice Alice
- 46. Word Delimiter PLONE CONFERENCE 2011 <filter class="solr.WordDelimiterFilterFactory" splitOnCaseChange="1" splitOnNumerics="1" stemEnglishPossessive="1" generateWordParts="1" generateNumberParts="1" catenateWords="0" catenateNumbers="0" catenateAll="0" preserveOriginal="1"/> StudlyCaps1234-5678 Caps StudlyCaps1234-5678 Studly 1234 5678
- 47. Edge N Gram PLONE CONFERENCE 2011 <filter class="solr.EdgeNGramFilterFactory" minGramSize="4" maxGramSize="100" side="front"/> Conqueror Conqueror Conquero Conquer Conque Conqu Conq
- 48. Stop Words PLONE CONFERENCE 2011 <filter class="solr.StopFilterFactory" ignoreCase="true" words="stopwords.txt"/> That narrow WAS escape a said narrow Alice escape good said deal Alice frightened a good deal frightened
- 49. Synonyms PLONE CONFERENCE 2011 <filter class="solr.SynonymFilterFactory" synonyms="synonyms.txt" ignoreCase="true" expand="true"/> # synonyms.txt foozball foosball foosball # add multiple terms baby-foot foozball, foosball, baby-foot tele television # merge into one t.v. television tv, t.v., tele => television tv television
- 50. Language Stemming PLONE CONFERENCE 2011 <filter class="solr.EnglishPorterFilterFactory" protected="protwords.txt"/> dry dri drying dri dried dri
- 51. Language Stemming PLONE CONFERENCE 2011 <filter class="solr.ElisionFilterFactory" articles="stopwordarticles.txt"/> qu'il il ne ne comprend comprend pas pas l'anglais anglais <filter class="solr.EnglishPorterFilterFactory" language="French"/> considere consider consideres consider considerent consider
- 52. Solr Debugging
- 53. Schema Browser PLONE CONFERENCE 2011
- 54. Analysis PLONE CONFERENCE 2011
- 55. Analysis PLONE CONFERENCE 2011
- 56. Analysis PLONE CONFERENCE 2011
- 57. Analysis PLONE CONFERENCE 2011
- 58. Analysis PLONE CONFERENCE 2011
- 59. Search Interface PLONE CONFERENCE 2011
- 60. Crafting a URL PLONE CONFERENCE 2011 http://localhost:8983/solr/select? qf=SearchableText^1.0&rows=10&fl=*,score&debugQuery=on& explainOther=True&indent=true&defType=dismax&q=test q=test qf=SearchableText^1.0 defType=dismax debugQuery=on explainOther=on indent=on
- 61. Verbose XML* PLONE CONFERENCE 2011 <lst name="responseHeader"> <int name="status">0</int> <int name="QTime">2</int> <lst name="params"> <str name="explainOther">True</str> <str name="fl">*,score</str> <str name="debugQuery">on</str> <str name="indent">true</str> <str name="q">test</str> <str name="qf">SearchableText^1.0</str> <str name="rows">10</str> <str name="defType">dismax</str> </lst> </lst> * like there is any other kind
- 62. Verbose XML* PLONE CONFERENCE 2011 <result name="response" numFound="2" start="0" maxScore="0.70710677"> <doc> <float name="score">0.70710677</float> <int name="docid">-643919099</int> </doc> <doc> <float name="score">0.3788861</float> <int name="docid">-643919097</int> </doc> </result> * like there is any other kind
- 63. Verbose XML* <lst name="debug"> PLONE CONFERENCE 2011 <str name="rawquerystring">test</str> <str name="querystring">test</str> <str name="parsedquery">+DisjunctionMaxQuery((SearchableText:test)) ()</str> <str name="parsedquery_toString">+(SearchableText:test) ()</str> <lst name="explain"> <str name="-643919099"> 0.70710677 = (MATCH) sum of: 0.70710677 = (MATCH) fieldWeight(SearchableText:test in 4), product of: 1.4142135 = tf(termFreq(SearchableText:test)=2) 1.0 = idf(docFreq=5, maxDocs=6) 0.5 = fieldNorm(field=SearchableText, doc=4) </str> <str name="-643919097"> 0.3788861 = (MATCH) sum of: 0.3788861 = (MATCH) fieldWeight(SearchableText:test in 0), product of: 1.7320508 = tf(termFreq(SearchableText:test)=3) 1.0 = idf(docFreq=5, maxDocs=6) 0.21875 = fieldNorm(field=SearchableText, doc=0) </str> </lst> * like there is any other kind
- 64. Links PLONE CONFERENCE 2011 • Solr (http://lucene.apache.org/solr) • Solr Wiki (http://wiki.apache.org/solr) • Books (http://www.packtpub.com/books/all?keys=solr) • SolrIndex (http://pypi.python.org/pypi/alm.solrindex/) • collective.solr (http://pypi.python.org/pypi/collective.solr)
- 65. Flickr Credits PLONE CONFERENCE 2011 • http://www.flickr.com/photos/naturegeak/5642083189/ (who) • http://www.flickr.com/photos/eklektikos/2541408630/ (schema) • http://www.flickr.com/photos/sidelong/13954593/ (char filter) • http://www.flickr.com/photos/benimoto/2214240119/ (tokenizers) • http://www.flickr.com/photos/chaunceydavis/3264077445/ (filters) • http://www.flickr.com/photos/comedynose/3271760209/ (configuration) • http://www.flickr.com/photos/nicksart/4821509371/ (debugging) Thanks to
- 66. Questions? Check out .co m/d emos s ixfeetup