SlideShare a Scribd company logo

Semantic & Multilingual Strategies in Lucene/Solr

Trey Grainger, profile picture
Trey Grainger

The document presents strategies for semantic and multilingual text analysis in Lucene/Solr, discussing various analyzers and filters for processing text in multiple languages. It outlines methods for handling different languages per document, as well as automatic language identification techniques. In addition, it covers the advantages of stemming versus lemmatization, and the potential for one field for all languages through stacked token streams.

Technology
1 of 47
Downloaded 260 times
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
Most read
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
Semantic & Multilingual Strategies in Lucene/Solr Trey Grainger Director of Engineering, Search & Analytics@CareerBuilder
Outline •Introduction •Text Analysis Refresher •Language-specific text Analysis •Multilingual Search Strategies •Automatic Language Identification •Semantic Search Strategies (understanding “meaning”) •Conclusion
About Me Trey Grainger Director of Engineering, Search & Analytics Joined CareerBuilderin 2007 as Software Engineer MBA, Management of Technology –GA Tech BA, Computer Science, Business, & Philosophy –Furman University Mining Massive Datasets (in progress) -Stanford University Fun outside of CB: •Co-author of Solr in Action, plus several research papers •Frequent conference speaker •Founder of Celiaccess.com, the gluten-free search engine •Lucene/Solrcontributor
At CareerBuilder, SolrPowers...
Text Analysis Refresher
Text Analysis Refresher A text field in Lucene/Solrhas an Analyzer containing: ①Zero or more CharFilters Takes incoming text and “cleans it up” before it is tokenized ②One Tokenizer Splits incoming text into a Token Stream containing Zero or more Tokens ③Zero or more TokenFilters Examines and optionally modifies each Token in the Token Stream *From Solrin Action, Chapter 6
Text Analysis Refresher A text field in Lucene/Solrhas an Analyzer containing: ①Zero or more CharFilters Takes incoming text and “cleans it up” before it is tokenized ②One Tokenizer Splits incoming text into a Token Stream containing Zero or more Tokens ③Zero or more TokenFilters Examines and optionally modifies each Token in the Token Stream *From Solrin Action, Chapter 6
Text Analysis Refresher A text field in Lucene/Solrhas an Analyzer containing: ①Zero or more CharFilters Takes incoming text and “cleans it up” before it is tokenized ②OneTokenizer Splits incoming text into a Token Stream containing Zero or more Tokens ③Zero or more TokenFilters Examines and optionally modifies each Token in the Token Stream *From Solrin Action, Chapter 6
Text Analysis Refresher A text field in Lucene/Solrhas an Analyzer containing: ①Zero or more CharFilters Takes incoming text and “cleans it up” before it is tokenized ②One Tokenizer Splits incoming text into a Token Stream containing Zero or more Tokens ③Zero or more TokenFilters Examines and optionally modifies each Token in the Token Stream *From Solrin Action, Chapter 6
Language-specific Text Analysis
Example English Analysis Chains <fieldTypename="text_en" class="solr.TextField" positionIncrementGap="100"> <analyzer> <tokenizerclass="solr.StandardTokenizerFactory"/> <filter class="solr.StopFilterFactory" words="lang/stopwords_en.txt” ignoreCase="true" /> <filter class="solr.LowerCaseFilterFactory"/> <filter class="solr.EnglishPossessiveFilterFactory"/> <filter class="solr.KeywordMarkerFilterFactory" protected="lang/en_protwords.txt"/> <filter class="solr.PorterStemFilterFactory"/> </analyzer> </fieldType> <fieldTypename="text_en" class="solr.TextField" positionIncrementGap="100"> <analyzer> <charFilterclass="solr.HTMLStripCharFilterFactory"/> <tokenizerclass="solr.WhitespaceTokenizerFactory"/> <filter class="solr.SynonymFilterFactory" synonyms="lang/en_synonyms.txt" IignoreCase="true" expand="true"/> <filter class="solr.WordDelimiterFilterFactory" generateWordParts="1" generateNumberParts="1" catenateWords="1" catenateNumbers="1" catenateAll="0" splitOnCaseChange="1"/> <filter class="solr.ASCIIFoldingFilterFactory"/> <filter class="solr.KStemFilterFactory"/> <filter class="solr.RemoveDuplicatesTokenFilterFactory"/> </analyzer> </fieldType>
Per-language Analysis Chains *Some of the 32 different languages configurations in Appendix B of Solrin Action
Per-language Analysis Chains *Some of the 32 different languages configurations in Appendix B of Solrin Action
Which Stemmer do I choose? *From Solrin Action, Chapter 14
Common English Stemmers *From Solrin Action, Chapter 14
When Stemming goes awry Fixing Stemming Mistakes: •Unfortunately, every stemmer will have problem-cases that aren’t handled as you would expect •Thankfully, Stemmers can be overriden •KeywordMarkerFilter: protects a list of terms you specify from being stemmed •StemmerOverrideFilter: applies a list of custom term mappings you specify Alternate strategy: •Use Lemmatization(root-form analysis) instead of Stemming •Commercial vendorshelp tremendously in this space(see http://www.basistech.com/case-study-career-builder/) •The Hunspellstemmer enables dictionary-based support of varying quality in over 100 languages
Stemming vs. Lemmatization •Stemming: algorithmic manipulation of text, based upon common per-language rules •Lemmatization: finds the dictionary form of a term (lemma means “root”) -dramatically improves precision(only matching terms that “should” match), while not significantly impacting recall(all terms that should match do match). *From Solrin Action, Chapter 14
Multilingual Search Strategies
Multilingual Search Strategies How do you handle: …a different language per document? …multiple languages in the same document? …multiple languages in the same field? Strategies: 1)Separate field per language 2)Separate collection/core per language 3)All languages in one field
Strategy 1: Separate field per language *From Solrin Action, Chapter 14
Separate field per language <field name="id" type="string" indexed="true" stored="true" /> <field name="title" type="string" indexed="true" stored="true" /> <field name="content_english" type="text_english" indexed="true” stored="true" /> <field name="content_french" type="text_french" indexed="true” stored="true" /> <field name="content_spanish" type="text_spanish" indexed="true” stored="true" /> <fieldTypename="text_english" class="solr.TextField" positionIncrementGap="100"> <analyzer> <tokenizerclass="solr.StandardTokenizerFactory"/> <filter class="solr.StopFilterFactory” ignoreCase="true" words="lang/stopwords_en.txt"/> <filter class="solr.LowerCaseFilterFactory"/> <filter class="solr.EnglishPossessiveFilterFactory"/> <filter class="solr.KeywordMarkerFilterFactory" protected="protwords.txt"/> <filter class="solr.KStemFilterFactory"/> </analyzer> </fieldType> <fieldTypename="text_spanish" class="solr.TextField" positionIncrementGap="100"> <analyzer> <tokenizerclass="solr.StandardTokenizerFactory"/> <filter class="solr.LowerCaseFilterFactory"/> <filter class="solr.StopFilterFactory" ignoreCase="true" words="lang/ stopwords_es.txt" format="snowball"/> <filter class="solr.SpanishLightStemFilterFactory"/> </analyzer> </fieldType> <fieldTypename="text_french" class="solr.TextField" positionIncrementGap="100"> <analyzer> <tokenizerclass="solr.StandardTokenizerFactory"/> <filter class="solr.ElisionFilterFactory” ignoreCase="true" articles="lang/contractions_fr.txt"/> <filter class="solr.LowerCaseFilterFactory"/> <filter class="solr.StopFilterFactory" ignoreCase="true" words="lang/stopwords_fr.txt” format="snowball"/> <filter class="solr.FrenchLightStemFilterFactory"/> </analyzer> </fieldType> schema.xml *From Solrin Action, Chapter 14
Separate field per language: one language per document <doc> <field name="id">1</field> <fieldname="title">The Adventures of Huckleberry Finn</field> <field name="content_english">YOU don't know about me without you have read a book by the name of The Adventures of Tom Sawyer; but that ain'tno matter. That book was made by Mr. Mark Twain, and he told the truth, mainly. There was things which he stretched, but mainly he told the truth. <field> </doc> <doc> <field name="id ">2</field> <field name="title">Les Misérables</field> <field name="content_french">Nuln'auraitpule dire; tout cequ'onsavait, c'estque, lorsqu'ilrevintd'Italie, ilétaitprêtre. </field> </doc> <doc> <field name="id">3</field> <field name="title">Don Quixote</field> <field name="content_spanish">Demasiadacordurapuedeserla peorde las locuras, verla vidacomoesy no comodeberíade ser. </field> </doc> Query: http://localhost:8983/solr/field-per-language/select? fl=title& defType=edismax& qf=content_englishcontent_frenchcontent_spanish& q="he told the truth" OR"ilétaitprêtre" OR"verla vidacomoes" Response: { "response":{"numFound":3,"start":0,"docs":[ { "title":["The Adventures of Huckleberry Finn"]}, { "title":["Don Quixote"]}, { "title":["Les Misérables"]}] } *From Solrin Action, Chapter 14
Separate field per language: multiple languages per document Query 1: http://localhost:8983/solr/field-per-language/select? fl=title& defType=edismax& qf=content_englishcontent_frenchcontent_spanish& q="wisdom” Query 2: http://localhost:8983/solr/field-per-language/select?... q="sabiduría” Query 3: http://localhost:8983/solr/field-per-language/select?... q="sagesse” Response: (same for queries 1–3) { "response":{"numFound":1,"start":0,"docs":[ { "title":["Proverbs"]}] } Documents: <doc> <field name="id">4</field> <field name="title">Proverbs</field> <field name="content_spanish"> No la abandonesy ellavelarásobre ti, ámalay ellateprotegerá. Lo principal esla sabiduría; adquiere sabiduría, y con todolo queobtengasadquiereinteligencia. </field> <field name="content_english">Do not forsake wisdom, and she will protect you; love her, and she will watch over you. Wisdom is supreme; therefore get wisdom. Though it cost all you have, get understanding. </field> <field name="content_french">N'abandonnepas la sagesse, et ellete gardera, aime-la, et elleteprotégera. Voicile début de la sagesse: acquierslasagesse, procure-toile discernementau prix de tout cequetupossèdes. <field> </doc> *From Solrin Action, Chapter 14
Summary: Separate field per language *From Solrin Action, Chapter 14
Strategy 2: Separate collection per language *From Solrin Action, Chapter 14
Separate collection per language: schema.xml *From Solrin Action, Chapter 14
Separate collection per language: Indexing & Querying Indexing: cd $SOLR_IN_ACTION/example-docs/ java -jar -Durl=http://localhost:8983/solr/english/update post.jar ➥ch14/documents/english.xml java -jar -Durl=http://localhost:8983/solr/spanish/update post.jar ➥ch14/documents/spanish.xml java -jar -Durl=http://localhost:8983/solr/french/update post.jar ➥ch14/documents/french.xml Query (collections in SolrCloud): http://localhost:8983/solr/aggregator/select? shards=english,spanish,french df=content& q=query in any language here Query (specific cores): http://localhost:8983/solr/aggregator/select? shards=localhost:8983/solr/english, localhost:8983/solr/spanish, localhost:8983/solr/french& df=content& q=query in any language here Documents: All documents just have a single “content” field. The documents get routedto a different language-specific Solrcollection based upon the language of the content field. *From Solrin Action, Chapter 14
Summary: Separate index per language *From Solrin Action, Chapter 14
Strategy 3: One Field for all languages *From Solrin Action, Chapter 14
One Field for all languages: Feature Status •Note: This feature is not yet committed to Solr •I’m working on it in my free time. Currently it supports: •Update Request Processorwhich canautomatically detect the languages of documentsand choose the correct analyzers •Field Type which allows dynamically choosing one or more analyzers on a per-field (indexing) and per term (querying) basis. •Current Code from Solr in Actionis available and is freely available on github. •There is a JIRA ticket open to ultimately contribute this back to Solr: Solr-6492 •Some work is still necessary to make querying more user friendly.
One Field for all languages Step 1: Define Multilingual Field schema.xml: <fieldTypename="multilingual_text" class="sia.ch14.MultiTextField" sortMissingLast="true" defaultFieldType="text_general" fieldMappings="en:text_english, es:text_spanish, fr:text_french, de:text_german"/>[1] <field name="text" type="multilingual_text" indexed="true" multiValued="true" /> [1]Note that "text_english", "text_spanish", "text_french", and "text_german" refer to field types defined elsewhere in the schema.xml [2]Uses the "defaultFieldType", in this case "text_general", defined elsewhere in schema.xml <add><doc>… <field name="text">general keywords</field> [2] <field name="text”>en,es|theschool, lasescuelas</field>… </doc></add> <add><doc>… <field name="text">en|theschool</field> <field name="text">es|lasescuelas</field>… </doc></add> Step 2: Index documents http://localhost:8983/solr/collection1/select? q=es|escuelaOR en,es,de|schoolOR school [2] Step 3: Search
One Field For All Languages: Stacked Token Streams 1) English Field 2) Spanish Field 3) English + Spanish combined in Multilingual Text Field multilingual_text ①For each language requested, the appropriate field type is chosen ②The input text is passed separately to the Analyzer chain for each field type ③The resulting Token Streams from each Analyzer chain arestacked into a unified Token Stream based upon their position increments *Screenshot from Solrin Action, Chapter 14
Strategy 3: All languages in one field * *See Solrin Action, Chapter 14
Automatic Language Identification
Identifying languages in documents solrconfig.xml ... <updateRequestProcessorChainname="langid"> <processorclass="org.apache.solr.update.processor. LangDetectLanguageIdentifierUpdateProcessorFactory"> <lstname="invariants"> <strname="langid.fl">content, content_lang1,content_lang2,content_lang3</str> <strname="langid.langField">language</str> <strname="langid.langsField">languages</str> ... </lst> </processor> .. </updateRequestProcessorChain> … <requestHandlername="/update" class="solr.UpdateRequestHandler"> <lstname="invariants"> <strname="update.chain">langid</str> </lst> </requestHandler> ... schema.xml ... <field name="language" type="string" indexed="true" stored="true" /> <field name="languages" type="string" indexed="true" stored="true" multiValued="true"/> ... *See Solrin Action, Chapter 14
Identifying languages in documents Sending documents: cd $SOLR_IN_ACTION/example-docs/ java -Durl=http://localhost:8983/solr/langid/update ➥-jar post.jarch14/documents/langid.xml Query http://localhost:8983/solr/langid/select? q=*:*& fl=title,language,languages Results [{ "title":"TheAdventures of HuckelberryFinn", "language":"en", "languages":["en"]}, { "title":"LesMisérables", "language":"fr", "languages":["fr"]}, { "title":"DonQuoxite", "language":"es", "languages":["es"]}, { "title":"Proverbs", "language":"fr", "languages":["fr”, "en”,"es"]}] *See Solrin Action, Chapter 14
Mapping data to language-specific fields solrconfig.xml ... <updateRequestProcessorChainname="langid"> <processorclass="org.apache.solr.update.processor. LangDetectLanguageIdentifierUpdateProcessorFactory"> <lstname="invariants"> <strname="langid.fl">content</str> <strname="langid.langField">language</str> <strname="langid.map">true</str> <strname="langid.map.fl">content</str> <strname="langid.whitelist">en,es,fr</str> <strname="langid.map.lcmap"> en:englishes:spanishfr:french</str> <strname="langid.fallback">en</str> </lst> </processor> ... </updateRequestProcessorChain> ... Indexed Documents: [{ "title":"TheAdventures of Huckleberry Finn", "language":"en", "content_english":[ "YOU don't know about me without..."]}, { "title":"LesMisérables", "language":"fr", "content_french":[ "Nuln'auraitpule dire; tout ce..."]}, { "title":"DonQuixote", "language":"es", "content_spanish":[ "Demasiadacordurapuedeserla peor..."]}] }] *See Solrin Action, Chapter 14
Semantic Strategies
The need for Semantic Search User’s Query: machine learning research and development Portland, OR software engineer AND hadoopjava Traditional Query Parsing: (machine ANDlearningANDresearch ANDdevelopmentANDportland) OR(software ANDengineer ANDhadoopANDjava) Semantic Query Parsing: "machine learning" AND"research and development" AND"Portland, OR” AND"software engineer" ANDhadoopANDjava Semantically Expanded Query: ("machine learning"^10OR"data scientist" OR"data mining" OR"computer vision") AND("research and development"^10OR"r&d") ANDAND("Portland, OR"^10OR"Portland, Oregon" OR{!geofiltpt=45.512,-122.676 d=50sfield=geo}) AND("software engineer"^10OR"software developer") AND(hadoop^10OR"big data" ORhbaseORhive) AND(java^10 ORj2ee)
Semantic Search Architecture –Query Parsing 1)Generate Model of Domain-specific phrases •Can mine query logs or actual text of documents for significant phrases within your domain [1] 2)Feed known phrases to SolrTextTagger(uses LuceneFST for high-throughput term lookups) 3)Use SolrTextTaggerto perform entity extraction on incoming queries(tagging documents is also optional) 4)Shown on next slide: Pass extracted entities to a Query Augmentation phase to rewrite query with enhanced semantic understanding(synonyms, related keywords, related categories, etc.) [1] K. Aljadda, M. Korayem, T. Grainger, C. Russell. "CrowdsourcedQuery Augmentation through Semantic Discovery of Domain-specific Jargon," in IEEE Big Data 2014. [2]https://github.com/OpenSextant/SolrTextTagger
machine learning Keywords: Search Behavior, Application Behavior, etc. Job Title Classifier, Skills Extractor, Job Level Classifier, etc. Clustering relationships Semantic Query Augmentation keywords:((machine learning)^10OR { AT_LEAST_2: ("data mining"^0.9,matlab^0.8, "data scientist"^0.75, "artificial intelligence"^0.7, "neural networks"^0.55))} { BOOST_TO_TOP:(job_title:( "software engineer" OR "data manager" OR "data scientist" OR "hadoopengineer"))} Modified Query: Related Occupations machine learning: {15-1031.00 .58Computer Software Engineers, Applications 15-1011.00 .55 Computer and Information Scientists, Research 15-1032.00 .52 Computer Software Engineers, Systems Software } machine learning: { software engineer .65, data manager .3, data scientist .25, hadoopengineer .2, } Common Job Titles Semantic Search Architecture –Query Augmentation Related Phrases machine learning: { data mining .9, matlab.8, data scientist .75, artificial intelligence .7, neural networks .55 } Known keyword phrases java developer machine learningregistered nurse
Differentiating related terms Synonyms: cpa=> certified public accountant rn=> registered nurser.n. => registered nurseAmbiguous Terms*: driver=> driver (trucking)~80% driver => driver (software)~20% Related Terms: r.n. => nursing, bsnhadoop=> mapreduce, hive, pig *differentiated based upon user and query context
Semantic Search “under the hood”
2014 Publications & Presentations Books: Solrin Action-A comprehensive guide to implementing scalable search using Apache Solr Research papers: ●Towards a Job title Classification System ●Augmenting Recommendation Systems Using a Model of Semantically-related Terms Extracted from User Behavior ●sCooL: A system for academic institution name normalization ●CrowdsourcedQuery Augmentation through Semantic Discovery of Domain-specific jargon ●PGMHD: A Scalable Probabilistic Graphical Model for Massive Hierarchical Data Problems ●SKILL: A System for Skill Identification and Normalization Speaking Engagements: ●WSDM 2014 Workshop: “Web-Scale Classification: Classifying Big Data from the Web” ●Atlanta SolrMeetup ●Atlanta Big Data Meetup ●The Second International Symposium on Big Data and Data Analytics ●Lucene/SolrRevolution 2014 ●RecSys2014 ●IEEE Big Data Conference 2014
Conclusion •Language analysis options for each language are very configurable •There are multiple strategies for handling multilingual content based upon your use case •When in doubt, automatic language detection can be easily leveraged in your indexing pipeline •The next generation of query/relevancy improvements will be able to understand the intent of the user.
Contact Info Yes, WE ARE HIRING@CareerBuilder. Come talk with me if you are interested… Trey Grainger trey.grainger@careerbuilder.com@treygrainger http://solrinaction.com Conference discount (43% off):lusorevcftw Other presentations: http://www.treygrainger.com
Previous presentations:

More Related Content

PDF
Information Extraction
Rubén Izquierdo Beviá
 
PPTX
Natural Language processing
Sanzid Kawsar
 
PDF
Phonetic Matching with Apache Solr
Markus Günther
 
PPT
Pratique De L’Orientation Scolaire Et Aggravation De La Pauvreté Dans Les MéN...
ROCARE / ERNWACA
 
PPTX
NLP Project Presentation
Aryak Sengupta
 
PDF
Paraphrase Detection in NLP
Yuriy Guts
 
PDF
ナイーブベイズによる言語判定
Shuyo Nakatani
 
PPT
Natural language processing
Basha Chand
 
Information Extraction
Rubén Izquierdo Beviá
 
Natural Language processing
Sanzid Kawsar
 
Phonetic Matching with Apache Solr
Markus Günther
 
Pratique De L’Orientation Scolaire Et Aggravation De La Pauvreté Dans Les MéN...
ROCARE / ERNWACA
 
NLP Project Presentation
Aryak Sengupta
 
Paraphrase Detection in NLP
Yuriy Guts
 
ナイーブベイズによる言語判定
Shuyo Nakatani
 
Natural language processing
Basha Chand
 

What's hot (20)

PPT
Introduction to Natural Language Processing
rohitnayak
 
PPTX
Natural language processing: feature extraction
Gabriel Hamilton
 
PPTX
A Simple Introduction to Word Embeddings
Bhaskar Mitra
 
PDF
Consuming RealTime Signals in Solr
Umesh Prasad
 
PPTX
Introduction to Text Mining
Minha Hwang
 
PPTX
Opinion Mining
Nour El Houda Megherbi
 
PPTX
Introduction to Named Entity Recognition
Tomer Lieber
 
PPT
Informatique Quantique
Cynapsys It Hotspot
 
PPTX
Using Signals in Lucidworks Fusion
Lucidworks
 
PPTX
Flink Forward San Francisco 2019: Moving from Lambda and Kappa Architectures ...
Flink Forward
 
PDF
入門Transducers
sohta
 
PDF
Data Security at Scale through Spark and Parquet Encryption
Databricks
 
PDF
Flexible Indexing in Lucene 4.0
Lucidworks (Archived)
 
PDF
Introduction to Kneser-Ney Smoothing on Top of Generalized Language Models fo...
Martin Körner
 
PPTX
Information retrieval 9 tf idf weights
Vaibhav Khanna
 
PPTX
[Paper review] BERT
JEE HYUN PARK
 
PDF
Introduction To Apache Lucene
Mindfire Solutions
 
PPTX
Demystifying the use of circuit breakers with MuleSoft
Sandeep Deshmukh
 
PPTX
Metrique
Amine AOUNZOU
 
PDF
Cloud-Native Apache Spark Scheduling with YuniKorn Scheduler
Databricks
 
Introduction to Natural Language Processing
rohitnayak
 
Natural language processing: feature extraction
Gabriel Hamilton
 
A Simple Introduction to Word Embeddings
Bhaskar Mitra
 
Consuming RealTime Signals in Solr
Umesh Prasad
 
Introduction to Text Mining
Minha Hwang
 
Opinion Mining
Nour El Houda Megherbi
 
Introduction to Named Entity Recognition
Tomer Lieber
 
Informatique Quantique
Cynapsys It Hotspot
 
Using Signals in Lucidworks Fusion
Lucidworks
 
Flink Forward San Francisco 2019: Moving from Lambda and Kappa Architectures ...
Flink Forward
 
入門Transducers
sohta
 
Data Security at Scale through Spark and Parquet Encryption
Databricks
 
Flexible Indexing in Lucene 4.0
Lucidworks (Archived)
 
Introduction to Kneser-Ney Smoothing on Top of Generalized Language Models fo...
Martin Körner
 
Information retrieval 9 tf idf weights
Vaibhav Khanna
 
[Paper review] BERT
JEE HYUN PARK
 
Introduction To Apache Lucene
Mindfire Solutions
 
Demystifying the use of circuit breakers with MuleSoft
Sandeep Deshmukh
 
Metrique
Amine AOUNZOU
 
Cloud-Native Apache Spark Scheduling with YuniKorn Scheduler
Databricks
 
Ad

Similar to Semantic & Multilingual Strategies in Lucene/Solr (20)

PDF
Basics of Solr and Solr Integration with AEM6
DEEPAK KHETAWAT
 
PPTX
Ir 03
Mohammed Romi
 
PPTX
The Apache Solr Smart Data Ecosystem
Trey Grainger
 
ODP
Lucene And Solr Intro
pascaldimassimo
 
PPTX
Text based search engine on a fixed corpus and utilizing indexation and ranki...
Soham Mondal
 
PDF
Elasticsearch Analyzers Field-Level Optimization.pdf
Inexture Solutions
 
PDF
Find it, possibly also near you!
Paul Borgermans
 
PDF
Lucene for Solr Developers
Erik Hatcher
 
PPT
CrashCourse: XML technologies
ESRI Bulgaria
 
PDF
Lucene for Solr Developers
Erik Hatcher
 
PDF
Intro to Elasticsearch
Clifford James
 
PPTX
Self-learned Relevancy with Apache Solr
Trey Grainger
 
PPTX
Building Search & Recommendation Engines
Trey Grainger
 
PPTX
Text analytics
Utkarsh Sharma
 
PDF
Get the most out of Solr search with PHP
Paul Borgermans
 
PDF
Search pitb
Nawab Iqbal
 
PPT
Enterprise Search Solution: Apache SOLR. What's available and why it's so cool
Ecommerce Solution Provider SysIQ
 
PDF
Search explained T3DD15
Hans Höchtl
 
PPT
Lucene Bootcamp -1
GokulD
 
PPTX
NLTK
Girish Khanzode
 
Basics of Solr and Solr Integration with AEM6
DEEPAK KHETAWAT
 
Ir 03
Mohammed Romi
 
The Apache Solr Smart Data Ecosystem
Trey Grainger
 
Lucene And Solr Intro
pascaldimassimo
 
Text based search engine on a fixed corpus and utilizing indexation and ranki...
Soham Mondal
 
Elasticsearch Analyzers Field-Level Optimization.pdf
Inexture Solutions
 
Find it, possibly also near you!
Paul Borgermans
 
Lucene for Solr Developers
Erik Hatcher
 
CrashCourse: XML technologies
ESRI Bulgaria
 
Lucene for Solr Developers
Erik Hatcher
 
Intro to Elasticsearch
Clifford James
 
Self-learned Relevancy with Apache Solr
Trey Grainger
 
Building Search & Recommendation Engines
Trey Grainger
 
Text analytics
Utkarsh Sharma
 
Get the most out of Solr search with PHP
Paul Borgermans
 
Search pitb
Nawab Iqbal
 
Enterprise Search Solution: Apache SOLR. What's available and why it's so cool
Ecommerce Solution Provider SysIQ
 
Search explained T3DD15
Hans Höchtl
 
Lucene Bootcamp -1
GokulD
 
NLTK
Girish Khanzode
 
Ad

More from Trey Grainger (20)

PDF
Balancing the Dimensions of User Intent
Trey Grainger
 
PDF
Reflected Intelligence: Real world AI in Digital Transformation
Trey Grainger
 
PDF
Thought Vectors and Knowledge Graphs in AI-powered Search
Trey Grainger
 
PDF
Natural Language Search with Knowledge Graphs (Chicago Meetup)
Trey Grainger
 
PDF
The Next Generation of AI-powered Search
Trey Grainger
 
PDF
Natural Language Search with Knowledge Graphs (Activate 2019)
Trey Grainger
 
PDF
AI, Search, and the Disruption of Knowledge Management
Trey Grainger
 
PDF
Measuring Relevance in the Negative Space
Trey Grainger
 
PDF
Natural Language Search with Knowledge Graphs (Haystack 2019)
Trey Grainger
 
PDF
The Future of Search and AI
Trey Grainger
 
PPTX
How to Build a Semantic Search System
Trey Grainger
 
PPTX
The Relevance of the Apache Solr Semantic Knowledge Graph
Trey Grainger
 
PPTX
Searching for Meaning
Trey Grainger
 
PPTX
The Intent Algorithms of Search & Recommendation Engines
Trey Grainger
 
PPTX
The Apache Solr Semantic Knowledge Graph
Trey Grainger
 
PPTX
Intent Algorithms: The Data Science of Smart Information Retrieval Systems
Trey Grainger
 
PPTX
South Big Data Hub: Text Data Analysis Panel
Trey Grainger
 
PPTX
The Semantic Knowledge Graph
Trey Grainger
 
PPTX
Reflected Intelligence: Lucene/Solr as a self-learning data system
Trey Grainger
 
PDF
Reflected intelligence evolving self-learning data systems
Trey Grainger
 
Balancing the Dimensions of User Intent
Trey Grainger
 
Reflected Intelligence: Real world AI in Digital Transformation
Trey Grainger
 
Thought Vectors and Knowledge Graphs in AI-powered Search
Trey Grainger
 
Natural Language Search with Knowledge Graphs (Chicago Meetup)
Trey Grainger
 
The Next Generation of AI-powered Search
Trey Grainger
 
Natural Language Search with Knowledge Graphs (Activate 2019)
Trey Grainger
 
AI, Search, and the Disruption of Knowledge Management
Trey Grainger
 
Measuring Relevance in the Negative Space
Trey Grainger
 
Natural Language Search with Knowledge Graphs (Haystack 2019)
Trey Grainger
 
The Future of Search and AI
Trey Grainger
 
How to Build a Semantic Search System
Trey Grainger
 
The Relevance of the Apache Solr Semantic Knowledge Graph
Trey Grainger
 
Searching for Meaning
Trey Grainger
 
The Intent Algorithms of Search & Recommendation Engines
Trey Grainger
 
The Apache Solr Semantic Knowledge Graph
Trey Grainger
 
Intent Algorithms: The Data Science of Smart Information Retrieval Systems
Trey Grainger
 
South Big Data Hub: Text Data Analysis Panel
Trey Grainger
 
The Semantic Knowledge Graph
Trey Grainger
 
Reflected Intelligence: Lucene/Solr as a self-learning data system
Trey Grainger
 
Reflected intelligence evolving self-learning data systems
Trey Grainger
 

Recently uploaded (20)

PDF
7 ChatGPT Prompts to Help You Define Your Ideal Customer Profile.pdf
SOFTTECHHUB
 
PDF
Unlocking AI with Model Context Protocol (MCP)
Brian McKeiver
 
PPTX
KOM of Painting work and Equipment Insulation REV00 update 25-dec.pptx
muhammadmuneebnaeem7
 
PDF
Encapsulation_ Review paper, used for researhc scholars
gurumoop
 
PPT
Teaching material agriculture food technology
LiaRayya
 
PDF
Electronic commerce courselecture one. Pdf
OmerMohamed64
 
PDF
Network Security Unit 5.pdf for BCA BBA.
Serpent6
 
PPTX
Big Data Technologies - Introduction.pptx
kuthubussaman1
 
PPTX
20250228 LYD VKU AI Blended-Learning.pptx
DatVoNgoc
 
PPTX
sap open course for s4hana steps from ECC to s4
sreeni2106invites
 
PPTX
Cloud computing and distributed systems.
kkkkkhan
 
PDF
Dropbox Q2 2025 Financial Results & Investor Presentation
Dropbox
 
PDF
cuic standard and advanced reporting.pdf
SimoneTitaniumTomase
 
PDF
Review of recent advances in non-invasive hemoglobin estimation
IAESIJAI
 
PDF
How UI/UX Design Impacts User Retention in Mobile Apps.pdf
SynapseIndia
 
PPTX
Detection-First SIEM: Rule Types, Dashboards, and Threat-Informed Strategy
ReZa AdineH
 
PDF
Empathic Computing: Creating Shared Understanding
Mark Billinghurst
 
PPT
“AI and Expert System Decision Support & Business Intelligence Systems”
ZubinRadhakrishnan
 
PDF
Machine learning based COVID-19 study performance prediction
IAESIJAI
 
PDF
Spectral efficient network and resource selection model in 5G networks
IAESIJAI
 
7 ChatGPT Prompts to Help You Define Your Ideal Customer Profile.pdf
SOFTTECHHUB
 
Unlocking AI with Model Context Protocol (MCP)
Brian McKeiver
 
KOM of Painting work and Equipment Insulation REV00 update 25-dec.pptx
muhammadmuneebnaeem7
 
Encapsulation_ Review paper, used for researhc scholars
gurumoop
 
Teaching material agriculture food technology
LiaRayya
 
Electronic commerce courselecture one. Pdf
OmerMohamed64
 
Network Security Unit 5.pdf for BCA BBA.
Serpent6
 
Big Data Technologies - Introduction.pptx
kuthubussaman1
 
20250228 LYD VKU AI Blended-Learning.pptx
DatVoNgoc
 
sap open course for s4hana steps from ECC to s4
sreeni2106invites
 
Cloud computing and distributed systems.
kkkkkhan
 
Dropbox Q2 2025 Financial Results & Investor Presentation
Dropbox
 
cuic standard and advanced reporting.pdf
SimoneTitaniumTomase
 
Review of recent advances in non-invasive hemoglobin estimation
IAESIJAI
 
How UI/UX Design Impacts User Retention in Mobile Apps.pdf
SynapseIndia
 
Detection-First SIEM: Rule Types, Dashboards, and Threat-Informed Strategy
ReZa AdineH
 
Empathic Computing: Creating Shared Understanding
Mark Billinghurst
 
“AI and Expert System Decision Support & Business Intelligence Systems”
ZubinRadhakrishnan
 
Machine learning based COVID-19 study performance prediction
IAESIJAI
 
Spectral efficient network and resource selection model in 5G networks
IAESIJAI
 

Semantic & Multilingual Strategies in Lucene/Solr

  • 1. Semantic & Multilingual Strategies in Lucene/Solr Trey Grainger Director of Engineering, Search & Analytics@CareerBuilder
  • 2. Outline •Introduction •Text Analysis Refresher •Language-specific text Analysis •Multilingual Search Strategies •Automatic Language Identification •Semantic Search Strategies (understanding “meaning”) •Conclusion
  • 3. About Me Trey Grainger Director of Engineering, Search & Analytics Joined CareerBuilderin 2007 as Software Engineer MBA, Management of Technology –GA Tech BA, Computer Science, Business, & Philosophy –Furman University Mining Massive Datasets (in progress) -Stanford University Fun outside of CB: •Co-author of Solr in Action, plus several research papers •Frequent conference speaker •Founder of Celiaccess.com, the gluten-free search engine •Lucene/Solrcontributor
  • 6. Text Analysis Refresher A text field in Lucene/Solrhas an Analyzer containing: ①Zero or more CharFilters Takes incoming text and “cleans it up” before it is tokenized ②One Tokenizer Splits incoming text into a Token Stream containing Zero or more Tokens ③Zero or more TokenFilters Examines and optionally modifies each Token in the Token Stream *From Solrin Action, Chapter 6
  • 7. Text Analysis Refresher A text field in Lucene/Solrhas an Analyzer containing: ①Zero or more CharFilters Takes incoming text and “cleans it up” before it is tokenized ②One Tokenizer Splits incoming text into a Token Stream containing Zero or more Tokens ③Zero or more TokenFilters Examines and optionally modifies each Token in the Token Stream *From Solrin Action, Chapter 6
  • 8. Text Analysis Refresher A text field in Lucene/Solrhas an Analyzer containing: ①Zero or more CharFilters Takes incoming text and “cleans it up” before it is tokenized ②OneTokenizer Splits incoming text into a Token Stream containing Zero or more Tokens ③Zero or more TokenFilters Examines and optionally modifies each Token in the Token Stream *From Solrin Action, Chapter 6
  • 9. Text Analysis Refresher A text field in Lucene/Solrhas an Analyzer containing: ①Zero or more CharFilters Takes incoming text and “cleans it up” before it is tokenized ②One Tokenizer Splits incoming text into a Token Stream containing Zero or more Tokens ③Zero or more TokenFilters Examines and optionally modifies each Token in the Token Stream *From Solrin Action, Chapter 6
  • 11. Example English Analysis Chains <fieldTypename="text_en" class="solr.TextField" positionIncrementGap="100"> <analyzer> <tokenizerclass="solr.StandardTokenizerFactory"/> <filter class="solr.StopFilterFactory" words="lang/stopwords_en.txt” ignoreCase="true" /> <filter class="solr.LowerCaseFilterFactory"/> <filter class="solr.EnglishPossessiveFilterFactory"/> <filter class="solr.KeywordMarkerFilterFactory" protected="lang/en_protwords.txt"/> <filter class="solr.PorterStemFilterFactory"/> </analyzer> </fieldType> <fieldTypename="text_en" class="solr.TextField" positionIncrementGap="100"> <analyzer> <charFilterclass="solr.HTMLStripCharFilterFactory"/> <tokenizerclass="solr.WhitespaceTokenizerFactory"/> <filter class="solr.SynonymFilterFactory" synonyms="lang/en_synonyms.txt" IignoreCase="true" expand="true"/> <filter class="solr.WordDelimiterFilterFactory" generateWordParts="1" generateNumberParts="1" catenateWords="1" catenateNumbers="1" catenateAll="0" splitOnCaseChange="1"/> <filter class="solr.ASCIIFoldingFilterFactory"/> <filter class="solr.KStemFilterFactory"/> <filter class="solr.RemoveDuplicatesTokenFilterFactory"/> </analyzer> </fieldType>
  • 12. Per-language Analysis Chains *Some of the 32 different languages configurations in Appendix B of Solrin Action
  • 13. Per-language Analysis Chains *Some of the 32 different languages configurations in Appendix B of Solrin Action
  • 14. Which Stemmer do I choose? *From Solrin Action, Chapter 14
  • 15. Common English Stemmers *From Solrin Action, Chapter 14
  • 16. When Stemming goes awry Fixing Stemming Mistakes: •Unfortunately, every stemmer will have problem-cases that aren’t handled as you would expect •Thankfully, Stemmers can be overriden •KeywordMarkerFilter: protects a list of terms you specify from being stemmed •StemmerOverrideFilter: applies a list of custom term mappings you specify Alternate strategy: •Use Lemmatization(root-form analysis) instead of Stemming •Commercial vendorshelp tremendously in this space(see http://www.basistech.com/case-study-career-builder/) •The Hunspellstemmer enables dictionary-based support of varying quality in over 100 languages
  • 17. Stemming vs. Lemmatization •Stemming: algorithmic manipulation of text, based upon common per-language rules •Lemmatization: finds the dictionary form of a term (lemma means “root”) -dramatically improves precision(only matching terms that “should” match), while not significantly impacting recall(all terms that should match do match). *From Solrin Action, Chapter 14
  • 19. Multilingual Search Strategies How do you handle: …a different language per document? …multiple languages in the same document? …multiple languages in the same field? Strategies: 1)Separate field per language 2)Separate collection/core per language 3)All languages in one field
  • 20. Strategy 1: Separate field per language *From Solrin Action, Chapter 14
  • 21. Separate field per language <field name="id" type="string" indexed="true" stored="true" /> <field name="title" type="string" indexed="true" stored="true" /> <field name="content_english" type="text_english" indexed="true” stored="true" /> <field name="content_french" type="text_french" indexed="true” stored="true" /> <field name="content_spanish" type="text_spanish" indexed="true” stored="true" /> <fieldTypename="text_english" class="solr.TextField" positionIncrementGap="100"> <analyzer> <tokenizerclass="solr.StandardTokenizerFactory"/> <filter class="solr.StopFilterFactory” ignoreCase="true" words="lang/stopwords_en.txt"/> <filter class="solr.LowerCaseFilterFactory"/> <filter class="solr.EnglishPossessiveFilterFactory"/> <filter class="solr.KeywordMarkerFilterFactory" protected="protwords.txt"/> <filter class="solr.KStemFilterFactory"/> </analyzer> </fieldType> <fieldTypename="text_spanish" class="solr.TextField" positionIncrementGap="100"> <analyzer> <tokenizerclass="solr.StandardTokenizerFactory"/> <filter class="solr.LowerCaseFilterFactory"/> <filter class="solr.StopFilterFactory" ignoreCase="true" words="lang/ stopwords_es.txt" format="snowball"/> <filter class="solr.SpanishLightStemFilterFactory"/> </analyzer> </fieldType> <fieldTypename="text_french" class="solr.TextField" positionIncrementGap="100"> <analyzer> <tokenizerclass="solr.StandardTokenizerFactory"/> <filter class="solr.ElisionFilterFactory” ignoreCase="true" articles="lang/contractions_fr.txt"/> <filter class="solr.LowerCaseFilterFactory"/> <filter class="solr.StopFilterFactory" ignoreCase="true" words="lang/stopwords_fr.txt” format="snowball"/> <filter class="solr.FrenchLightStemFilterFactory"/> </analyzer> </fieldType> schema.xml *From Solrin Action, Chapter 14
  • 22. Separate field per language: one language per document <doc> <field name="id">1</field> <fieldname="title">The Adventures of Huckleberry Finn</field> <field name="content_english">YOU don't know about me without you have read a book by the name of The Adventures of Tom Sawyer; but that ain'tno matter. That book was made by Mr. Mark Twain, and he told the truth, mainly. There was things which he stretched, but mainly he told the truth. <field> </doc> <doc> <field name="id ">2</field> <field name="title">Les Misérables</field> <field name="content_french">Nuln'auraitpule dire; tout cequ'onsavait, c'estque, lorsqu'ilrevintd'Italie, ilétaitprêtre. </field> </doc> <doc> <field name="id">3</field> <field name="title">Don Quixote</field> <field name="content_spanish">Demasiadacordurapuedeserla peorde las locuras, verla vidacomoesy no comodeberíade ser. </field> </doc> Query: http://localhost:8983/solr/field-per-language/select? fl=title& defType=edismax& qf=content_englishcontent_frenchcontent_spanish& q="he told the truth" OR"ilétaitprêtre" OR"verla vidacomoes" Response: { "response":{"numFound":3,"start":0,"docs":[ { "title":["The Adventures of Huckleberry Finn"]}, { "title":["Don Quixote"]}, { "title":["Les Misérables"]}] } *From Solrin Action, Chapter 14
  • 23. Separate field per language: multiple languages per document Query 1: http://localhost:8983/solr/field-per-language/select? fl=title& defType=edismax& qf=content_englishcontent_frenchcontent_spanish& q="wisdom” Query 2: http://localhost:8983/solr/field-per-language/select?... q="sabiduría” Query 3: http://localhost:8983/solr/field-per-language/select?... q="sagesse” Response: (same for queries 1–3) { "response":{"numFound":1,"start":0,"docs":[ { "title":["Proverbs"]}] } Documents: <doc> <field name="id">4</field> <field name="title">Proverbs</field> <field name="content_spanish"> No la abandonesy ellavelarásobre ti, ámalay ellateprotegerá. Lo principal esla sabiduría; adquiere sabiduría, y con todolo queobtengasadquiereinteligencia. </field> <field name="content_english">Do not forsake wisdom, and she will protect you; love her, and she will watch over you. Wisdom is supreme; therefore get wisdom. Though it cost all you have, get understanding. </field> <field name="content_french">N'abandonnepas la sagesse, et ellete gardera, aime-la, et elleteprotégera. Voicile début de la sagesse: acquierslasagesse, procure-toile discernementau prix de tout cequetupossèdes. <field> </doc> *From Solrin Action, Chapter 14
  • 24. Summary: Separate field per language *From Solrin Action, Chapter 14
  • 25. Strategy 2: Separate collection per language *From Solrin Action, Chapter 14
  • 26. Separate collection per language: schema.xml *From Solrin Action, Chapter 14
  • 27. Separate collection per language: Indexing & Querying Indexing: cd $SOLR_IN_ACTION/example-docs/ java -jar -Durl=http://localhost:8983/solr/english/update post.jar ➥ch14/documents/english.xml java -jar -Durl=http://localhost:8983/solr/spanish/update post.jar ➥ch14/documents/spanish.xml java -jar -Durl=http://localhost:8983/solr/french/update post.jar ➥ch14/documents/french.xml Query (collections in SolrCloud): http://localhost:8983/solr/aggregator/select? shards=english,spanish,french df=content& q=query in any language here Query (specific cores): http://localhost:8983/solr/aggregator/select? shards=localhost:8983/solr/english, localhost:8983/solr/spanish, localhost:8983/solr/french& df=content& q=query in any language here Documents: All documents just have a single “content” field. The documents get routedto a different language-specific Solrcollection based upon the language of the content field. *From Solrin Action, Chapter 14
  • 28. Summary: Separate index per language *From Solrin Action, Chapter 14
  • 29. Strategy 3: One Field for all languages *From Solrin Action, Chapter 14
  • 30. One Field for all languages: Feature Status •Note: This feature is not yet committed to Solr •I’m working on it in my free time. Currently it supports: •Update Request Processorwhich canautomatically detect the languages of documentsand choose the correct analyzers •Field Type which allows dynamically choosing one or more analyzers on a per-field (indexing) and per term (querying) basis. •Current Code from Solr in Actionis available and is freely available on github. •There is a JIRA ticket open to ultimately contribute this back to Solr: Solr-6492 •Some work is still necessary to make querying more user friendly.
  • 31. One Field for all languages Step 1: Define Multilingual Field schema.xml: <fieldTypename="multilingual_text" class="sia.ch14.MultiTextField" sortMissingLast="true" defaultFieldType="text_general" fieldMappings="en:text_english, es:text_spanish, fr:text_french, de:text_german"/>[1] <field name="text" type="multilingual_text" indexed="true" multiValued="true" /> [1]Note that "text_english", "text_spanish", "text_french", and "text_german" refer to field types defined elsewhere in the schema.xml [2]Uses the "defaultFieldType", in this case "text_general", defined elsewhere in schema.xml <add><doc>… <field name="text">general keywords</field> [2] <field name="text”>en,es|theschool, lasescuelas</field>… </doc></add> <add><doc>… <field name="text">en|theschool</field> <field name="text">es|lasescuelas</field>… </doc></add> Step 2: Index documents http://localhost:8983/solr/collection1/select? q=es|escuelaOR en,es,de|schoolOR school [2] Step 3: Search
  • 32. One Field For All Languages: Stacked Token Streams 1) English Field 2) Spanish Field 3) English + Spanish combined in Multilingual Text Field multilingual_text ①For each language requested, the appropriate field type is chosen ②The input text is passed separately to the Analyzer chain for each field type ③The resulting Token Streams from each Analyzer chain arestacked into a unified Token Stream based upon their position increments *Screenshot from Solrin Action, Chapter 14
  • 33. Strategy 3: All languages in one field * *See Solrin Action, Chapter 14
  • 35. Identifying languages in documents solrconfig.xml ... <updateRequestProcessorChainname="langid"> <processorclass="org.apache.solr.update.processor. LangDetectLanguageIdentifierUpdateProcessorFactory"> <lstname="invariants"> <strname="langid.fl">content, content_lang1,content_lang2,content_lang3</str> <strname="langid.langField">language</str> <strname="langid.langsField">languages</str> ... </lst> </processor> .. </updateRequestProcessorChain> … <requestHandlername="/update" class="solr.UpdateRequestHandler"> <lstname="invariants"> <strname="update.chain">langid</str> </lst> </requestHandler> ... schema.xml ... <field name="language" type="string" indexed="true" stored="true" /> <field name="languages" type="string" indexed="true" stored="true" multiValued="true"/> ... *See Solrin Action, Chapter 14
  • 36. Identifying languages in documents Sending documents: cd $SOLR_IN_ACTION/example-docs/ java -Durl=http://localhost:8983/solr/langid/update ➥-jar post.jarch14/documents/langid.xml Query http://localhost:8983/solr/langid/select? q=*:*& fl=title,language,languages Results [{ "title":"TheAdventures of HuckelberryFinn", "language":"en", "languages":["en"]}, { "title":"LesMisérables", "language":"fr", "languages":["fr"]}, { "title":"DonQuoxite", "language":"es", "languages":["es"]}, { "title":"Proverbs", "language":"fr", "languages":["fr”, "en”,"es"]}] *See Solrin Action, Chapter 14
  • 37. Mapping data to language-specific fields solrconfig.xml ... <updateRequestProcessorChainname="langid"> <processorclass="org.apache.solr.update.processor. LangDetectLanguageIdentifierUpdateProcessorFactory"> <lstname="invariants"> <strname="langid.fl">content</str> <strname="langid.langField">language</str> <strname="langid.map">true</str> <strname="langid.map.fl">content</str> <strname="langid.whitelist">en,es,fr</str> <strname="langid.map.lcmap"> en:englishes:spanishfr:french</str> <strname="langid.fallback">en</str> </lst> </processor> ... </updateRequestProcessorChain> ... Indexed Documents: [{ "title":"TheAdventures of Huckleberry Finn", "language":"en", "content_english":[ "YOU don't know about me without..."]}, { "title":"LesMisérables", "language":"fr", "content_french":[ "Nuln'auraitpule dire; tout ce..."]}, { "title":"DonQuixote", "language":"es", "content_spanish":[ "Demasiadacordurapuedeserla peor..."]}] }] *See Solrin Action, Chapter 14
  • 39. The need for Semantic Search User’s Query: machine learning research and development Portland, OR software engineer AND hadoopjava Traditional Query Parsing: (machine ANDlearningANDresearch ANDdevelopmentANDportland) OR(software ANDengineer ANDhadoopANDjava) Semantic Query Parsing: "machine learning" AND"research and development" AND"Portland, OR” AND"software engineer" ANDhadoopANDjava Semantically Expanded Query: ("machine learning"^10OR"data scientist" OR"data mining" OR"computer vision") AND("research and development"^10OR"r&d") ANDAND("Portland, OR"^10OR"Portland, Oregon" OR{!geofiltpt=45.512,-122.676 d=50sfield=geo}) AND("software engineer"^10OR"software developer") AND(hadoop^10OR"big data" ORhbaseORhive) AND(java^10 ORj2ee)
  • 40. Semantic Search Architecture –Query Parsing 1)Generate Model of Domain-specific phrases •Can mine query logs or actual text of documents for significant phrases within your domain [1] 2)Feed known phrases to SolrTextTagger(uses LuceneFST for high-throughput term lookups) 3)Use SolrTextTaggerto perform entity extraction on incoming queries(tagging documents is also optional) 4)Shown on next slide: Pass extracted entities to a Query Augmentation phase to rewrite query with enhanced semantic understanding(synonyms, related keywords, related categories, etc.) [1] K. Aljadda, M. Korayem, T. Grainger, C. Russell. "CrowdsourcedQuery Augmentation through Semantic Discovery of Domain-specific Jargon," in IEEE Big Data 2014. [2]https://github.com/OpenSextant/SolrTextTagger
  • 41. machine learning Keywords: Search Behavior, Application Behavior, etc. Job Title Classifier, Skills Extractor, Job Level Classifier, etc. Clustering relationships Semantic Query Augmentation keywords:((machine learning)^10OR { AT_LEAST_2: ("data mining"^0.9,matlab^0.8, "data scientist"^0.75, "artificial intelligence"^0.7, "neural networks"^0.55))} { BOOST_TO_TOP:(job_title:( "software engineer" OR "data manager" OR "data scientist" OR "hadoopengineer"))} Modified Query: Related Occupations machine learning: {15-1031.00 .58Computer Software Engineers, Applications 15-1011.00 .55 Computer and Information Scientists, Research 15-1032.00 .52 Computer Software Engineers, Systems Software } machine learning: { software engineer .65, data manager .3, data scientist .25, hadoopengineer .2, } Common Job Titles Semantic Search Architecture –Query Augmentation Related Phrases machine learning: { data mining .9, matlab.8, data scientist .75, artificial intelligence .7, neural networks .55 } Known keyword phrases java developer machine learningregistered nurse
  • 42. Differentiating related terms Synonyms: cpa=> certified public accountant rn=> registered nurser.n. => registered nurseAmbiguous Terms*: driver=> driver (trucking)~80% driver => driver (software)~20% Related Terms: r.n. => nursing, bsnhadoop=> mapreduce, hive, pig *differentiated based upon user and query context
  • 44. 2014 Publications & Presentations Books: Solrin Action-A comprehensive guide to implementing scalable search using Apache Solr Research papers: ●Towards a Job title Classification System ●Augmenting Recommendation Systems Using a Model of Semantically-related Terms Extracted from User Behavior ●sCooL: A system for academic institution name normalization ●CrowdsourcedQuery Augmentation through Semantic Discovery of Domain-specific jargon ●PGMHD: A Scalable Probabilistic Graphical Model for Massive Hierarchical Data Problems ●SKILL: A System for Skill Identification and Normalization Speaking Engagements: ●WSDM 2014 Workshop: “Web-Scale Classification: Classifying Big Data from the Web” ●Atlanta SolrMeetup ●Atlanta Big Data Meetup ●The Second International Symposium on Big Data and Data Analytics ●Lucene/SolrRevolution 2014 ●RecSys2014 ●IEEE Big Data Conference 2014
  • 45. Conclusion •Language analysis options for each language are very configurable •There are multiple strategies for handling multilingual content based upon your use case •When in doubt, automatic language detection can be easily leveraged in your indexing pipeline •The next generation of query/relevancy improvements will be able to understand the intent of the user.
  • 46. Contact Info Yes, WE ARE HIRING@CareerBuilder. Come talk with me if you are interested… Trey Grainger trey.grainger@careerbuilder.com@treygrainger http://solrinaction.com Conference discount (43% off):lusorevcftw Other presentations: http://www.treygrainger.com