SlideShare a Scribd company logo

Open source enterprise search and retrieval platform

Download as KEY, PDF
M
mteutelink

The document discusses how Apache open source software is used in implementing an enterprise search and retrieval platform. It provides an overview of enterprise search, including its key features and challenges. It then outlines the logical architecture of an enterprise search solution, covering the collection and publication processes. The collection process involves pulling or pushing content from various sources, validating, enriching and indexing it. The publication process involves handling search requests, filtering, grouping results and returning responses.

1 of 43
Downloaded 23 times
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
Open Source Search & Retrieval Platform Enterprise Search EAI Marc Teutelink Semantic Web Datum 21 augustus 2010
How Apache open source software is used during the implementation of an Enterprise Search and Retrieval Platform (Lucene/SOLR, Nutch, Tika, ServiceMix/Camel, Felix/Ace)
Marc Teutelink marc.teutelink@luminis.eu @mteutelink •Software architect at Luminis •15+ years experience in software development; specialized in Enterprise Search, Enterprise Application Integration and Semantic Web technology •Currently writing “Enterprise Search in Action” for Manning (Mid-2011)
Agenda •Enterprise Search • What is Enterprise Search: Functions and features • Challenges • Logical Architecture •Enterprise Search Solution • Technology Stack • Collection Process • Publication Process • Enricher framework • Deployment •Conclusion
What is Enterprise Search? “Enterprise Search offers a solution for searching, finding and presenting enterprise related information in the larger sense of the word” Enterprise search is all about searching through documents from any type and format from any sources located anywhere with the upmost flexibility • Web search: limited to public documents on the web • Desktop search: limited to private documents on the local machine • Enterprise search: no limitations on document type and location
Enterprise Search (features) •Information Sources and Types • Wide range of sources: local and remote filesystems, content repositories, e-mail, databases, internet, intranet and extranet • Type not limited: any type ranging from structured to unstructured data, text and binary formats and compound formats (zip) •Usage • Not limited to interactive use  automated business processes •Security • Integrations with enterprise security infrastructure •User Interaction and personalization • Identity enables more personalized search results
Enterprise Search (features) •Extended metadata • More metadata  better and more precise search results • More control over schema (for example Dynamic Fields) •Ranking • More control over ranking: personalized ranking (group) •Data extraction and derivation • Extract data using various techniques: Xpath, Xquery • Derive data: using external knowledge models: RDBMS, RDF Store, Web Services • Conditional extraction & derivation •Managing and monitoring • On-the-fly management (JMX) • Real time monitoring
Enterprise Search (features) •User Interfaces • Web search • All about selling advertisements to the mass • Generalistic & minimalistic screens; focus on adds • Enterprise search • All about finding: rich navigation; focus on quick find • Small targeted audience • Specialized and customized screens (use of ontologies, taxonomies and classifications) • Use of identity (results customized to user) and web 2.0 • Grouping • field collapsing, faceted search & clustering
Enterprise Search (Challenges) •Performance and scalability •Rich functions and features •Managebility •Flexibility •Easy maintenance •Quick issue and problem solving •Reduce total cost of ownerschip
Enterprise Search (Challenges) •Performance and scalability •Rich functions and features •Managebility •Flexibility •Easy maintenance •Quick issue and problem solving •Reduce total cost of ownerschip Commercial Search Engines?
Enterprise Search (Challenges) •Performance and scalability •Rich functions and features •Managebility •Flexibility •Easy maintenance •Quick issue and problem solving •Reduce total cost of ownerschip Apache Based (Open Source) Search & Retrieval Platform
Enterprise Search (Logical Architecture) Collection Process Publication Process Sources Actors Pull Pull Push HTTP/Get HTTP/Post API Stateless Statefull (Crawling) (Harvesting) (SOAP/ReST) (URL) (SOAP/ReST) (Java,Perl,...) (XSLT, SolrJS) (Webapp Framework) Content Inbound Request Inbound Response Outbound Syntactic Semantic Syntactic Semantic Content Validation Request Validation Redirection Enhancement Redirection Enhancement Extraction Enhancement Filtering (Suggestions) (add/remove clauses) (more like this) (metadata, editorial) Content Enrichment Request Enrichment Response Enrichment Filtering Grouping Sorting Indexing Searching & Ordering Search Engine
Enterprise Search (Collection Process) Collection Process Sources Sources • Any document format • Any type Pull Pull (Harvesting) Push (Crawling) (SOAP/ReST) • Structured and unstructured Content Inbound • Textual and binary Syntactic Semantic Content Validation • Compound • Residing Anywhere Extraction Enhancement Filtering Content Enrichment • Security Indexing Search Engine
Enterprise Search (Collection Process) Collection Process Content Inbound Sources • Pull (Crawling/Spidering) • Internet, intranet & extranet Pull Pull Push (Crawling) (Harvesting) (SOAP/ReST) • Local and remote filesystems Content Inbound Syntactic Semantic • Pull (Harvesting) Content Validation • Databases Extraction Enhancement Filtering • Content Repositories / Mgmt Systems Content Enrichment • Webservices inbound Indexing • Push Search Engine • Webservices (SOAP/REST) • Real time indexing
Enterprise Search (Collection Process) Collection Process Content Validation Sources • Syntactic validation • Based on DTD / XML-Schema Pull Pull Push (Crawling) (Harvesting) (SOAP/ReST) • Structure and limited content Content Inbound • Semantic validation Syntactic Semantic • Based on algorithms: Content Validation • Groovy, XPath, Regex, … Extraction Enhancement Filtering • Think about exception handling Content Enrichment • Placed anywhere in flow • During inbound: XML-Schema validation Indexing • After Enrichment: Validate derived metadata Search Engine
Enterprise Search (Collection Process) Collection Process Content Enrichment Sources • Extraction • Metadata Pull Pull Push (Crawling) (Harvesting) (SOAP/ReST) • Content (free text of document) Content Inbound • Enhancing Syntactic Semantic • Derive new and alter existing metadata Content Validation • Filtering Extraction Enhancement Filtering • Remove (parts of) metadata Content Enrichment • Leverage external knowledge models Indexing • Conditional enrichment Search Engine
Enterprise Search (Collection Process) Collection Process Indexing Sources • Store in search engine(s) • Content based routing Pull Pull Push (Crawling) (Harvesting) (SOAP/ReST) • Document boosting Content Inbound Syntactic Semantic Content Validation Extraction Enhancement Filtering Content Enrichment Indexing Search Engine
Enterprise Search (Publication Process) Publication Process Request Inbound • HTTP/Get Actors • URL based with parameters • Response in XML, JSON, … • HTTP/Post HTTP/Get HTTP/Post API Stateless Statefull (URL) (SOAP/ReST) (Java,Perl,...) (XSLT, SolrJS) (Webapp Framework) Request Inbound Response Outbound • XML (SOAP, REST) request Syntactic Semantic Request Validation • XML (SOAP, REST) response Redirection Enhancement Redirection Enhancement • API (Suggestions) (add/remove clauses) (more like this) (metadata, editorial) Request Enrichment Response Enrichment • Java, Perl, … Filtering Grouping Sorting • Wrappers on HTTP/Get Searching & Ordering Search Engine
Enterprise Search (Publication Process) Publication Process Request Validation • Syntactic Validation Actors • Correct Query syntax? • Semantic Validation • Correct Field Filters? HTTP/Get HTTP/Post API Stateless Statefull (URL) (SOAP/ReST) (Java,Perl,...) (XSLT, SolrJS) (Webapp Framework) Request Inbound Response Outbound • Based on algorithms: Groovy, Regex Syntactic Semantic Request Validation Redirection Enhancement Redirection Enhancement • Placed anywhere in flow (Suggestions) Request Enrichment (add/remove clauses) (more like this) Response Enrichment (metadata, editorial) • @inbound: XML-Schema validation • @enrichment: Validate derived request clauses Filtering Grouping Sorting Searching & Ordering Search Engine
Enterprise Search (Publication Process) Publication Process Request Enrichment • Redirection Actors • Spelling suggestions • Metadata suggestions HTTP/Get (URL) HTTP/Post (SOAP/ReST) API (Java,Perl,...) Stateless (XSLT, SolrJS) Statefull (Webapp Framework) • Enhancing Request Inbound Response Outbound • Add/Remove clauses Syntactic Semantic • Stemming, Synonyms, stop words Request Validation Redirection Enhancement Redirection Enhancement (Suggestions) (add/remove clauses) (more like this) (metadata, editorial) Request Enrichment Response Enrichment Filtering Grouping Sorting Searching & Ordering Search Engine
Enterprise Search (Publication Process) Publication Process Searching & Ordering • Filtering Actors • Field Search • Grouping • Add group information HTTP/Get HTTP/Post API Stateless Statefull (URL) (SOAP/ReST) (Java,Perl,...) (XSLT, SolrJS) (Webapp Framework) Request Inbound Response Outbound • Field collapsing, Faceted Search & Clustering • Sorting Syntactic Semantic Request Validation Redirection Enhancement Redirection Enhancement • Sort on Field (Suggestions) (add/remove clauses) (more like this) (metadata, editorial) Request Enrichment Response Enrichment • Ranking Filtering Grouping Sorting Searching & Ordering Search Engine
Enterprise Search (Publication Process) Publication Process Response Enrichment • Redirection Actors • Suggestions • More like this HTTP/Get (URL) HTTP/Post (SOAP/ReST) API (Java,Perl,...) Stateless (XSLT, SolrJS) Statefull (Webapp Framework) • Enhancing Request Inbound Response Outbound • Add/Remove response fields Syntactic Semantic • Schema information Request Validation • Editorial information Redirection Enhancement Redirection Enhancement (Suggestions) (add/remove clauses) (more like this) (metadata, editorial) Request Enrichment Response Enrichment Filtering Grouping Sorting Searching & Ordering Search Engine
Enterprise Search (Publication Process) Publication Process Response outbound • Stateless Actors • No security • XSLT, SolrJS HTTP/Get (URL) HTTP/Post (SOAP/ReST) API (Java,Perl,...) Stateless (XSLT, SolrJS) Statefull (Webapp Framework) • Statefull Request Inbound Response Outbound • Security Syntactic Semantic • Web2.0 Request Validation • Web Application Framework Redirection Enhancement Redirection Enhancement (Suggestions) (add/remove clauses) (more like this) (metadata, editorial) Request Enrichment Response Enrichment Filtering Grouping Sorting Searching & Ordering Search Engine
Technology Stack (Collection Process) •Use ESB for the flow: Apache ServiceMix with Camel • Leverage standard ESB components (Transformers, Validation, Splitter, Filter, Routers, Scripting) • Leverage standard ESB transports (WS, SMTP, JMS, JCR, JDBC, FILE) • Custom: Crawler Apache Nutch • Leverage only crawl framework • Extend NutchIndexWriter; asynchronously pushing crawled documents back into ESB flow (reply-to) •ESB Makes distributed flow possibleContent based routing •Hot deploy Easy maintenance •Reusing services across collection processes •Search Engine independent
Collection Process Flow Content Inbound 1 2 D D D N Document Push Inbound Syntactic Validation Splitter Messages Documents (Message Endpoint) (Channel Purger) Message Channel Content Validation Content Enrichment Content Indexer Semantic Validation Channel Channel Transformer HTTP Transport (Channel Purger) Content Filter (Message Translator) (Channel Adapter) ? Content Enricher D Invalid Message Enricher SOLR Document Message ! Lucene/Solr INDEX Invalid Message Channel Lucene/SOLR (SOLRJ)
Technology Stack (Publication Process) •Use flow from Apache Lucene/Solr • Leverage standard Solr components (synonyms, stopwords, stemming, MLT, spelling, faceted search, …) • Custom components: using Solr’s extendability framework • Security: authority field in schema with Apache Shiro integration • Field filters (zipcode,…) •User interfaces • Stateless: SolrJs, XSLTResponseWriter & VelocityResponseWriter • Statefull: Apache Wicket with Spring
Enterprise Search (Logical Architecture) Collection Process Publication Process Sources Actors Pull Pull Push HTTP/Get HTTP/Post API Stateless Statefull (Crawling) (Harvesting) (SOAP/ReST) (URL) (SOAP/ReST) (Java,Perl,...) (XSLT, SolrJS) (Webapp Framework) Content Inbound Request Inbound Response Outbound Syntactic Semantic Syntactic Semantic Content Validation Request Validation Redirection Enhancement Redirection Enhancement Extraction Enhancement Filtering (Suggestions) (add/remove clauses) (more like this) (metadata, editorial) Content Enrichment Request Enrichment Response Enrichment Filtering Grouping Sorting Indexing Searching & Ordering Search Engine
Enterprise Search (Logical Architecture) Collection Process Publication Process Sources Actors ServiceMix/Camel Apache Wicket Nutch SolrJS/XSLT Pull Pull Push HTTP/Get HTTP/Post API Stateless Statefull (Crawling) (Harvesting) (SOAP/ReST) (URL) (SOAP/ReST) (Java,Perl,...) (XSLT, SolrJS) (Webapp Framework) Content Inbound Request Inbound Response Outbound Syntactic Semantic Syntactic Semantic Content Validation Request Validation Redirection Enhancement Redirection Enhancement Extraction Enhancement Filtering (Suggestions) (add/remove clauses) (more like this) (metadata, editorial) Content Enrichment Request Enrichment Response Enrichment Filtering Grouping Sorting Indexing Searching & Ordering Search Engine Lucene/SOLR
Enterprise Search (Logical Architecture) Collection Process Publication Process Sources Actors Pull Pull Push HTTP/Get HTTP/Post API Stateless Statefull (Crawling) (Harvesting) (SOAP/ReST) (URL) (SOAP/ReST) (Java,Perl,...) (XSLT, SolrJS) (Webapp Framework) Content Inbound Request Inbound Response Outbound Syntactic Semantic Syntactic Semantic Content Validation LuminisRequest Validation Enricher Framework Redirection Enhancement Redirection Enhancement Extraction Enhancement Filtering (Suggestions) (add/remove clauses) (more like this) (metadata, editorial) Content Enrichment Request Enrichment Response Enrichment Filtering Grouping Sorting Indexing Searching & Ordering Search Engine
Luminis Enricher Framework •Custom Enricher Framework • Existing ESB & SOLR enricher capabilities not sufficient. • Enriching = one or more actions (extraction, enhancing & filtering) performed on documents with fields • Same enricher to be used for: • Collection process: • Documents  enriching, filtering & splitting • Publication process: • Search requests’first-components’ searchcomponent • Search response’last-components’ searchcomponent
Luminis Enricher Framework Content Inbound 1 2 D D D N Document Push Inbound Syntactic Validation Splitter Messages Documents (Message Endpoint) (Channel Purger) Message •Custom Enricher Framework • Existing ESB & SOLR enricher capabilities not sufficient. Channel Content Validation Content Enrichment Content Indexer • Enriching = one or more actions (extraction, enhancing & filtering) performed on documents with fields Semantic Validation Channel Channel SOLR Indexer (Channel Purger) Content Filter (Channel Adapter) • Same enricher to be used for: ? Content Enricher • Collection process: D Invalid Message Enricher SOLR Document Message • Documents  enriching, filtering & splitting ! • Publication process: Invalid Message Lucene/Solr INDEX • Search requests’first-components’ searchcomponent Channel Lucene/SOLR (SOLRJ) • Search response’last-components’ searchcomponent
Luminis Enricher Framework Content Inbound 1 2 D D D N Document Push Inbound Syntactic Validation Splitter Messages Documents (Message Endpoint) (Channel Purger) Message •Custom Enricher Framework • Existing ESB & SOLR enricher capabilities not sufficient. Channel <<XSLT>> XML2HTML Content Validation Content Enrichment Content Indexer • Enriching = one or more actions (extraction, enhancing & <<SOLRQueryRequest>> <<(X)HTML>> Query Resultaat <<SearchHandler>> filtering) performed on documents with fields <<XML>> <<QueryResponseWriter>> RequestHandler Response XSLTResponseWriter Semantic Validation Channel Channel SOLR Indexer "first-components" (Channel Purger) "components" "last-components" Content Filter (Channel Adapter) • Same enricher to be used for: ? Content Enricher • Collection process: D Invalid Message Enricher SOLR Document Message <<SearchComponent>> • Documents  enriching, filtering & splitting <<SearchComponent>> <<SearchComponent>> <<SearchComponent>> <<SearchComponent>> <<SearchComponent>> query facet mlt highlight stats debug ! • Publication process: Invalid Message Lucene/Solr INDEX • Search requests’first-components’ searchcomponent Channel Lucene/SOLR (SOLRJ) • Search response’last-components’ searchcomponent
Luminis Enricher Framework (architecture) •Pipe-and-filter architecture • Documents flow through series of actions • Output from one action is input to another action • Fields from input document can be used in action’s clauses: values in expressions filled by replacing velocity type patterns with field values •Conditional flows supported •Reuse of flows & Subflows supported
Luminis Enricher Framework (architecture) •Pipe-and-filter architecture • Documents flow through series of actions • Output from one action is input to another action • Fields from input document can be used in action’s clauses: values in expressions filled by replacing velocity type patterns with field values •Conditional flows supported •Reuse of flows & Subflows supported Action Document (select C where ${B}) [[A1],[B],[C1]] YES Document Action Document [[A1,A2],[B]] [[A1],[B]] If [B=3] (remove A2) NO Action Document (select C where ${A}) [[A1],[B],[C2]]
Luminis Enricher Framework (Configuration) •Enricher flow and expression configuration via XML based DSL • Conditional: if-then-else & switch-case-else (with regex support) • Actions: Add & remove fields and field values using expressions • Expression handlers currently supported: • Field • Function (execute methods via Java Reflection) • HttpClient (retrieve content by URL described by field values) • Xslt, Xpath, Xquery (external XML databases) • JDBC • SparQL (OpenRDF) • Apache Lucene/Solr • Apache Tika (Meta and Text extraction)
Luminis Enricher Framework (Examples) <enricher name="Field" > <field name="a">AA1</field> <field name="b">BB1</field> <field name="b">BB2</field> <multivalue-field name="c">CC1</multivalue-field> <multivalue-field name="c">CC2</multivalue-field> <if test="field::c" pattern="CC2"> <then> <field name="e">EE1</field> </then> </if> <if test="field::a"> <then> <field name="f">FF1</field> </then> </if> <rename-field name="b">d</rename-field> <remove-field name="a"/> </enricher>
Luminis Enricher Framework (Examples) <enricher name="XPath” xmlns:str="http://exslt.org/strings" <enricher name="Field" > xmlns:fn="http://www.w3.org/2005/xpath-functions" <field name="a">AA1</field> xmlns:html="http://www.w3.org/1999/xhtml"> <field name="b">BB1</field> field name="Description" expression-type="xpath"> //html:meta[@name='DC.description']/@content <field name="b">BB2</field> </field> <multivalue-field name="c">CC1</multivalue-field> <multivalue-field name="Type" expression-type="xpath"> <multivalue-field name="c">CC2</multivalue-field> //html:meta[@name='DC.type' and <if test="field::c" pattern="CC2"> (@scheme='OVERHEIDbm.bekendmakingtypeGemeente' or <then> @scheme='OVERHEIDbm.bekendmakingtypeProvincie' or <field name="e">EE1</field> @scheme='OVERHEIDbm.bekendmakingtypeWaterschap') </then> ]/@content </if> </multivalue-field> <field name="publisher" expression-type="xpath"> <if test="field::a"> fn:string-join(('Blow, ', 'blow, ', 'thou ', 'winter ', 'wind!'), '') <then> </field> <field name="f">FF1</field> <field name="publisher" expression-type="xpath"> </then> fn:concat(//html:meta[@name='OVERHEID.organisationType']/@content, </if> //html:meta[@name='DC.creator']/@content) </field> <rename-field name="b">d</rename-field> </enricher> <remove-field name="a"/> </enricher>
Luminis Enricher Framework (Examples) <enricher name="SPARQL"> <field name="place">http://www.my.com/#channels</field> <enricher name="XPath” <field expression-type="sparql" repository="TESTRDF"> xmlns:str="http://exslt.org/strings" <![CDATA[ <enricher name="Field" > xmlns:fn="http://www.w3.org/2005/xpath-functions" PREFIX skos: <http://www.w3.org/2004/02/skos/core#> <field name="a">AA1</field> xmlns:html="http://www.w3.org/1999/xhtml"> SELECT ?definition <field name="b">BB1</field> field name="Description" expression-type="xpath"> WHERE { //html:meta[@name='DC.description']/@content <field name="b">BB2</field> ?${place} skos:definition ?definition. </field> <multivalue-field name="c">CC1</multivalue-field> } <multivalue-field name="Type" expression-type="xpath"> ]]> <multivalue-field name="c">CC2</multivalue-field> //html:meta[@name='DC.type' and </field> <if test="field::c" pattern="CC2"> (@scheme='OVERHEIDbm.bekendmakingtypeGemeente' or </enricher> <then> @scheme='OVERHEIDbm.bekendmakingtypeProvincie' or <field name="e">EE1</field> @scheme='OVERHEIDbm.bekendmakingtypeWaterschap') </then> ]/@content </if> </multivalue-field> <field name="publisher" expression-type="xpath"> <if test="field::a"> fn:string-join(('Blow, ', 'blow, ', 'thou ', 'winter ', 'wind!'), '') <then> </field> <field name="f">FF1</field> <field name="publisher" expression-type="xpath"> </then> fn:concat(//html:meta[@name='OVERHEID.organisationType']/@content, </if> //html:meta[@name='DC.creator']/@content) </field> <rename-field name="b">d</rename-field> </enricher> <remove-field name="a"/> </enricher>
Luminis Enricher Framework (Examples) <enricher name="SPARQL"> <field name="place">http://www.my.com/#channels</field> <enricher name="XPath” <field expression-type="sparql" repository="TESTRDF"> xmlns:str="http://exslt.org/strings" <![CDATA[ <enricher name="Field" > xmlns:fn="http://www.w3.org/2005/xpath-functions" PREFIX skos: <http://www.w3.org/2004/02/skos/core#> <field name="a">AA1</field> xmlns:html="http://www.w3.org/1999/xhtml"> SELECT ?definition <field name="b">BB1</field> field name="Description" expression-type="xpath"> WHERE { //html:meta[@name='DC.description']/@content <field name="b">BB2</field> ?${place} skos:definition ?definition. </field> <multivalue-field name="c">CC1</multivalue-field> } <multivalue-field name="Type" expression-type="xpath"> ]]> <multivalue-field name="c">CC2</multivalue-field> //html:meta[@name='DC.type' and </field> <if test="field::c" pattern="CC2"> (@scheme='OVERHEIDbm.bekendmakingtypeGemeente' or </enricher> <then> @scheme='OVERHEIDbm.bekendmakingtypeProvincie' or <field name="e">EE1</field> @scheme='OVERHEIDbm.bekendmakingtypeWaterschap') <enricher name=”HttpAndTika"> </then> ]/@content </if> <field name="content.url"><![CDATA[http://na.apachecon.com/c/acna2010/speakers/501]]></field> </multivalue-field> <field expression-type=”http" name="content.file">field:content.url</field> <field name="publisher" expression-type="xpath"> <if test="field::a"> <field name="auteur" source="field::content.file">xpath://H1</field> fn:string-join(('Blow, ', 'blow, ', 'thou ', 'winter ', 'wind!'), '') <then> <multivalue-field expression-type=”tika.meta” source="field::content.file”/> </field> <field name="f">FF1</field> <field name=”content" expression-type=”tika.text” source="field::content.file”/> <field name="publisher" expression-type="xpath"> </then> <switch test=”field::content.url fn:concat(//html:meta[@name='OVERHEID.organisationType']/@content, </if> <case pattern=".*.rijksweb.nl.*"><field name=”source">Rijksweb</field></case> //html:meta[@name='DC.creator']/@content) <case name="b">d</rename-field> <rename-field pattern=".*.deventer.nl.*"><field name=”source">Gemeente Deventer</field></case> </field> <case</enricher> pattern="file:.*"><field name=”source">Locale Harde Schijf</field></case> <remove-field name="a"/> <else><field name=”source">Overige</field></else> </enricher> </switch> </enricher>
Luminis Enricher Framework (Technology) •Enricher and expresion handlers are Java based OSGi services: • Hot pluggable and updatable • Flow and expression configuration changes no restart • Extendible: New expression handlers immediatly available in actions after installing OSGi bundle •Runs in Apache Felix • Collection Process: ServiceMix contains OSGi container • Publication Process: Custom OSGi loader for Lucene/Solr •Centralized & transactional provisioning (Apache Ace) ‑ Components & Configuration
Deployment Architecture <<HTTP>> <<device>> Firewall <<device>> HTTP Load Balancer <<HTTP>> <<device>> Deployment Server <<device>> <<HTTP>> Felix OSGi Master Collection Server (Apache) <<device>> <<Container>> Slave Publication Server Ace Apache Tomcat (Slave2) (Apache) ServiceMix Felix OSGi (Apache) (Apache) <<Container>> <<device>> Enricher Nutch Apache Tomcat (Luminis) Slave Publication Server (Apache) Felix OSGi (Slave1) (Apache) <<config>> SOLR::solrconfig.xml Lucene/SOLR <<HTTP/ReST>> <<PROVISIONING>> <<config>> (Apache) Lucene/SOLR Luminis:Enricher.xml <<Container>> (Apache) <<config>> Tika Apache Tomcat SOLR::schema.xml (Apache) <<HTTP/ReST>> Felix OSGi Wicket <<config>> servicenix::config.xml (Apache) (Apache) OpenRDF <<HTTP>> Lucene/SOLR Enricher (Apache) (Luminis) <<HTTP>> <<config>> <<Data Container>> Wicket SOLR::solrconfig.xml SQL <<JDBC>> (Apache) <<config>> Luminis:Enricher.xml <<JDBC>> <<Database>> <<RDFTripleStore>> Enricher <<config>> Knowledge Models Knowledge Models SOLR::schema.xml (Luminis) <<config>> SOLR::solrconfig.xml <<config>> Luminis:Enricher.xml <<config>> SOLR::schema.xml
Conclusions •Enterprise Search Solution is not Google search •Open Source paves the way; misses some ingredients • Useful ingredients: Lucene/Solr, Nutch, Tika, ServiceMix/Camel, Wicket, MySQL, OpenRDF, Felix/Ace • Missing ingredients: Enricher •Interesting developments: • Apache Chemistry (CMIS) • Apache Clerezza • Apache Nutch • Apache Connectors Framework (ManifoldCF)
Questions & (answers?) Marc Teutelink marc.teutelink@luminis.eu @mteutelink MEAP December 2010 

More Related Content

PDF
It's Just Search: Presented by Erik Hatcher, Lucidworks
Lucidworks
 
KEY
Content extraction with apache tika
Jukka Zitting
 
PDF
Indexing Text and HTML Files with Solr
Lucidworks (Archived)
 
PPT
Content Analysis with Apache Tika
Paolo Mottadelli
 
PPTX
SharePoint 2013 Search Architecture with Russ Houberg
knowledgelakemarketing
 
PPTX
Best Practices for SharePoint 2010 Search
Agnes Molnar
 
PPTX
MetaVis Webinar - 10 Things I Like in SharePoint 2013 Search
Agnes Molnar
 
PPTX
ESPC13 - 10 Things I Like in SharePoint 2013 Search
Agnes Molnar
 
It's Just Search: Presented by Erik Hatcher, Lucidworks
Lucidworks
 
Content extraction with apache tika
Jukka Zitting
 
Indexing Text and HTML Files with Solr
Lucidworks (Archived)
 
Content Analysis with Apache Tika
Paolo Mottadelli
 
SharePoint 2013 Search Architecture with Russ Houberg
knowledgelakemarketing
 
Best Practices for SharePoint 2010 Search
Agnes Molnar
 
MetaVis Webinar - 10 Things I Like in SharePoint 2013 Search
Agnes Molnar
 
ESPC13 - 10 Things I Like in SharePoint 2013 Search
Agnes Molnar
 

Similar to Open source enterprise search and retrieval platform (20)

PPTX
SPConnections - Search Administration in SharePoint 2013
Agnes Molnar
 
PPTX
SPCAdriatics - 10 Things I Like In SharePoint 2013 Search
Agnes Molnar
 
PPTX
10 Things I Like in SharePoint 2013 Search
SPC Adriatics
 
PPTX
Fishbowl Solutions WebCenter Search Webinar Presentation
Kim Negaard
 
PPTX
SPConnections Amsterdam: Beyond the Search Center - Application or Solution? ...
Agnes Molnar
 
PDF
NoSQL, Apache SOLR and Apache Hadoop
Dmitry Kan
 
PPTX
How to SEO a Terrific - and Profitable - User Experience
BrightEdge
 
PDF
Elasticsearch Introduction at BigData meetup
Eric Rodriguez (Hiring in Lex)
 
PDF
FAST Search for SharePoint
C/D/H Technology Consultants
 
PPT
SPLive Orlando - 10 Things I Like in SharePoint 2013 Search
Agnes Molnar
 
PPTX
Find Information Faster Using SharePoint 2010 Search
Perficient, Inc.
 
PPT
SPLive Orlando - Beyond the Search Center - Application or Solution?
Agnes Molnar
 
PDF
Planning SharePoint 2013 Search for IT PROs
Benjamin Athawes
 
PDF
Naukri Search Team achievements, 2009-2010
Aditya Varun Chadha
 
PPTX
Lab EPiServer Find - Advanced developer scenarios
Patrick van Kleef
 
PDF
Drupal and Apache Stanbol
Alkuvoima
 
PPSX
Lifecycle of a FAST Search Implementation
Perficient, Inc.
 
PDF
Concept Searching Portal Solutions Search Engine Face Off
martingarland
 
PPTX
SPConnections - What's new in SharePoint 2013 Search
Agnes Molnar
 
KEY
FatWire Tutorial For Site Studio Developers
Brian Huff
 
SPConnections - Search Administration in SharePoint 2013
Agnes Molnar
 
SPCAdriatics - 10 Things I Like In SharePoint 2013 Search
Agnes Molnar
 
10 Things I Like in SharePoint 2013 Search
SPC Adriatics
 
Fishbowl Solutions WebCenter Search Webinar Presentation
Kim Negaard
 
SPConnections Amsterdam: Beyond the Search Center - Application or Solution? ...
Agnes Molnar
 
NoSQL, Apache SOLR and Apache Hadoop
Dmitry Kan
 
How to SEO a Terrific - and Profitable - User Experience
BrightEdge
 
Elasticsearch Introduction at BigData meetup
Eric Rodriguez (Hiring in Lex)
 
FAST Search for SharePoint
C/D/H Technology Consultants
 
SPLive Orlando - 10 Things I Like in SharePoint 2013 Search
Agnes Molnar
 
Find Information Faster Using SharePoint 2010 Search
Perficient, Inc.
 
SPLive Orlando - Beyond the Search Center - Application or Solution?
Agnes Molnar
 
Planning SharePoint 2013 Search for IT PROs
Benjamin Athawes
 
Naukri Search Team achievements, 2009-2010
Aditya Varun Chadha
 
Lab EPiServer Find - Advanced developer scenarios
Patrick van Kleef
 
Drupal and Apache Stanbol
Alkuvoima
 
Lifecycle of a FAST Search Implementation
Perficient, Inc.
 
Concept Searching Portal Solutions Search Engine Face Off
martingarland
 
SPConnections - What's new in SharePoint 2013 Search
Agnes Molnar
 
FatWire Tutorial For Site Studio Developers
Brian Huff
 
Ad

Open source enterprise search and retrieval platform

  • 1. Open Source Search & Retrieval Platform Enterprise Search EAI Marc Teutelink Semantic Web Datum 21 augustus 2010
  • 2. How Apache open source software is used during the implementation of an Enterprise Search and Retrieval Platform (Lucene/SOLR, Nutch, Tika, ServiceMix/Camel, Felix/Ace)
  • 3. Marc Teutelink marc.teutelink@luminis.eu @mteutelink •Software architect at Luminis •15+ years experience in software development; specialized in Enterprise Search, Enterprise Application Integration and Semantic Web technology •Currently writing “Enterprise Search in Action” for Manning (Mid-2011)
  • 4. Agenda •Enterprise Search • What is Enterprise Search: Functions and features • Challenges • Logical Architecture •Enterprise Search Solution • Technology Stack • Collection Process • Publication Process • Enricher framework • Deployment •Conclusion
  • 5. What is Enterprise Search? “Enterprise Search offers a solution for searching, finding and presenting enterprise related information in the larger sense of the word” Enterprise search is all about searching through documents from any type and format from any sources located anywhere with the upmost flexibility • Web search: limited to public documents on the web • Desktop search: limited to private documents on the local machine • Enterprise search: no limitations on document type and location
  • 6. Enterprise Search (features) •Information Sources and Types • Wide range of sources: local and remote filesystems, content repositories, e-mail, databases, internet, intranet and extranet • Type not limited: any type ranging from structured to unstructured data, text and binary formats and compound formats (zip) •Usage • Not limited to interactive use  automated business processes •Security • Integrations with enterprise security infrastructure •User Interaction and personalization • Identity enables more personalized search results
  • 7. Enterprise Search (features) •Extended metadata • More metadata  better and more precise search results • More control over schema (for example Dynamic Fields) •Ranking • More control over ranking: personalized ranking (group) •Data extraction and derivation • Extract data using various techniques: Xpath, Xquery • Derive data: using external knowledge models: RDBMS, RDF Store, Web Services • Conditional extraction & derivation •Managing and monitoring • On-the-fly management (JMX) • Real time monitoring
  • 8. Enterprise Search (features) •User Interfaces • Web search • All about selling advertisements to the mass • Generalistic & minimalistic screens; focus on adds • Enterprise search • All about finding: rich navigation; focus on quick find • Small targeted audience • Specialized and customized screens (use of ontologies, taxonomies and classifications) • Use of identity (results customized to user) and web 2.0 • Grouping • field collapsing, faceted search & clustering
  • 9. Enterprise Search (Challenges) •Performance and scalability •Rich functions and features •Managebility •Flexibility •Easy maintenance •Quick issue and problem solving •Reduce total cost of ownerschip
  • 10. Enterprise Search (Challenges) •Performance and scalability •Rich functions and features •Managebility •Flexibility •Easy maintenance •Quick issue and problem solving •Reduce total cost of ownerschip Commercial Search Engines?
  • 11. Enterprise Search (Challenges) •Performance and scalability •Rich functions and features •Managebility •Flexibility •Easy maintenance •Quick issue and problem solving •Reduce total cost of ownerschip Apache Based (Open Source) Search & Retrieval Platform
  • 12. Enterprise Search (Logical Architecture) Collection Process Publication Process Sources Actors Pull Pull Push HTTP/Get HTTP/Post API Stateless Statefull (Crawling) (Harvesting) (SOAP/ReST) (URL) (SOAP/ReST) (Java,Perl,...) (XSLT, SolrJS) (Webapp Framework) Content Inbound Request Inbound Response Outbound Syntactic Semantic Syntactic Semantic Content Validation Request Validation Redirection Enhancement Redirection Enhancement Extraction Enhancement Filtering (Suggestions) (add/remove clauses) (more like this) (metadata, editorial) Content Enrichment Request Enrichment Response Enrichment Filtering Grouping Sorting Indexing Searching & Ordering Search Engine
  • 13. Enterprise Search (Collection Process) Collection Process Sources Sources • Any document format • Any type Pull Pull (Harvesting) Push (Crawling) (SOAP/ReST) • Structured and unstructured Content Inbound • Textual and binary Syntactic Semantic Content Validation • Compound • Residing Anywhere Extraction Enhancement Filtering Content Enrichment • Security Indexing Search Engine
  • 14. Enterprise Search (Collection Process) Collection Process Content Inbound Sources • Pull (Crawling/Spidering) • Internet, intranet & extranet Pull Pull Push (Crawling) (Harvesting) (SOAP/ReST) • Local and remote filesystems Content Inbound Syntactic Semantic • Pull (Harvesting) Content Validation • Databases Extraction Enhancement Filtering • Content Repositories / Mgmt Systems Content Enrichment • Webservices inbound Indexing • Push Search Engine • Webservices (SOAP/REST) • Real time indexing
  • 15. Enterprise Search (Collection Process) Collection Process Content Validation Sources • Syntactic validation • Based on DTD / XML-Schema Pull Pull Push (Crawling) (Harvesting) (SOAP/ReST) • Structure and limited content Content Inbound • Semantic validation Syntactic Semantic • Based on algorithms: Content Validation • Groovy, XPath, Regex, … Extraction Enhancement Filtering • Think about exception handling Content Enrichment • Placed anywhere in flow • During inbound: XML-Schema validation Indexing • After Enrichment: Validate derived metadata Search Engine
  • 16. Enterprise Search (Collection Process) Collection Process Content Enrichment Sources • Extraction • Metadata Pull Pull Push (Crawling) (Harvesting) (SOAP/ReST) • Content (free text of document) Content Inbound • Enhancing Syntactic Semantic • Derive new and alter existing metadata Content Validation • Filtering Extraction Enhancement Filtering • Remove (parts of) metadata Content Enrichment • Leverage external knowledge models Indexing • Conditional enrichment Search Engine
  • 17. Enterprise Search (Collection Process) Collection Process Indexing Sources • Store in search engine(s) • Content based routing Pull Pull Push (Crawling) (Harvesting) (SOAP/ReST) • Document boosting Content Inbound Syntactic Semantic Content Validation Extraction Enhancement Filtering Content Enrichment Indexing Search Engine
  • 18. Enterprise Search (Publication Process) Publication Process Request Inbound • HTTP/Get Actors • URL based with parameters • Response in XML, JSON, … • HTTP/Post HTTP/Get HTTP/Post API Stateless Statefull (URL) (SOAP/ReST) (Java,Perl,...) (XSLT, SolrJS) (Webapp Framework) Request Inbound Response Outbound • XML (SOAP, REST) request Syntactic Semantic Request Validation • XML (SOAP, REST) response Redirection Enhancement Redirection Enhancement • API (Suggestions) (add/remove clauses) (more like this) (metadata, editorial) Request Enrichment Response Enrichment • Java, Perl, … Filtering Grouping Sorting • Wrappers on HTTP/Get Searching & Ordering Search Engine
  • 19. Enterprise Search (Publication Process) Publication Process Request Validation • Syntactic Validation Actors • Correct Query syntax? • Semantic Validation • Correct Field Filters? HTTP/Get HTTP/Post API Stateless Statefull (URL) (SOAP/ReST) (Java,Perl,...) (XSLT, SolrJS) (Webapp Framework) Request Inbound Response Outbound • Based on algorithms: Groovy, Regex Syntactic Semantic Request Validation Redirection Enhancement Redirection Enhancement • Placed anywhere in flow (Suggestions) Request Enrichment (add/remove clauses) (more like this) Response Enrichment (metadata, editorial) • @inbound: XML-Schema validation • @enrichment: Validate derived request clauses Filtering Grouping Sorting Searching & Ordering Search Engine
  • 20. Enterprise Search (Publication Process) Publication Process Request Enrichment • Redirection Actors • Spelling suggestions • Metadata suggestions HTTP/Get (URL) HTTP/Post (SOAP/ReST) API (Java,Perl,...) Stateless (XSLT, SolrJS) Statefull (Webapp Framework) • Enhancing Request Inbound Response Outbound • Add/Remove clauses Syntactic Semantic • Stemming, Synonyms, stop words Request Validation Redirection Enhancement Redirection Enhancement (Suggestions) (add/remove clauses) (more like this) (metadata, editorial) Request Enrichment Response Enrichment Filtering Grouping Sorting Searching & Ordering Search Engine
  • 21. Enterprise Search (Publication Process) Publication Process Searching & Ordering • Filtering Actors • Field Search • Grouping • Add group information HTTP/Get HTTP/Post API Stateless Statefull (URL) (SOAP/ReST) (Java,Perl,...) (XSLT, SolrJS) (Webapp Framework) Request Inbound Response Outbound • Field collapsing, Faceted Search & Clustering • Sorting Syntactic Semantic Request Validation Redirection Enhancement Redirection Enhancement • Sort on Field (Suggestions) (add/remove clauses) (more like this) (metadata, editorial) Request Enrichment Response Enrichment • Ranking Filtering Grouping Sorting Searching & Ordering Search Engine
  • 22. Enterprise Search (Publication Process) Publication Process Response Enrichment • Redirection Actors • Suggestions • More like this HTTP/Get (URL) HTTP/Post (SOAP/ReST) API (Java,Perl,...) Stateless (XSLT, SolrJS) Statefull (Webapp Framework) • Enhancing Request Inbound Response Outbound • Add/Remove response fields Syntactic Semantic • Schema information Request Validation • Editorial information Redirection Enhancement Redirection Enhancement (Suggestions) (add/remove clauses) (more like this) (metadata, editorial) Request Enrichment Response Enrichment Filtering Grouping Sorting Searching & Ordering Search Engine
  • 23. Enterprise Search (Publication Process) Publication Process Response outbound • Stateless Actors • No security • XSLT, SolrJS HTTP/Get (URL) HTTP/Post (SOAP/ReST) API (Java,Perl,...) Stateless (XSLT, SolrJS) Statefull (Webapp Framework) • Statefull Request Inbound Response Outbound • Security Syntactic Semantic • Web2.0 Request Validation • Web Application Framework Redirection Enhancement Redirection Enhancement (Suggestions) (add/remove clauses) (more like this) (metadata, editorial) Request Enrichment Response Enrichment Filtering Grouping Sorting Searching & Ordering Search Engine
  • 24. Technology Stack (Collection Process) •Use ESB for the flow: Apache ServiceMix with Camel • Leverage standard ESB components (Transformers, Validation, Splitter, Filter, Routers, Scripting) • Leverage standard ESB transports (WS, SMTP, JMS, JCR, JDBC, FILE) • Custom: Crawler Apache Nutch • Leverage only crawl framework • Extend NutchIndexWriter; asynchronously pushing crawled documents back into ESB flow (reply-to) •ESB Makes distributed flow possibleContent based routing •Hot deploy Easy maintenance •Reusing services across collection processes •Search Engine independent
  • 25. Collection Process Flow Content Inbound 1 2 D D D N Document Push Inbound Syntactic Validation Splitter Messages Documents (Message Endpoint) (Channel Purger) Message Channel Content Validation Content Enrichment Content Indexer Semantic Validation Channel Channel Transformer HTTP Transport (Channel Purger) Content Filter (Message Translator) (Channel Adapter) ? Content Enricher D Invalid Message Enricher SOLR Document Message ! Lucene/Solr INDEX Invalid Message Channel Lucene/SOLR (SOLRJ)
  • 26. Technology Stack (Publication Process) •Use flow from Apache Lucene/Solr • Leverage standard Solr components (synonyms, stopwords, stemming, MLT, spelling, faceted search, …) • Custom components: using Solr’s extendability framework • Security: authority field in schema with Apache Shiro integration • Field filters (zipcode,…) •User interfaces • Stateless: SolrJs, XSLTResponseWriter & VelocityResponseWriter • Statefull: Apache Wicket with Spring
  • 27. Enterprise Search (Logical Architecture) Collection Process Publication Process Sources Actors Pull Pull Push HTTP/Get HTTP/Post API Stateless Statefull (Crawling) (Harvesting) (SOAP/ReST) (URL) (SOAP/ReST) (Java,Perl,...) (XSLT, SolrJS) (Webapp Framework) Content Inbound Request Inbound Response Outbound Syntactic Semantic Syntactic Semantic Content Validation Request Validation Redirection Enhancement Redirection Enhancement Extraction Enhancement Filtering (Suggestions) (add/remove clauses) (more like this) (metadata, editorial) Content Enrichment Request Enrichment Response Enrichment Filtering Grouping Sorting Indexing Searching & Ordering Search Engine
  • 28. Enterprise Search (Logical Architecture) Collection Process Publication Process Sources Actors ServiceMix/Camel Apache Wicket Nutch SolrJS/XSLT Pull Pull Push HTTP/Get HTTP/Post API Stateless Statefull (Crawling) (Harvesting) (SOAP/ReST) (URL) (SOAP/ReST) (Java,Perl,...) (XSLT, SolrJS) (Webapp Framework) Content Inbound Request Inbound Response Outbound Syntactic Semantic Syntactic Semantic Content Validation Request Validation Redirection Enhancement Redirection Enhancement Extraction Enhancement Filtering (Suggestions) (add/remove clauses) (more like this) (metadata, editorial) Content Enrichment Request Enrichment Response Enrichment Filtering Grouping Sorting Indexing Searching & Ordering Search Engine Lucene/SOLR
  • 29. Enterprise Search (Logical Architecture) Collection Process Publication Process Sources Actors Pull Pull Push HTTP/Get HTTP/Post API Stateless Statefull (Crawling) (Harvesting) (SOAP/ReST) (URL) (SOAP/ReST) (Java,Perl,...) (XSLT, SolrJS) (Webapp Framework) Content Inbound Request Inbound Response Outbound Syntactic Semantic Syntactic Semantic Content Validation LuminisRequest Validation Enricher Framework Redirection Enhancement Redirection Enhancement Extraction Enhancement Filtering (Suggestions) (add/remove clauses) (more like this) (metadata, editorial) Content Enrichment Request Enrichment Response Enrichment Filtering Grouping Sorting Indexing Searching & Ordering Search Engine
  • 30. Luminis Enricher Framework •Custom Enricher Framework • Existing ESB & SOLR enricher capabilities not sufficient. • Enriching = one or more actions (extraction, enhancing & filtering) performed on documents with fields • Same enricher to be used for: • Collection process: • Documents  enriching, filtering & splitting • Publication process: • Search requests’first-components’ searchcomponent • Search response’last-components’ searchcomponent
  • 31. Luminis Enricher Framework Content Inbound 1 2 D D D N Document Push Inbound Syntactic Validation Splitter Messages Documents (Message Endpoint) (Channel Purger) Message •Custom Enricher Framework • Existing ESB & SOLR enricher capabilities not sufficient. Channel Content Validation Content Enrichment Content Indexer • Enriching = one or more actions (extraction, enhancing & filtering) performed on documents with fields Semantic Validation Channel Channel SOLR Indexer (Channel Purger) Content Filter (Channel Adapter) • Same enricher to be used for: ? Content Enricher • Collection process: D Invalid Message Enricher SOLR Document Message • Documents  enriching, filtering & splitting ! • Publication process: Invalid Message Lucene/Solr INDEX • Search requests’first-components’ searchcomponent Channel Lucene/SOLR (SOLRJ) • Search response’last-components’ searchcomponent
  • 32. Luminis Enricher Framework Content Inbound 1 2 D D D N Document Push Inbound Syntactic Validation Splitter Messages Documents (Message Endpoint) (Channel Purger) Message •Custom Enricher Framework • Existing ESB & SOLR enricher capabilities not sufficient. Channel <<XSLT>> XML2HTML Content Validation Content Enrichment Content Indexer • Enriching = one or more actions (extraction, enhancing & <<SOLRQueryRequest>> <<(X)HTML>> Query Resultaat <<SearchHandler>> filtering) performed on documents with fields <<XML>> <<QueryResponseWriter>> RequestHandler Response XSLTResponseWriter Semantic Validation Channel Channel SOLR Indexer "first-components" (Channel Purger) "components" "last-components" Content Filter (Channel Adapter) • Same enricher to be used for: ? Content Enricher • Collection process: D Invalid Message Enricher SOLR Document Message <<SearchComponent>> • Documents  enriching, filtering & splitting <<SearchComponent>> <<SearchComponent>> <<SearchComponent>> <<SearchComponent>> <<SearchComponent>> query facet mlt highlight stats debug ! • Publication process: Invalid Message Lucene/Solr INDEX • Search requests’first-components’ searchcomponent Channel Lucene/SOLR (SOLRJ) • Search response’last-components’ searchcomponent
  • 33. Luminis Enricher Framework (architecture) •Pipe-and-filter architecture • Documents flow through series of actions • Output from one action is input to another action • Fields from input document can be used in action’s clauses: values in expressions filled by replacing velocity type patterns with field values •Conditional flows supported •Reuse of flows & Subflows supported
  • 34. Luminis Enricher Framework (architecture) •Pipe-and-filter architecture • Documents flow through series of actions • Output from one action is input to another action • Fields from input document can be used in action’s clauses: values in expressions filled by replacing velocity type patterns with field values •Conditional flows supported •Reuse of flows & Subflows supported Action Document (select C where ${B}) [[A1],[B],[C1]] YES Document Action Document [[A1,A2],[B]] [[A1],[B]] If [B=3] (remove A2) NO Action Document (select C where ${A}) [[A1],[B],[C2]]
  • 35. Luminis Enricher Framework (Configuration) •Enricher flow and expression configuration via XML based DSL • Conditional: if-then-else & switch-case-else (with regex support) • Actions: Add & remove fields and field values using expressions • Expression handlers currently supported: • Field • Function (execute methods via Java Reflection) • HttpClient (retrieve content by URL described by field values) • Xslt, Xpath, Xquery (external XML databases) • JDBC • SparQL (OpenRDF) • Apache Lucene/Solr • Apache Tika (Meta and Text extraction)
  • 36. Luminis Enricher Framework (Examples) <enricher name="Field" > <field name="a">AA1</field> <field name="b">BB1</field> <field name="b">BB2</field> <multivalue-field name="c">CC1</multivalue-field> <multivalue-field name="c">CC2</multivalue-field> <if test="field::c" pattern="CC2"> <then> <field name="e">EE1</field> </then> </if> <if test="field::a"> <then> <field name="f">FF1</field> </then> </if> <rename-field name="b">d</rename-field> <remove-field name="a"/> </enricher>
  • 37. Luminis Enricher Framework (Examples) <enricher name="XPath” xmlns:str="http://exslt.org/strings" <enricher name="Field" > xmlns:fn="http://www.w3.org/2005/xpath-functions" <field name="a">AA1</field> xmlns:html="http://www.w3.org/1999/xhtml"> <field name="b">BB1</field> field name="Description" expression-type="xpath"> //html:meta[@name='DC.description']/@content <field name="b">BB2</field> </field> <multivalue-field name="c">CC1</multivalue-field> <multivalue-field name="Type" expression-type="xpath"> <multivalue-field name="c">CC2</multivalue-field> //html:meta[@name='DC.type' and <if test="field::c" pattern="CC2"> (@scheme='OVERHEIDbm.bekendmakingtypeGemeente' or <then> @scheme='OVERHEIDbm.bekendmakingtypeProvincie' or <field name="e">EE1</field> @scheme='OVERHEIDbm.bekendmakingtypeWaterschap') </then> ]/@content </if> </multivalue-field> <field name="publisher" expression-type="xpath"> <if test="field::a"> fn:string-join(('Blow, ', 'blow, ', 'thou ', 'winter ', 'wind!'), '') <then> </field> <field name="f">FF1</field> <field name="publisher" expression-type="xpath"> </then> fn:concat(//html:meta[@name='OVERHEID.organisationType']/@content, </if> //html:meta[@name='DC.creator']/@content) </field> <rename-field name="b">d</rename-field> </enricher> <remove-field name="a"/> </enricher>
  • 38. Luminis Enricher Framework (Examples) <enricher name="SPARQL"> <field name="place">http://www.my.com/#channels</field> <enricher name="XPath” <field expression-type="sparql" repository="TESTRDF"> xmlns:str="http://exslt.org/strings" <![CDATA[ <enricher name="Field" > xmlns:fn="http://www.w3.org/2005/xpath-functions" PREFIX skos: <http://www.w3.org/2004/02/skos/core#> <field name="a">AA1</field> xmlns:html="http://www.w3.org/1999/xhtml"> SELECT ?definition <field name="b">BB1</field> field name="Description" expression-type="xpath"> WHERE { //html:meta[@name='DC.description']/@content <field name="b">BB2</field> ?${place} skos:definition ?definition. </field> <multivalue-field name="c">CC1</multivalue-field> } <multivalue-field name="Type" expression-type="xpath"> ]]> <multivalue-field name="c">CC2</multivalue-field> //html:meta[@name='DC.type' and </field> <if test="field::c" pattern="CC2"> (@scheme='OVERHEIDbm.bekendmakingtypeGemeente' or </enricher> <then> @scheme='OVERHEIDbm.bekendmakingtypeProvincie' or <field name="e">EE1</field> @scheme='OVERHEIDbm.bekendmakingtypeWaterschap') </then> ]/@content </if> </multivalue-field> <field name="publisher" expression-type="xpath"> <if test="field::a"> fn:string-join(('Blow, ', 'blow, ', 'thou ', 'winter ', 'wind!'), '') <then> </field> <field name="f">FF1</field> <field name="publisher" expression-type="xpath"> </then> fn:concat(//html:meta[@name='OVERHEID.organisationType']/@content, </if> //html:meta[@name='DC.creator']/@content) </field> <rename-field name="b">d</rename-field> </enricher> <remove-field name="a"/> </enricher>
  • 39. Luminis Enricher Framework (Examples) <enricher name="SPARQL"> <field name="place">http://www.my.com/#channels</field> <enricher name="XPath” <field expression-type="sparql" repository="TESTRDF"> xmlns:str="http://exslt.org/strings" <![CDATA[ <enricher name="Field" > xmlns:fn="http://www.w3.org/2005/xpath-functions" PREFIX skos: <http://www.w3.org/2004/02/skos/core#> <field name="a">AA1</field> xmlns:html="http://www.w3.org/1999/xhtml"> SELECT ?definition <field name="b">BB1</field> field name="Description" expression-type="xpath"> WHERE { //html:meta[@name='DC.description']/@content <field name="b">BB2</field> ?${place} skos:definition ?definition. </field> <multivalue-field name="c">CC1</multivalue-field> } <multivalue-field name="Type" expression-type="xpath"> ]]> <multivalue-field name="c">CC2</multivalue-field> //html:meta[@name='DC.type' and </field> <if test="field::c" pattern="CC2"> (@scheme='OVERHEIDbm.bekendmakingtypeGemeente' or </enricher> <then> @scheme='OVERHEIDbm.bekendmakingtypeProvincie' or <field name="e">EE1</field> @scheme='OVERHEIDbm.bekendmakingtypeWaterschap') <enricher name=”HttpAndTika"> </then> ]/@content </if> <field name="content.url"><![CDATA[http://na.apachecon.com/c/acna2010/speakers/501]]></field> </multivalue-field> <field expression-type=”http" name="content.file">field:content.url</field> <field name="publisher" expression-type="xpath"> <if test="field::a"> <field name="auteur" source="field::content.file">xpath://H1</field> fn:string-join(('Blow, ', 'blow, ', 'thou ', 'winter ', 'wind!'), '') <then> <multivalue-field expression-type=”tika.meta” source="field::content.file”/> </field> <field name="f">FF1</field> <field name=”content" expression-type=”tika.text” source="field::content.file”/> <field name="publisher" expression-type="xpath"> </then> <switch test=”field::content.url fn:concat(//html:meta[@name='OVERHEID.organisationType']/@content, </if> <case pattern=".*.rijksweb.nl.*"><field name=”source">Rijksweb</field></case> //html:meta[@name='DC.creator']/@content) <case name="b">d</rename-field> <rename-field pattern=".*.deventer.nl.*"><field name=”source">Gemeente Deventer</field></case> </field> <case</enricher> pattern="file:.*"><field name=”source">Locale Harde Schijf</field></case> <remove-field name="a"/> <else><field name=”source">Overige</field></else> </enricher> </switch> </enricher>
  • 40. Luminis Enricher Framework (Technology) •Enricher and expresion handlers are Java based OSGi services: • Hot pluggable and updatable • Flow and expression configuration changes no restart • Extendible: New expression handlers immediatly available in actions after installing OSGi bundle •Runs in Apache Felix • Collection Process: ServiceMix contains OSGi container • Publication Process: Custom OSGi loader for Lucene/Solr •Centralized & transactional provisioning (Apache Ace) ‑ Components & Configuration
  • 41. Deployment Architecture <<HTTP>> <<device>> Firewall <<device>> HTTP Load Balancer <<HTTP>> <<device>> Deployment Server <<device>> <<HTTP>> Felix OSGi Master Collection Server (Apache) <<device>> <<Container>> Slave Publication Server Ace Apache Tomcat (Slave2) (Apache) ServiceMix Felix OSGi (Apache) (Apache) <<Container>> <<device>> Enricher Nutch Apache Tomcat (Luminis) Slave Publication Server (Apache) Felix OSGi (Slave1) (Apache) <<config>> SOLR::solrconfig.xml Lucene/SOLR <<HTTP/ReST>> <<PROVISIONING>> <<config>> (Apache) Lucene/SOLR Luminis:Enricher.xml <<Container>> (Apache) <<config>> Tika Apache Tomcat SOLR::schema.xml (Apache) <<HTTP/ReST>> Felix OSGi Wicket <<config>> servicenix::config.xml (Apache) (Apache) OpenRDF <<HTTP>> Lucene/SOLR Enricher (Apache) (Luminis) <<HTTP>> <<config>> <<Data Container>> Wicket SOLR::solrconfig.xml SQL <<JDBC>> (Apache) <<config>> Luminis:Enricher.xml <<JDBC>> <<Database>> <<RDFTripleStore>> Enricher <<config>> Knowledge Models Knowledge Models SOLR::schema.xml (Luminis) <<config>> SOLR::solrconfig.xml <<config>> Luminis:Enricher.xml <<config>> SOLR::schema.xml
  • 42. Conclusions •Enterprise Search Solution is not Google search •Open Source paves the way; misses some ingredients • Useful ingredients: Lucene/Solr, Nutch, Tika, ServiceMix/Camel, Wicket, MySQL, OpenRDF, Felix/Ace • Missing ingredients: Enricher •Interesting developments: • Apache Chemistry (CMIS) • Apache Clerezza • Apache Nutch • Apache Connectors Framework (ManifoldCF)
  • 43. Questions & (answers?) Marc Teutelink marc.teutelink@luminis.eu @mteutelink MEAP December 2010 

Editor's Notes