Solr Presentation

Solr Windows Installation Download and install Tomcat for Windows using the MSI installer. Install it with the tcnative.dll file. Say you installed it in c:\tomcat\ Check if Tomcat is installed correctly by going to http://localhost:8080/ Change the c:\tomcat\conf\server.xml file to add the URIEncoding Connector element as shown below. <Connector port="8080" maxHttpHeaderSize="8192" URIEncoding="UTF-8" maxThreads="150" minSpareThreads="25" maxSpareThreads="75" enableLookups="false" redirectPort="8443" acceptCount="100" connectionTimeout="20000" disableUploadTimeout="true" /> Download and unzip the Solr distribution zip file into (say) c:\temp\solrZip\

Solr Windows Installation Make a directory called solr where you intend the application server to function, say c:\web\solr\ Copy the contents of the example\solr directory c:\temp\solrZip\example\solr\ to c:\web\solr\ Stop the Tomcat service Copy the *solr*.war file from c:\temp\solrZip\dist\ to the Tomcat webapps directory c:\tomcat\webapps\ Rename the *solr*.war file solr.war Use the system tray icon to configure Tomcat to start with the following Java option: -Dsolr.solr.home=c:\web\solr Alternative to the previous step goto C:\tomcat\conf\Catalina\localhost and create a file named ”solr.xml” having this line of code (see below in the notes) . But to run the server this way you will not keep the solr.war in webapps folder of tomcat but rather in some other folder like in this case I have kept it in ${catalina.home}/newsolr/solr.war. Start the Tomcat service Go to the solr admin page to verify that the installation is working. It will be at http://localhost:8080/solr/admin

In Solr and Lucene, an index is built of one or more Documents. A Document consists of one or more Fields. A Field consists of a name, content, and metadata telling Solr how to handle the content. For instance, Fields can contain strings, numbers, booleans, or dates, as well as any types you wish to add. A Field can be described using a number of options that tell Solr how to treat the content during indexing and searching.

Field attributes The contents of a stored Field are saved in the index. This is useful for retrieving and highlighting the contents for display but is not necessary for the actual search. For example, many applications store pointers to the location of contents rather than the actual contents of a file. stored Indexed Fields are searchable and sortable. You also can run Solr's analysis process on indexed Fields, which can alter the content to improve or change results. The following section provides more information about Solr's analysis process. indexed Description Attribute name

Example "Solr Home" Directory ============================= This directory is provided as an example of what a "Solr Home" directory should look like. It's not strictly necessary that you copy all of the files in this directory when setting up a new instance of Solr, but it is recommended. Basic Directory Structure ------------------------- The Solr Home directory typically contains the following subdirectories... conf/ This directory is mandatory and must contain your solrconfig.xml and schema.xml. Any other optional configuration files would also be kept here. data/ This directory is the default location where Solr will keep your index, and is used by the replication scripts for dealing with snapshots. You can override this location in the solrconfig.xml and scripts.conf files. Solr will create this directory if it does not already exist. lib/ This directory is optional. If it exists, Solr will load any Jars found in this directory and use them to resolve any "plugins" specified in your solrconfig.xml or schema.xml (ie: Analyzers, Request Handlers, etc...) bin/ This directory is optional. It is the default location used for keeping the replication scripts.

What Is Solr SOLR is a REST layer for Lucene Began life at CNET to provide a robust search system Joined Apache Incubator in January 2006 Graduated to Lucene sub-project status in January 2007 A full text search server based on Lucene XML/HTTP Interfaces Loose Schema to define types and fields Web Administration Interface Extensive Caching Index Replication Extensible Open Architecture Written in Java5, deployable as a WAR

Why use SOLR? Easy to set up and get started Powerful full text searching Cross platform - Java and REST Under active development Fast Adds extra functionality on top of Lucene: replication CSV importing JSON results results highlighting synonym support

Adding Documents HTTP POST to /update <add><doc boost=“2”> <field name=“article”>05991</field> <field name=“title”>Apache Solr</field> <field name=“subject”>An intro...</field> <field name=“category”>search</field> <field name=“category”>lucene</field> <field name=“body”>Solr is a full...</field> </doc></add>

Deleting Documents Delete by Id <delete><id>05591</id></delete> Delete by Query (multiple documents) <delete> <query>manufacturer:microsoft</query> </delete>

Commit <commit/> makes changes visible closes IndexWriter removes duplicates opens new IndexSearcher newSearcher/firstSearcher events cache warming “ register” the new IndexSearcher <optimize/> same as commit, merges all index segments.

Lucene syntax Required search term – use a “+” +ipod +belkin Field-specific searching – use fieldName name:ipod manu:belkin Wildcard searching – use * or ? ip?d belk* *deo (currently requires modifying solr source) Range searching timestamp:[2006-07-16T12:30:00Z to *] Time needs to be full ISO Proximity searching – use a “~” "video ipod"~3 – up to 3 words apart Fuzzy searches – use a “~” ipod~ will find ipod and ipods belkin~0.7 will find words close spellings

Default Query Syntax Lucene Query Syntax [; sort specification] 1. mission impossible; releaseDate desc 2. +mission +impossible –actor:cruise 3. “mission impossible” –actor:cruise 4. title:spiderman^10 description:spiderman 5. description:“spiderman movie”~10 6. +HDTV +weight:[0 TO 100] 7. Wildcard queries: te?t, te*t, test*

Full control panel interface Start row/max rows – pagination Output type standard (xml), python, json, ruby, xslt Enable highlighting fields to highlight works on wildcard matches

Default Parameters Query Arguments for HTTP GET/POST to /select

Search Results http://localhost:8983/solr/select?q=video&start=0&rows=2&fl=name,price <response><responseHeader><status>0</status> <QTime>1</QTime></responseHeader> <result numFound="16173" start="0"> <doc> <str name="name">Apple 60 GB iPod with Video</str> <float name="price">399.0</float> </doc> <doc> <str name="name">ASUS Extreme N7800GTX/2DHTV</str> <float name="price">479.95</float> </doc> </result> </response>

Caching IndexSearcher’s view of an index is fixed • Aggressive caching possible • Consistency for multi-query requests filterCache – unordered set of document ids matching a query resultCache – ordered subset of document ids matching a query documentCache – the stored fields of documents userCaches – application specific, custom query handlers

Warming for Speed Lucene IndexReader warming field norms, FieldCache, tii – the term index Static Cache warming Configurable static requests to warm new Searchers Smart Cache Warming (autowarming) Using MRU items in the current cache to prepopulate the new cache Warming in parallel with live requests

Schema Lucene has no notion of a schema Sorting - string vs. numeric Ranges - val:42 included in val:[1 TO 5] ? Lucene QueryParser has date-range support, but must guess. Defines fields, their types, properties Defines unique key field, default search field, Similarity implementation

Field Definitions Field Attributes: name, type, indexed, stored, multiValued, omitNorms <field name="id“ type="string" indexed="true" stored="true"/> <field name="sku“ type="textTight” indexed="true" stored="true"/> <field name="name“ type="text“ indexed="true" stored="true"/> <field name=“reviews“ type="text“ indexed="true“ stored=“false"/> <field name="category“ type="text_ws“ indexed="true" stored="true“ multiValued="true"/> Dynamic Fields, in the spirit of Lucene! <dynamicField name="*_i" type="sint“ indexed="true" stored="true"/> <dynamicField name="*_s" type="string“ indexed="true" stored="true"/> <dynamicField name="*_t" type="text“ indexed="true" stored="true"/>

Configuring Relevancy <fieldtype name="text" class="solr.TextField"> <analyzer> <tokenizer class="solr.WhitespaceTokenizerFactory"/> <filter class="solr.LowerCaseFilterFactory"/> <filter class="solr.SynonymFilterFactory" synonyms="synonyms.txt“/> <filter class="solr.StopFilterFactory“ words=“stopwords.txt”/> <filter class="solr.EnglishPorterFilterFactory" protected="protwords.txt"/> </analyzer> </fieldtype>

copyField Copies one field to another at index time Usecase: Analyze same field different ways copy into a field with a different analyzer boost exact-case, exact-punctuation matches language translations, thesaurus, soundex <field name=“title” type=“text”/> <field name=“title_exact” type=“text_exact” stored=“false”/> <copyField source=“title” dest=“title_exact”/> Usecase: Index multiple fields into single searchable field

Web Admin Interface Show Config, Schema, Distribution info Query Interface Statistics Caches: lookups, hits, hitratio, inserts, evictions, size RequestHandlers: requests, errors UpdateHandler: adds, deletes, commits, optimizes IndexReader, open-time, index-version, numDocs, maxDocs, Analysis Debugger Shows tokens after each Analyzer stage Shows token matches for query vs index

Selling Points Fast Powerful & Configurable High Relevancy Mature Product Same features as software costing $$$ Leverage Community Lucene committers, IR experts Free consulting: shared problems & solutions

Where are we going? OOTB Simple Faceted Browsing Automatic Database Indexing Federated Search HA with failover Alternate output formats (JSON, Ruby) Highlighter integration Spellchecker Alternate APIs (Google Data, OpenSearch)

Solr Presentation

More Related Content

What's hot (20)

Viewers also liked (20)

Similar to Solr Presentation (20)

Recently uploaded (20)

Solr Presentation