Start Your Search Engines: Optimizing Solr to Improve Results

Editor's Notes

• #7: • Complex migration and integration projects are a backbone of our company• ExactTarget Gold Partner with full integration between ExactTarget and Magento• We’ve built e-commerce sites ground up, handled complicated product catalog migrations for large B2B companies, and integrated email, ecommerce, digital experience, and business analytic solutions for B2C retail companies. 
• #8: For the next hour we will be speaking about the integrations of Solr and Magento and making the setup work best for your ecommerce site.Today we are going to go over more advanced topics such as:Basic Troubleshooting-Useful Solr tools and Common problems and solutions.Advanced optimization of search results.-Making changes in Solr configuration to better your results. In the previous presentation we covered modifications direclty in Magento. Today we will be covering changes done to Solr.Improving search speed-Optimizing Magento to improve search.
• #9: Solr is an open source enterprise search platform from the Apache Lucene project. Its major features include powerful full-text search, hit highlighting, faceted search, dynamic clustering, database integration, and rich document handling.Magento Enterprise integrates with Solr right out of the box.
• #10: We did a more in depth introduction of Solr September of 2011. You can watch the full video by going to the URL displayed or by going to Magento's webinar section on their website. It covers setup, indexing, and fine tuning search results through Magento.
• #11: So now let us go over useful Solr tools and Common problems and solutions.
• #12: Web Interface (5 minutes)Schema fileIf you make file changes you can confirm Solr has loaded them by looking for them in this file.Show config fileIf you make file changes you can confirm Solr has loaded them by looking for them in this file.Schema BrowserNumber of docs in the indexActual indexed fields and some statistics about them.Ping URLThe URL used to test if Solr is running properly.Solr StatsRequest handlers used and other high level stats and configurations.readDir pathLuke (5 minutes)Lucene Index BrowserTokenized terms for searchCommand Line (5 minutes)Show logs during indexShow logs during query
• #13: Do you have the right URL and port?For example the default port for Tomcat for 8080 and Jetty is 8983.Show test button.What the button actually does.Ping URL to Solr and the response.PHP Setting to fix it and why. (90% of the time it's fixed by this.)
• #14: What the problem is…Bad data, Solr not committing changes.Final commit vs Partial commit.How to diagnose this issue. (Tailing the log look for rollback)It tells which product ID has critical error.
• #16: Direct configuration changes in Solr to better suite you business needs.There are two different types of settings in Solr: Query time and Index time.Query time settings are settings that take effect when a Query is ran. These do not require a reindex of data.Index time settings are used during index, if a change is made to index time setting then you must reindex to see the changes take place.
• #17: When dealing with queries there are 3 types of "clauses" that Lucene knows about: mandatory, prohibited, and 'optional' (aka: "SHOULD") By default all words or phrases specified in the "q" param are treated as "optional" clauses unless they are preceeded by a "+" or a "-". When dealing with these "optional" clauses, the "mm" option makes it possible to say that a certain minimum number of those clauses must match (mm). Specifying this minimum number can be done in complex ways, equating to ideas like...   At least 2 of the optional clauses must match, regardless of how many clauses there are: "2"At least 75% of the optional clauses must match, rounded down: "75%"  If there are less than 3 optional clauses, they all must match; if there are 3 or more, then 75% must match, rounded up: "2<-25%"  If there are less than 3 optional clauses, they all must match; for 3 to 5 clauses, one less than the number of clauses must match, for 6 or more clauses, 80% must match, rounded down: "2<-1 5<80%"This is modified in the query time configuration file solrconfig.xmlThis setting is language specific.No reindex will be needed
• #18: Perhaps there will be a situation when products will need to be promoted in your search or boosted. With Solr's "Boost Query" parameter this can easily be accomplished.This is modified in the query time configuration file solrconfig.xmlThis setting is language specific.No reindex will be needed
• #20: How Magento and Solr Work together-First of all it's not Solr it's Magento-Solr returns product IDs not data. Magento does the data grabber
• #22: Checking query time logging for the "Q" time in milliseconds.Solr optimization that we do not have time to cover here. Go to: http://wiki.apache.org/solr/SolrPerformanceFactors
• #23: Make sure you have the most recent version of MySQLMake sure your MySQL settings are tuned per Magento's recommendations.Use the Memory (HEAP) storage engine for temp tables.Leverage MySQL query caching as recommended by Magento.