SlideShare a Scribd company logo

Solr Anti Patterns

Sematext Group, Inc. , profile picture
Sematext Group, Inc.

The document discusses various Solr anti-patterns and best practices for optimizing Solr performance, including properly configuring request handlers, schema fields, thread pools, caching, indexing, and faceting. It provides examples of incorrect configurations that can cause issues and recommendations for improved configurations to avoid problems and optimize querying, indexing, and response times.

Technology
1 of 71
Downloaded 71 times
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
Solr Anti Patterns
Solr Anti - patterns Rafał Kuć, Sematext Group, Inc. @kucrafal @sematext http://sematext.com
About me Sematext consultant & engineer Solr.pl co-founder Father & husband
The (not so) perfect migration http://en.wikipedia.org/wiki/Bird_migration http://www.likesbooks.com/aarafterhours/?p=750
From 3.1 to 4.10 (and hopefully not back) March 2011 September 2014
The lonely solrconfig.xml <requestHandler name="/update" class="solr.XmlUpdateRequestHandler" /> <requestHandler name="/update/javabin" class="solr.BinaryUpdateRequestHandler" /> <requestHandler name="/update/csv" class="solr.CSVRequestHandler" /> <requestHandler name="/update/json" class="solr.JsonUpdateRequestHandler" /> <luceneMatchVersion>LUCENE_31</luceneMatchVersion> <directoryFactory name="DirectoryFactory" class="${solr.directoryFactory:solr.StandardDirectoryFactory}"/>
DOC DOC DOC And faulty indexing EXCEPTIONS :)
And faulty indexing <?xml version="1.0" encoding="UTF-8"?> <response> <lst name="responseHeader"> <int name="status">400</int> <int name="QTime">0</int> </lst> <lst name="error"> <str name="msg">missing content stream</str> <int name="code">400</int> </lst> </response> 109173 [qtp1223685984-20] ERROR org.apache.solr.core.SolrCore ľ org.apache.solr.common.SolrException: missing content stream at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:69) at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135) at org.apache.solr.core.SolrCore.execute(SolrCore.java:1967) at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:777) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:418) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:207) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1075) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135) at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255) at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116) at org.eclipse.jetty.server.Server.handle(Server.java:368) at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489) at org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:53) at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:942) at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1004) at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:647) at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235) at org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:72) at org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:264) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543) at java.lang.Thread.run(Unknown Source)
Let’s make that right <requestHandler name="/update" class="solr.UpdateRequestHandler" /> <requestHandler name="/update/json" class="solr.UpdateRequestHandler"> <lst name="defaults"> <str name="stream.contentType">application/json</str> </lst> </requestHandler> <luceneMatchVersion>LUCENE_4.10.0</luceneMatchVersion> <directoryFactory name="DirectoryFactory" class="${solr.directoryFactory:solr.NRTCachingDirectoryFactory}"/> <nrtMode>true</nrtMode> <updateLog> <str name="dir"> ${solr.ulog.dir:} </str> </updateLog>
The old schema.xml <fieldType name="int" class="solr.IntField" omitNorms="true"/> <fieldType name="long" class="solr.LongField" omitNorms="true"/> <fieldType name="float" class="solr.FloatField" omitNorms="true"/> <fieldType name="double" class="solr.DoubleField" omitNorms="true"/> <fieldType name="date" class="solr.DateField" sortMissingLast="true" omitNorms="true"/> <fieldType name="sint" class="solr.SortableIntField" sortMissingLast="true" omitNorms="true"/> <fieldType name="slong" class="solr.SortableLongField" sortMissingLast="true" omitNorms="true"/> <fieldType name="sfloat" class="solr.SortableFloatField" sortMissingLast="true" omitNorms="true"/> <fieldType name="sdouble" class="solr.SortableDoubleField" sortMissingLast="true" omitNorms="true"/>
<fieldType name="int" class="solr.IntField" omitNorms="true"/> <fieldType name="long" class="solr.LongField" omitNorms="true"/> <fieldType name="float" class="solr.FloatField" omitNorms="true"/> <fieldType name="double" class="solr.DoubleField" omitNorms="true"/> <fieldType name="date" class="solr.DateField" sortMissingLast="true" omitNorms="true"/> <fieldType name="sint" class="solr.SortableIntField" sortMissingLast="true" omitNorms="true"/> <fieldType name="slong" class="solr.SortableLongField" sortMissingLast="true" omitNorms="true"/> <fieldType name="sfloat" class="solr.SortableFloatField" sortMissingLast="true" omitNorms="true"/> <fieldType name="sdouble" class="solr.SortableDoubleField" sortMissingLast="true" omitNorms="true"/> The old schema.xml
The new schema.xml <fieldType name="int" class="solr.TrieIntField" precisionStep="0" positionIncrementGap="0"/> <fieldType name="float" class="solr.TrieFloatField" precisionStep="0" positionIncrementGap="0"/> <fieldType name="long" class="solr.TrieLongField" precisionStep="0" positionIncrementGap="0"/> <fieldType name="double" class="solr.TrieDoubleField" precisionStep="0" positionIncrementGap="0"/> <fieldType name="date" class="solr.TrieDateField" precisionStep="0" positionIncrementGap="0"/> <fieldType name="tint" class="solr.TrieIntField" precisionStep="8" positionIncrementGap="0"/> <fieldType name="tfloat" class="solr.TrieFloatField" precisionStep="8" positionIncrementGap="0"/> <fieldType name="tlong" class="solr.TrieLongField" precisionStep="8" positionIncrementGap="0"/> <fieldType name="tdouble" class="solr.TrieDoubleField" precisionStep="8" positionIncrementGap="0"/> <fieldType name="tdate" class="solr.TrieDateField" precisionStep="6" positionIncrementGap="0"/>
Threads? What threads? <Set name="ThreadPool"> <New class="org.eclipse.jetty.util.thread.QueuedThreadPool"> <Set name="minThreads">10</Set> <Set name="maxThreads">200</Set> <Set name="detailedDump">false</Set> </New> </Set>
I see deadlocks
Threads? What threads? <Set name="ThreadPool"> <New class="org.eclipse.jetty.util.thread.QueuedThreadPool"> <Set name="minThreads">10</Set> <Set name="maxThreads">200</Set> <Set name="detailedDump">false</Set> </New> </Set>
OK, so now we can actually run queries <Set name="ThreadPool"> <New class="org.eclipse.jetty.util.thread.QueuedThreadPool"> <Set name="minThreads">10</Set> <Set name="maxThreads">10000</Set> <Set name="detailedDump">false</Set> </New> </Set>
The ZooKeeper
The ZooKeeper
The ZooKeeper
The ZooKeeper
The ZooKeeper
The ZooKeeper – production
The ZooKeeper – production -DzkHost=zk1:2181,zk2:2181,zk3:2181
The ZooKeeper – production -DzkHost=zk1:2181,zk2:2181,zk3:2181
The ZooKeeper – production -DzkHost=zk1:2181,zk2:2181,zk3:2181
The ZooKeeper – production -DzkHost=zk1:2181,zk2:2181,zk3:2181
Let’s cache everything <filterCache class="solr.LRUCache" size="1048576" initialSize="1048576" autowarmCount="524288"/> <queryResultCache class="solr.LRUCache" size="1048576" initialSize="1048576" autowarmCount="524288"/><documentCache class="solr.LRUCache" size="1048576" initialSize="1048576" autowarmCount="0"/>
And now let’s look at the warmup times
And now let’s look at the warmup times
OK, show us the way „Mr. Consultant” <filterCache class="solr.FastLRUCache" size="1024" initialSize="1024" autowarmCount="512"/> <queryResultCache class="solr.LRUCache" size="16000" initialSize="16000" autowarmCount="8000"/><documentCache class="solr.LRUCache" size="16384" initialSize="16384" autowarmCount="0"/>
Let’s look at the warmup times again
Let’s look at the warmup times again
Bulks are for noobs Application Application Application Doc Doc Doc
Bulks are for noobs Application Application Application Doc Doc Doc
But let’s use bulks, just in case
But let’s use bulks, just in case
We need to refresh and hard commit <autoCommit> <maxTime>1000</maxTime> <openSearcher>true</openSearcher> </autoCommit> <autoSoftCommit> <maxTime>1000</maxTime> </autoSoftCommit>
Maybe we should only refresh? <autoCommit> <maxTime>60000</maxTime> <openSearcher>false</openSearcher> </autoCommit> <autoSoftCommit> <maxTime>1000</maxTime> </autoSoftCommit>
OK, let’s go easy with refreshing <autoCommit> <maxTime>60000</maxTime> <openSearcher>false</openSearcher> </autoCommit> <autoSoftCommit> <maxTime>30000</maxTime> </autoSoftCommit>
But I really need all that data curl -XGET 'localhost:8983/solr/select?q=*:*&start=3000000&rows=100'
<?xml version="1.0" encoding="UTF-8"?> <response> <lst name="responseHeader"> <int name="status">0</int> <int name="QTime">9418</int> <lst name="params"> <str name="start">3000000</str> <str name="q">*:*</str> <str name="rows">100</str> </lst> </lst> <result name="response" numFound="3284000" start="3000000"> . . . </result> </response> But I really need all that data curl -XGET 'localhost:8983/solr/select?q=*:*&start=3000000&rows=100'
<?xml version="1.0" encoding="UTF-8"?> <response> <lst name="responseHeader"> <int name="status">0</int> <int name="QTime">9418</int> <lst name="params"> <str name="start">3000000</str> <str name="q">*:*</str> <str name="rows">5</str> </lst> </lst> <result name="response" numFound="3284000" start="3000000"> . . . </result> </response> But I really need all that data <?xml version="1.0" encoding="UTF-8"?> <response> <lst name="error"> <str name="msg">java.lang.OutOfMemoryError: Java heap space</str> <str name="trace">java.lang.RuntimeException: java.lang.OutOfMemoryError: Java heap space at org.apache.solr.servlet.SolrDispatchFilter.sendError(SolrDispatchFilter.java:796) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:448) . . . Caused by: java.lang.OutOfMemoryError: Java heap space . . . </str> <int name="code">500</int> </lst> </response> curl -XGET 'localhost:8983/solr/select?q=*:*&start=3000000&rows=100'
But I really need all that data Query
But I really need all that data
But I really need all that data
But I really need all that data Response
Use the scroll Luke curl -XGET 'localhost:8983/solr/select?q=*:*&cursorMark=*&sort=score+desc,id+desc'
Use the scroll Luke curl -XGET 'localhost:8983/solr/select?q=*:*&cursorMark=*&sort=score+desc,id+desc' <?xml version="1.0" encoding="UTF-8"?> <response> <lst name="responseHeader"> <int name="status">0</int> <int name="QTime">189</int> <lst name="params"> <str name="sort">score desc,id desc</str> <str name="q">*:*</str> <str name="cursorMark">*</str> </lst> </lst> <result name="response" numFound="3284000" start="0"> <doc> ... </doc> . . . </result> <str name="nextCursorMark">AoIIP4AAACY5OTk5OTA=</str> </response>
Use the scroll Luke curl -XGET 'localhost:8983/solr/select?q=*:*&sort=score+desc,id+desc &cursorMark=AoIIP4AAACY5OTk5OTA='
Use the scroll Luke curl -XGET 'localhost:8983/solr/select?q=*:*&sort=score+desc,id+desc &cursorMark=AoIIP4AAACY5OTk5OTA=' <?xml version="1.0" encoding="UTF-8"?> <response> <lst name="responseHeader"> <int name="status">0</int> <int name="QTime">184</int> <lst name="params"> <str name="sort">score desc,id desc</str> <str name="q">*:*</str> <str name="cursorMark">AoIIP4AAACY5OTk5OTA=</str> </lst> </lst> <result name="response" numFound="3284000" start="0"> <doc> ... </doc> . . . </result> <str name="nextCursorMark">AoIIP4AAACY5OTk5ODE=</str> </response>
Limiting faceting, why bother? curl -XGET 'localhost:8983/solr/select?q=*:*&facet=true&facet.field=tag&… facet.limit=-1&facet.mincount=0'
Limiting faceting, why bother? curl -XGET 'localhost:8983/solr/select?q=*:*&facet=true&facet.field=tag&… facet.limit=-1&facet.mincount=0' <?xml version="1.0" encoding="UTF-8"?> <response> <lst name="responseHeader"> <int name="status">0</int> <int name="QTime">9967</int> <lst name="params"> ... </lst> </lst> <result name="response" numFound="3284000" start="0"> . . . </result> <lst name="facet_counts"> <lst name="facet_fields"> <lst name="tag"> ... </lst> </lst> </lst> </response>
Limiting faceting, why bother? curl -XGET 'localhost:8983/solr/select?q=*:*&facet=true&facet.field=tag&… facet.limit=-1&facet.mincount=0' <?xml version="1.0" encoding="UTF-8"?> <response> . . . <lst name="error"> <str name="msg">Error while processing facet fields: java.lang.OutOfMemoryError: Java heap space</str> <str name="trace">org.apache.solr.common.SolrException: Error while processing facet fields: java.lang.OutOfMemoryError: Java heap space . . . Caused by: java.lang.OutOfMemoryError: Java heap space at org.apache.solr.request.SimpleFacets.getFieldCacheCounts(SimpleFacets.java:685) . . . </str> <int name="code">500</int> </lst> </response>
Now let’s look at performance
Now let’s look at performance
Now let’s look at performance
Now let’s look at performance
Now let’s look at performance
Magic happens with small changes curl -XGET 'localhost:8983/solr/select?q=*:*&facet=true&facet.field=tag&… facet.limit=100&facet.mincount=1'
Magic happens with small changes
Magic happens with small changes
Magic happens with small changes
Magic happens with small changes
Magic happens with small changes
Magic happens with small changes
Magic happens with small changes
Monitoring in production http://sematext.com/spm/index.html
And remember… <luceneMatchVersion> 3.1 </luceneMatchVersion>
Quick summary http://www.soothetube.com/2013/12/29/thats-all-folks/
We are hiring! Dig Search? Dig Analytics? Dig Big Data? Dig Performance? Dig Logging? Dig working with and in open – source? We’re hiring world – wide! http://sematext.com/about/jobs.html
Thank you! Rafał Kuć @kucrafal rafal.kuc@sematext.com Sematext @sematext http://sematext.com http://blog.sematext.com

More Related Content

PDF
Solr Anti-Patterns: Presented by Rafał Kuć, Sematext
Lucidworks
 
PDF
Rapid Prototyping with Solr
Erik Hatcher
 
PPTX
Apache Solr + ajax solr
Net7
 
PDF
Solr & Lucene @ Etsy by Gregg Donovan
Gregg Donovan
 
PDF
Solr Troubleshooting - TreeMap approach
Alexandre Rafalovitch
 
PPS
Introduction to Solr
Jayesh Bhoyar
 
PDF
Solr @ Etsy - Apache Lucene Eurocon
Giovanni Fernandez-Kincade
 
PPTX
Open Source Search: An Analysis
Justin Finkelstein
 
Solr Anti-Patterns: Presented by Rafał Kuć, Sematext
Lucidworks
 
Rapid Prototyping with Solr
Erik Hatcher
 
Apache Solr + ajax solr
Net7
 
Solr & Lucene @ Etsy by Gregg Donovan
Gregg Donovan
 
Solr Troubleshooting - TreeMap approach
Alexandre Rafalovitch
 
Introduction to Solr
Jayesh Bhoyar
 
Solr @ Etsy - Apache Lucene Eurocon
Giovanni Fernandez-Kincade
 
Open Source Search: An Analysis
Justin Finkelstein
 

What's hot (20)

ODP
Mastering solr
jurcello
 
PDF
Make your gui shine with ajax solr
lucenerevolution
 
PDF
it's just search
Erik Hatcher
 
PDF
Solr workshop
Yasas Senarath
 
PDF
Solr Indexing and Analysis Tricks
Erik Hatcher
 
PDF
New SPL Features in PHP 5.3
Matthew Turland
 
PPTX
Let's write secure Drupal code! DUG Belgium - 08/08/2019
Balázs Tatár
 
PPTX
JSON in Solr: from top to bottom
Alexandre Rafalovitch
 
ODP
Intro to The PHP SPL
Chris Tankersley
 
PPTX
SPL: The Undiscovered Library - DataStructures
Mark Baker
 
ODP
Searching for AI - Leveraging Solr for classic Artificial Intelligence tasks
Alexandre Rafalovitch
 
PPTX
Rebuilding Solr 6 examples - layer by layer (LuceneSolrRevolution 2016)
Alexandre Rafalovitch
 
TXT
My shell
Ahmed Salah
 
PDF
The Origin of Lithium
Nate Abele
 
PDF
Solr & Lucene at Etsy
Lucidworks (Archived)
 
PDF
Solr and Lucene at Etsy - By Gregg Donovan
lucenerevolution
 
PDF
The State of Lithium
Nate Abele
 
PDF
Class-based views with Django
Simon Willison
 
PDF
Living with garbage
lucenerevolution
 
KEY
Drupal for ng_os
dstuartnz
 
Mastering solr
jurcello
 
Make your gui shine with ajax solr
lucenerevolution
 
it's just search
Erik Hatcher
 
Solr workshop
Yasas Senarath
 
Solr Indexing and Analysis Tricks
Erik Hatcher
 
New SPL Features in PHP 5.3
Matthew Turland
 
Let's write secure Drupal code! DUG Belgium - 08/08/2019
Balázs Tatár
 
JSON in Solr: from top to bottom
Alexandre Rafalovitch
 
Intro to The PHP SPL
Chris Tankersley
 
SPL: The Undiscovered Library - DataStructures
Mark Baker
 
Searching for AI - Leveraging Solr for classic Artificial Intelligence tasks
Alexandre Rafalovitch
 
Rebuilding Solr 6 examples - layer by layer (LuceneSolrRevolution 2016)
Alexandre Rafalovitch
 
My shell
Ahmed Salah
 
The Origin of Lithium
Nate Abele
 
Solr & Lucene at Etsy
Lucidworks (Archived)
 
Solr and Lucene at Etsy - By Gregg Donovan
lucenerevolution
 
The State of Lithium
Nate Abele
 
Class-based views with Django
Simon Willison
 
Living with garbage
lucenerevolution
 
Drupal for ng_os
dstuartnz
 
Ad

Viewers also liked (20)

PDF
Tuning Solr for Logs
Sematext Group, Inc.
 
PPTX
Tuning Elasticsearch Indexing Pipeline for Logs
Sematext Group, Inc.
 
PDF
Going All-In With Go For CLI Apps
Tom Elliott
 
PPTX
Running High Performance & Fault-tolerant Elasticsearch Clusters on Docker
Sematext Group, Inc.
 
PDF
Docker Logging Webinar
Sematext Group, Inc.
 
PDF
Top Node.js Metrics to Watch
Sematext Group, Inc.
 
PDF
Tuning Solr & Pipeline for Logs
Sematext Group, Inc.
 
PPTX
Musings on Secondary Indexing in HBase
Jesse Yates
 
PPTX
MongoDB and Apache HBase: Benchmarking
Olga Lavrentieva
 
PDF
Ease of use in Apache Solr
Anshum Gupta
 
ODP
Search Analytics with Flume and HBase
Sematext Group, Inc.
 
PDF
Docker Monitoring Webinar
Sematext Group, Inc.
 
PPTX
Apache HBase Application Archetypes
Cloudera, Inc.
 
PDF
From Zero to Production Hero: Log Analysis with Elasticsearch (from Velocity ...
Sematext Group, Inc.
 
PDF
Large Scale Log Analytics with Solr (from Lucene Revolution 2015)
Sematext Group, Inc.
 
PDF
Improvements to Flink & it's Applications in Alibaba Search
DataWorks Summit/Hadoop Summit
 
PDF
Introduction to solr
Sematext Group, Inc.
 
PDF
From Zero to Hero - Centralized Logging with Logstash & Elasticsearch
Sematext Group, Inc.
 
PDF
Numeric Range Queries in Lucene and Solr
Vadim Kirilchuk
 
PDF
Using Morphlines for On-the-Fly ETL
Cloudera, Inc.
 
Tuning Solr for Logs
Sematext Group, Inc.
 
Tuning Elasticsearch Indexing Pipeline for Logs
Sematext Group, Inc.
 
Going All-In With Go For CLI Apps
Tom Elliott
 
Running High Performance & Fault-tolerant Elasticsearch Clusters on Docker
Sematext Group, Inc.
 
Docker Logging Webinar
Sematext Group, Inc.
 
Top Node.js Metrics to Watch
Sematext Group, Inc.
 
Tuning Solr & Pipeline for Logs
Sematext Group, Inc.
 
Musings on Secondary Indexing in HBase
Jesse Yates
 
MongoDB and Apache HBase: Benchmarking
Olga Lavrentieva
 
Ease of use in Apache Solr
Anshum Gupta
 
Search Analytics with Flume and HBase
Sematext Group, Inc.
 
Docker Monitoring Webinar
Sematext Group, Inc.
 
Apache HBase Application Archetypes
Cloudera, Inc.
 
From Zero to Production Hero: Log Analysis with Elasticsearch (from Velocity ...
Sematext Group, Inc.
 
Large Scale Log Analytics with Solr (from Lucene Revolution 2015)
Sematext Group, Inc.
 
Improvements to Flink & it's Applications in Alibaba Search
DataWorks Summit/Hadoop Summit
 
Introduction to solr
Sematext Group, Inc.
 
From Zero to Hero - Centralized Logging with Logstash & Elasticsearch
Sematext Group, Inc.
 
Numeric Range Queries in Lucene and Solr
Vadim Kirilchuk
 
Using Morphlines for On-the-Fly ETL
Cloudera, Inc.
 
Ad

Similar to Solr Anti Patterns (20)

PDF
[제1회 루씬 한글분석기 기술세미나] solr로 나만의 검색엔진을 만들어보자
Donghyeok Kang
 
PDF
A noobs lesson on solr (configuration)
BTI360
 
PDF
Apache Solr Search Mastery
Acquia
 
PPTX
Solr02 fields
Ismaeel Enjreny
 
PPTX
Solr vs. Elasticsearch - Case by Case
Alexandre Rafalovitch
 
PDF
Rapid Prototyping with Solr
Erik Hatcher
 
PDF
Beyond full-text searches with Lucene and Solr
Bertrand Delacretaz
 
PDF
Rapid Prototyping with Solr
Erik Hatcher
 
PDF
Solr vs. Elasticsearch, Case by Case: Presented by Alexandre Rafalovitch, UN
Lucidworks
 
PPTX
Scaling Solr with Solr Cloud
Sematext Group, Inc.
 
PDF
Scaling Solr with SolrCloud
lucenerevolution
 
PDF
XML Schemas
Hoang Nguyen
 
PDF
Cassandra summit
mattstump
 
PDF
เกี่ยวกับ Apache solr 4.0
Somkiat Puisungnoen
 
PDF
Apache solr liferay
Binesh Gummadi
 
ODP
Dev8d Apache Solr Tutorial
Sourcesense
 
PDF
XamarinとAWSをつないでみた話
Takehito Tanabe
 
PDF
Spring Web Service, Spring Integration and Spring Batch
Eberhard Wolff
 
KEY
Eu odeio OpenSocial
Carlos Brando
 
PDF
Create WSDL
SJ Choudary
 
[제1회 루씬 한글분석기 기술세미나] solr로 나만의 검색엔진을 만들어보자
Donghyeok Kang
 
A noobs lesson on solr (configuration)
BTI360
 
Apache Solr Search Mastery
Acquia
 
Solr02 fields
Ismaeel Enjreny
 
Solr vs. Elasticsearch - Case by Case
Alexandre Rafalovitch
 
Rapid Prototyping with Solr
Erik Hatcher
 
Beyond full-text searches with Lucene and Solr
Bertrand Delacretaz
 
Rapid Prototyping with Solr
Erik Hatcher
 
Solr vs. Elasticsearch, Case by Case: Presented by Alexandre Rafalovitch, UN
Lucidworks
 
Scaling Solr with Solr Cloud
Sematext Group, Inc.
 
Scaling Solr with SolrCloud
lucenerevolution
 
XML Schemas
Hoang Nguyen
 
Cassandra summit
mattstump
 
เกี่ยวกับ Apache solr 4.0
Somkiat Puisungnoen
 
Apache solr liferay
Binesh Gummadi
 
Dev8d Apache Solr Tutorial
Sourcesense
 
XamarinとAWSをつないでみた話
Takehito Tanabe
 
Spring Web Service, Spring Integration and Spring Batch
Eberhard Wolff
 
Eu odeio OpenSocial
Carlos Brando
 
Create WSDL
SJ Choudary
 

More from Sematext Group, Inc. (19)

PDF
Tweaking the Base Score: Lucene/Solr Similarities Explained
Sematext Group, Inc.
 
PDF
OOPs, OOMs, oh my! Containerizing JVM apps
Sematext Group, Inc.
 
PPTX
Is observability good for your brain?
Sematext Group, Inc.
 
PDF
Introducing log analysis to your organization
Sematext Group, Inc.
 
PPTX
Solr Search Engine: Optimize Is (Not) Bad for You
Sematext Group, Inc.
 
PDF
Solr on Docker - the Good, the Bad and the Ugly
Sematext Group, Inc.
 
PDF
Monitoring and Log Management for
Sematext Group, Inc.
 
PDF
Building Resilient Log Aggregation Pipeline with Elasticsearch & Kafka
Sematext Group, Inc.
 
PDF
Elasticsearch for Logs & Metrics - a deep dive
Sematext Group, Inc.
 
PDF
How to Run Solr on Docker and Why
Sematext Group, Inc.
 
PPT
Running High Performance and Fault Tolerant Elasticsearch Clusters on Docker
Sematext Group, Inc.
 
PDF
Metrics, Logs, Transaction Traces, Anomaly Detection at Scale
Sematext Group, Inc.
 
PDF
Side by Side with Elasticsearch & Solr, Part 2
Sematext Group, Inc.
 
PDF
(Elastic)search in big data
Sematext Group, Inc.
 
PDF
Side by Side with Elasticsearch and Solr
Sematext Group, Inc.
 
PDF
Open Source Search Evolution
Sematext Group, Inc.
 
PDF
Elasticsearch and Solr for Logs
Sematext Group, Inc.
 
PDF
Introduction to Elasticsearch
Sematext Group, Inc.
 
PPTX
Administering and Monitoring SolrCloud Clusters
Sematext Group, Inc.
 
Tweaking the Base Score: Lucene/Solr Similarities Explained
Sematext Group, Inc.
 
OOPs, OOMs, oh my! Containerizing JVM apps
Sematext Group, Inc.
 
Is observability good for your brain?
Sematext Group, Inc.
 
Introducing log analysis to your organization
Sematext Group, Inc.
 
Solr Search Engine: Optimize Is (Not) Bad for You
Sematext Group, Inc.
 
Solr on Docker - the Good, the Bad and the Ugly
Sematext Group, Inc.
 
Monitoring and Log Management for
Sematext Group, Inc.
 
Building Resilient Log Aggregation Pipeline with Elasticsearch & Kafka
Sematext Group, Inc.
 
Elasticsearch for Logs & Metrics - a deep dive
Sematext Group, Inc.
 
How to Run Solr on Docker and Why
Sematext Group, Inc.
 
Running High Performance and Fault Tolerant Elasticsearch Clusters on Docker
Sematext Group, Inc.
 
Metrics, Logs, Transaction Traces, Anomaly Detection at Scale
Sematext Group, Inc.
 
Side by Side with Elasticsearch & Solr, Part 2
Sematext Group, Inc.
 
(Elastic)search in big data
Sematext Group, Inc.
 
Side by Side with Elasticsearch and Solr
Sematext Group, Inc.
 
Open Source Search Evolution
Sematext Group, Inc.
 
Elasticsearch and Solr for Logs
Sematext Group, Inc.
 
Introduction to Elasticsearch
Sematext Group, Inc.
 
Administering and Monitoring SolrCloud Clusters
Sematext Group, Inc.
 

Recently uploaded (20)

DOCX
The AUB Centre for AI in Media Proposal.docx
GAONE9
 
PDF
Blue Purple Modern Animated Computer Science Presentation.pdf.pdf
Akar16
 
PDF
Optimiser vos workloads AI/ML sur Amazon EC2 et AWS Graviton
Julien SIMON
 
PDF
How UI/UX Design Impacts User Retention in Mobile Apps.pdf
SynapseIndia
 
PDF
NewMind AI Weekly Chronicles - August'25 Week I
NewMind AI
 
PPTX
Understanding_Digital_Forensics_Presentation.pptx
ImranKhan423233
 
PDF
Mobile App Security Testing_ A Comprehensive Guide.pdf
flufftailshop
 
PDF
solutions_manual_-_materials___processing_in_manufacturing__demargo_.pdf
AbdullahSani29
 
PDF
[발표본] 너의 과제는 클라우드에 있어_KTDS_김동현_20250524.pdf
ssuserf8b8bd1
 
PDF
Network Security Unit 5.pdf for BCA BBA.
Serpent6
 
PDF
NewMind AI Monthly Chronicles - July 2025
NewMind AI
 
PDF
Chapter 3 Spatial Domain Image Processing.pdf
Getnet Tigabie Askale -(GM)
 
PDF
KodekX | Application Modernization Development
KodekX
 
PPTX
breach-and-attack-simulation-cybersecurity-india-chennai-defenderrabbit-2025....
defencerabbit Team
 
PPTX
Cloud computing and distributed systems.
kkkkkhan
 
PDF
Shreyas Phanse Resume: Experienced Backend Engineer | Java • Spring Boot • Ka...
SHREYAS PHANSE
 
PPTX
MYSQL Presentation for SQL database connectivity
Swati270511
 
PPT
Teaching material agriculture food technology
LiaRayya
 
PDF
Review of recent advances in non-invasive hemoglobin estimation
IAESIJAI
 
PDF
GDG Cloud Iasi [PUBLIC] Florian Blaga - Unveiling the Evolution of Cybersecur...
athlonica
 
The AUB Centre for AI in Media Proposal.docx
GAONE9
 
Blue Purple Modern Animated Computer Science Presentation.pdf.pdf
Akar16
 
Optimiser vos workloads AI/ML sur Amazon EC2 et AWS Graviton
Julien SIMON
 
How UI/UX Design Impacts User Retention in Mobile Apps.pdf
SynapseIndia
 
NewMind AI Weekly Chronicles - August'25 Week I
NewMind AI
 
Understanding_Digital_Forensics_Presentation.pptx
ImranKhan423233
 
Mobile App Security Testing_ A Comprehensive Guide.pdf
flufftailshop
 
solutions_manual_-_materials___processing_in_manufacturing__demargo_.pdf
AbdullahSani29
 
[발표본] 너의 과제는 클라우드에 있어_KTDS_김동현_20250524.pdf
ssuserf8b8bd1
 
Network Security Unit 5.pdf for BCA BBA.
Serpent6
 
NewMind AI Monthly Chronicles - July 2025
NewMind AI
 
Chapter 3 Spatial Domain Image Processing.pdf
Getnet Tigabie Askale -(GM)
 
KodekX | Application Modernization Development
KodekX
 
breach-and-attack-simulation-cybersecurity-india-chennai-defenderrabbit-2025....
defencerabbit Team
 
Cloud computing and distributed systems.
kkkkkhan
 
Shreyas Phanse Resume: Experienced Backend Engineer | Java • Spring Boot • Ka...
SHREYAS PHANSE
 
MYSQL Presentation for SQL database connectivity
Swati270511
 
Teaching material agriculture food technology
LiaRayya
 
Review of recent advances in non-invasive hemoglobin estimation
IAESIJAI
 
GDG Cloud Iasi [PUBLIC] Florian Blaga - Unveiling the Evolution of Cybersecur...
athlonica
 

Solr Anti Patterns

  • 2. Solr Anti - patterns Rafał Kuć, Sematext Group, Inc. @kucrafal @sematext http://sematext.com
  • 3. About me Sematext consultant & engineer Solr.pl co-founder Father & husband
  • 4. The (not so) perfect migration http://en.wikipedia.org/wiki/Bird_migration http://www.likesbooks.com/aarafterhours/?p=750
  • 5. From 3.1 to 4.10 (and hopefully not back) March 2011 September 2014
  • 6. The lonely solrconfig.xml <requestHandler name="/update" class="solr.XmlUpdateRequestHandler" /> <requestHandler name="/update/javabin" class="solr.BinaryUpdateRequestHandler" /> <requestHandler name="/update/csv" class="solr.CSVRequestHandler" /> <requestHandler name="/update/json" class="solr.JsonUpdateRequestHandler" /> <luceneMatchVersion>LUCENE_31</luceneMatchVersion> <directoryFactory name="DirectoryFactory" class="${solr.directoryFactory:solr.StandardDirectoryFactory}"/>
  • 8. And faulty indexing <?xml version="1.0" encoding="UTF-8"?> <response> <lst name="responseHeader"> <int name="status">400</int> <int name="QTime">0</int> </lst> <lst name="error"> <str name="msg">missing content stream</str> <int name="code">400</int> </lst> </response> 109173 [qtp1223685984-20] ERROR org.apache.solr.core.SolrCore ľ org.apache.solr.common.SolrException: missing content stream at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:69) at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135) at org.apache.solr.core.SolrCore.execute(SolrCore.java:1967) at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:777) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:418) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:207) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1075) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135) at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255) at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116) at org.eclipse.jetty.server.Server.handle(Server.java:368) at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489) at org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:53) at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:942) at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1004) at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:647) at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235) at org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:72) at org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:264) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543) at java.lang.Thread.run(Unknown Source)
  • 9. Let’s make that right <requestHandler name="/update" class="solr.UpdateRequestHandler" /> <requestHandler name="/update/json" class="solr.UpdateRequestHandler"> <lst name="defaults"> <str name="stream.contentType">application/json</str> </lst> </requestHandler> <luceneMatchVersion>LUCENE_4.10.0</luceneMatchVersion> <directoryFactory name="DirectoryFactory" class="${solr.directoryFactory:solr.NRTCachingDirectoryFactory}"/> <nrtMode>true</nrtMode> <updateLog> <str name="dir"> ${solr.ulog.dir:} </str> </updateLog>
  • 10. The old schema.xml <fieldType name="int" class="solr.IntField" omitNorms="true"/> <fieldType name="long" class="solr.LongField" omitNorms="true"/> <fieldType name="float" class="solr.FloatField" omitNorms="true"/> <fieldType name="double" class="solr.DoubleField" omitNorms="true"/> <fieldType name="date" class="solr.DateField" sortMissingLast="true" omitNorms="true"/> <fieldType name="sint" class="solr.SortableIntField" sortMissingLast="true" omitNorms="true"/> <fieldType name="slong" class="solr.SortableLongField" sortMissingLast="true" omitNorms="true"/> <fieldType name="sfloat" class="solr.SortableFloatField" sortMissingLast="true" omitNorms="true"/> <fieldType name="sdouble" class="solr.SortableDoubleField" sortMissingLast="true" omitNorms="true"/>
  • 11. <fieldType name="int" class="solr.IntField" omitNorms="true"/> <fieldType name="long" class="solr.LongField" omitNorms="true"/> <fieldType name="float" class="solr.FloatField" omitNorms="true"/> <fieldType name="double" class="solr.DoubleField" omitNorms="true"/> <fieldType name="date" class="solr.DateField" sortMissingLast="true" omitNorms="true"/> <fieldType name="sint" class="solr.SortableIntField" sortMissingLast="true" omitNorms="true"/> <fieldType name="slong" class="solr.SortableLongField" sortMissingLast="true" omitNorms="true"/> <fieldType name="sfloat" class="solr.SortableFloatField" sortMissingLast="true" omitNorms="true"/> <fieldType name="sdouble" class="solr.SortableDoubleField" sortMissingLast="true" omitNorms="true"/> The old schema.xml
  • 12. The new schema.xml <fieldType name="int" class="solr.TrieIntField" precisionStep="0" positionIncrementGap="0"/> <fieldType name="float" class="solr.TrieFloatField" precisionStep="0" positionIncrementGap="0"/> <fieldType name="long" class="solr.TrieLongField" precisionStep="0" positionIncrementGap="0"/> <fieldType name="double" class="solr.TrieDoubleField" precisionStep="0" positionIncrementGap="0"/> <fieldType name="date" class="solr.TrieDateField" precisionStep="0" positionIncrementGap="0"/> <fieldType name="tint" class="solr.TrieIntField" precisionStep="8" positionIncrementGap="0"/> <fieldType name="tfloat" class="solr.TrieFloatField" precisionStep="8" positionIncrementGap="0"/> <fieldType name="tlong" class="solr.TrieLongField" precisionStep="8" positionIncrementGap="0"/> <fieldType name="tdouble" class="solr.TrieDoubleField" precisionStep="8" positionIncrementGap="0"/> <fieldType name="tdate" class="solr.TrieDateField" precisionStep="6" positionIncrementGap="0"/>
  • 13. Threads? What threads? <Set name="ThreadPool"> <New class="org.eclipse.jetty.util.thread.QueuedThreadPool"> <Set name="minThreads">10</Set> <Set name="maxThreads">200</Set> <Set name="detailedDump">false</Set> </New> </Set>
  • 15. Threads? What threads? <Set name="ThreadPool"> <New class="org.eclipse.jetty.util.thread.QueuedThreadPool"> <Set name="minThreads">10</Set> <Set name="maxThreads">200</Set> <Set name="detailedDump">false</Set> </New> </Set>
  • 16. OK, so now we can actually run queries <Set name="ThreadPool"> <New class="org.eclipse.jetty.util.thread.QueuedThreadPool"> <Set name="minThreads">10</Set> <Set name="maxThreads">10000</Set> <Set name="detailedDump">false</Set> </New> </Set>
  • 22. The ZooKeeper – production
  • 23. The ZooKeeper – production -DzkHost=zk1:2181,zk2:2181,zk3:2181
  • 24. The ZooKeeper – production -DzkHost=zk1:2181,zk2:2181,zk3:2181
  • 25. The ZooKeeper – production -DzkHost=zk1:2181,zk2:2181,zk3:2181
  • 26. The ZooKeeper – production -DzkHost=zk1:2181,zk2:2181,zk3:2181
  • 27. Let’s cache everything <filterCache class="solr.LRUCache" size="1048576" initialSize="1048576" autowarmCount="524288"/> <queryResultCache class="solr.LRUCache" size="1048576" initialSize="1048576" autowarmCount="524288"/><documentCache class="solr.LRUCache" size="1048576" initialSize="1048576" autowarmCount="0"/>
  • 28. And now let’s look at the warmup times
  • 29. And now let’s look at the warmup times
  • 30. OK, show us the way „Mr. Consultant” <filterCache class="solr.FastLRUCache" size="1024" initialSize="1024" autowarmCount="512"/> <queryResultCache class="solr.LRUCache" size="16000" initialSize="16000" autowarmCount="8000"/><documentCache class="solr.LRUCache" size="16384" initialSize="16384" autowarmCount="0"/>
  • 31. Let’s look at the warmup times again
  • 32. Let’s look at the warmup times again
  • 33. Bulks are for noobs Application Application Application Doc Doc Doc
  • 34. Bulks are for noobs Application Application Application Doc Doc Doc
  • 35. But let’s use bulks, just in case
  • 36. But let’s use bulks, just in case
  • 37. We need to refresh and hard commit <autoCommit> <maxTime>1000</maxTime> <openSearcher>true</openSearcher> </autoCommit> <autoSoftCommit> <maxTime>1000</maxTime> </autoSoftCommit>
  • 38. Maybe we should only refresh? <autoCommit> <maxTime>60000</maxTime> <openSearcher>false</openSearcher> </autoCommit> <autoSoftCommit> <maxTime>1000</maxTime> </autoSoftCommit>
  • 39. OK, let’s go easy with refreshing <autoCommit> <maxTime>60000</maxTime> <openSearcher>false</openSearcher> </autoCommit> <autoSoftCommit> <maxTime>30000</maxTime> </autoSoftCommit>
  • 40. But I really need all that data curl -XGET 'localhost:8983/solr/select?q=*:*&start=3000000&rows=100'
  • 41. <?xml version="1.0" encoding="UTF-8"?> <response> <lst name="responseHeader"> <int name="status">0</int> <int name="QTime">9418</int> <lst name="params"> <str name="start">3000000</str> <str name="q">*:*</str> <str name="rows">100</str> </lst> </lst> <result name="response" numFound="3284000" start="3000000"> . . . </result> </response> But I really need all that data curl -XGET 'localhost:8983/solr/select?q=*:*&start=3000000&rows=100'
  • 42. <?xml version="1.0" encoding="UTF-8"?> <response> <lst name="responseHeader"> <int name="status">0</int> <int name="QTime">9418</int> <lst name="params"> <str name="start">3000000</str> <str name="q">*:*</str> <str name="rows">5</str> </lst> </lst> <result name="response" numFound="3284000" start="3000000"> . . . </result> </response> But I really need all that data <?xml version="1.0" encoding="UTF-8"?> <response> <lst name="error"> <str name="msg">java.lang.OutOfMemoryError: Java heap space</str> <str name="trace">java.lang.RuntimeException: java.lang.OutOfMemoryError: Java heap space at org.apache.solr.servlet.SolrDispatchFilter.sendError(SolrDispatchFilter.java:796) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:448) . . . Caused by: java.lang.OutOfMemoryError: Java heap space . . . </str> <int name="code">500</int> </lst> </response> curl -XGET 'localhost:8983/solr/select?q=*:*&start=3000000&rows=100'
  • 43. But I really need all that data Query
  • 44. But I really need all that data
  • 45. But I really need all that data
  • 46. But I really need all that data Response
  • 47. Use the scroll Luke curl -XGET 'localhost:8983/solr/select?q=*:*&cursorMark=*&sort=score+desc,id+desc'
  • 48. Use the scroll Luke curl -XGET 'localhost:8983/solr/select?q=*:*&cursorMark=*&sort=score+desc,id+desc' <?xml version="1.0" encoding="UTF-8"?> <response> <lst name="responseHeader"> <int name="status">0</int> <int name="QTime">189</int> <lst name="params"> <str name="sort">score desc,id desc</str> <str name="q">*:*</str> <str name="cursorMark">*</str> </lst> </lst> <result name="response" numFound="3284000" start="0"> <doc> ... </doc> . . . </result> <str name="nextCursorMark">AoIIP4AAACY5OTk5OTA=</str> </response>
  • 49. Use the scroll Luke curl -XGET 'localhost:8983/solr/select?q=*:*&sort=score+desc,id+desc &cursorMark=AoIIP4AAACY5OTk5OTA='
  • 50. Use the scroll Luke curl -XGET 'localhost:8983/solr/select?q=*:*&sort=score+desc,id+desc &cursorMark=AoIIP4AAACY5OTk5OTA=' <?xml version="1.0" encoding="UTF-8"?> <response> <lst name="responseHeader"> <int name="status">0</int> <int name="QTime">184</int> <lst name="params"> <str name="sort">score desc,id desc</str> <str name="q">*:*</str> <str name="cursorMark">AoIIP4AAACY5OTk5OTA=</str> </lst> </lst> <result name="response" numFound="3284000" start="0"> <doc> ... </doc> . . . </result> <str name="nextCursorMark">AoIIP4AAACY5OTk5ODE=</str> </response>
  • 51. Limiting faceting, why bother? curl -XGET 'localhost:8983/solr/select?q=*:*&facet=true&facet.field=tag&… facet.limit=-1&facet.mincount=0'
  • 52. Limiting faceting, why bother? curl -XGET 'localhost:8983/solr/select?q=*:*&facet=true&facet.field=tag&… facet.limit=-1&facet.mincount=0' <?xml version="1.0" encoding="UTF-8"?> <response> <lst name="responseHeader"> <int name="status">0</int> <int name="QTime">9967</int> <lst name="params"> ... </lst> </lst> <result name="response" numFound="3284000" start="0"> . . . </result> <lst name="facet_counts"> <lst name="facet_fields"> <lst name="tag"> ... </lst> </lst> </lst> </response>
  • 53. Limiting faceting, why bother? curl -XGET 'localhost:8983/solr/select?q=*:*&facet=true&facet.field=tag&… facet.limit=-1&facet.mincount=0' <?xml version="1.0" encoding="UTF-8"?> <response> . . . <lst name="error"> <str name="msg">Error while processing facet fields: java.lang.OutOfMemoryError: Java heap space</str> <str name="trace">org.apache.solr.common.SolrException: Error while processing facet fields: java.lang.OutOfMemoryError: Java heap space . . . Caused by: java.lang.OutOfMemoryError: Java heap space at org.apache.solr.request.SimpleFacets.getFieldCacheCounts(SimpleFacets.java:685) . . . </str> <int name="code">500</int> </lst> </response>
  • 54. Now let’s look at performance
  • 55. Now let’s look at performance
  • 56. Now let’s look at performance
  • 57. Now let’s look at performance
  • 58. Now let’s look at performance
  • 59. Magic happens with small changes curl -XGET 'localhost:8983/solr/select?q=*:*&facet=true&facet.field=tag&… facet.limit=100&facet.mincount=1'
  • 60. Magic happens with small changes
  • 61. Magic happens with small changes
  • 62. Magic happens with small changes
  • 63. Magic happens with small changes
  • 64. Magic happens with small changes
  • 65. Magic happens with small changes
  • 66. Magic happens with small changes
  • 70. We are hiring! Dig Search? Dig Analytics? Dig Big Data? Dig Performance? Dig Logging? Dig working with and in open – source? We’re hiring world – wide! http://sematext.com/about/jobs.html