Never Stop Exploring - Pushing the Limits of Solr: Presented by Anirudha Jadhav, Bloomberg L.P.
5 likes6,968 views
This document discusses optimizing Solr for near real-time indexing of large datasets. The author describes benchmarking different indexing configurations, finding that batching documents by time, size or number provides much higher indexing throughput than single documents. The author proposes a PID controller to dynamically adjust batching parameters based on indexing performance. Future work includes refining the PID controller, integrating it with benchmarking tools, and using it for hardware sizing.
![Indexing Workflow
We were talking about IBM during the fishing trip
Down
Cas)ng
[We] [were] [talking] [about] [IBM] [during] [the] [fishing] [trip]
Creates
tokens
by
lowercasing
all
le4ers
and
dropping
non-Ââle4ers.
[we] [were] [talking] [about] [ibm] [during] [the] [fishing] [trip]
[we] [were] [talking] [about] [ibm] [during] [the] [fishing] [trip]
[talk] [fish]
[talking] [about] [ibm] [fishing] [trip]
[talk] [big] [blue] [fish] [journey]
[chat]
Consider
the
sentence:
[we] [were] [talking] [about] [ibm] [during] [the] [fishing] [trip]
[talk] [fish]
Tokeniza)on
A
tokenizer
splits
the
stream
of
characters
into
a
series
of
tokens.
Stemming
Lemma)za)on
Stemming
algorithms
reduce
words
"fishing",
"fished",
"fish",
and
"fisher"
to
the
root
word,
"fish"
Lemma*za*on
expands
words
to
their
inflected
forms
(ie
fishing
-Ââ>
fished,
fishes,
fish
but
not
fisher)
Stop
Word
Removal
Remove
common
stop
words
âandâ,âorâ
etc.
which
introduce
noise
in
the
search
process
Synonym
Expansion
Mapping
of
words
based
upon
thesaurus
(synonyms,
acronyms,
hypernyms,
business
rules,
etc..)
For
example
talk
-Ââ>
chat,
IBM
-Ââ>
âbig
blueâ,
trip
-Ââ>
journey](https://image.slidesharecdn.com/neverstopexploring-pushingthelimitsofsolr-anirudhajadhav-141121125052-conversion-gate02/85/Never-Stop-Exploring-Pushing-the-Limits-of-Solr-Presented-by-Anirudha-Jadhav-Bloomberg-L-P-11-320.jpg)
Never Stop Exploring - Pushing the Limits of Solr: Presented by Anirudha Jadhav, Bloomberg L.P.
- 2. Never Stop Exploring: Pushing the Limits of Solr Anirudha Jadhav ©2014 Bloomberg L.P.
- 3. Who am I ? âą Big Search and Distributed database specialist âą Built a Search as a Service platform âą Lead Search Architect @ Bloomberg Vault âą Credit Derivatives Analytics Engineer @ Bloomberg âą Masters' @ Courant Institute of Mathematical Sciences, New York University âą Passionate about Search, Scuba Diving , Motorcycles and German Shepherds
- 5. Agenda âą Search at Bloomberg âą Goals and Objec5ves âą A li9le background âą Factors affec5ng indexing âą Our tests and benchmarks âą Design for a be9er NRT indexer âą Future work âą Q/A
- 7. Search at Bloomberg âą News Search âą Federated Search âą Complex re-ranking of search results âą Archival Search âą GeoSpatial Search âą Analytics and Statistics on Search
- 8. Objective Significantly increase Near Real Time (NRT) indexing throughput Eg. Building a Search application that receives market data
- 10. Indexing Data Flow in SolrCloud
- 11. Indexing Workflow We were talking about IBM during the fishing trip Down Cas)ng [We] [were] [talking] [about] [IBM] [during] [the] [fishing] [trip] Creates tokens by lowercasing all le4ers and dropping non-Ââle4ers. [we] [were] [talking] [about] [ibm] [during] [the] [fishing] [trip] [we] [were] [talking] [about] [ibm] [during] [the] [fishing] [trip] [talk] [fish] [talking] [about] [ibm] [fishing] [trip] [talk] [big] [blue] [fish] [journey] [chat] Consider the sentence: [we] [were] [talking] [about] [ibm] [during] [the] [fishing] [trip] [talk] [fish] Tokeniza)on A tokenizer splits the stream of characters into a series of tokens. Stemming Lemma)za)on Stemming algorithms reduce words "fishing", "fished", "fish", and "fisher" to the root word, "fish" Lemma*za*on expands words to their inflected forms (ie fishing -Ââ> fished, fishes, fish but not fisher) Stop Word Removal Remove common stop words âandâ,âorâ etc. which introduce noise in the search process Synonym Expansion Mapping of words based upon thesaurus (synonyms, acronyms, hypernyms, business rules, etc..) For example talk -Ââ> chat, IBM -Ââ> âbig blueâ, trip -Ââ> journey
- 12. Designing the Search Index Designing a good Search Applica)on also involves many aspects of user interac)on that directly influence indexing design âą Data Type and Data Distribu)on âą Server side parameters âą Networking âą Client side parameters âą Query pa4erns
- 14. Data and Distribution of Tokens Common types of data that we index in a search index âą Textual data ( human generated ) e.g. messages, news, blogs âą Textual data ( machine generated ) e.g. logs , 5ckets âą Numerical data âą Geospa5al data How does this affect search index designs ? âą Query speed and indexing speed depend on the size of an index âą Size is dependent on âą Number of documents in the index âą Average size of each document âą Distribu5on of tokens âą Index features eg. Face5ng, Highligh5ng
- 15. Server-side Factors âą Ratio of CPUâs to the number of solr cores running âą 2 Solr indices per CPU or a Thread âą Disk space âą Disk space for Solr index * 2 ( head room for merge cycles ) âą Memory âą JVM heap âą Off Heap âą DocValues
- 16. Networking Cluster design consideration âą Should a cluster span data centers ? âą Latency between datacenters âą Reliability and availability SLAâs âą Where does your Zookeeper ensemble live ? âą How many elec5on members âą Consider observers to scale zookeeper âą Dynamically promote an observer to elec5on member Manage concurrent connections on the server Monitor network latencies for QoS guarantees
- 17. Client-side Factors âą Managing connections and reusing connections âą Which format to use for indexing data âą javabin âą csv âą json âą xml âą How many simultaneous threads to use
- 18. Experiments with NRT Indexing Itâs not always efficient to send a single document to Solr for indexing How do you decide how many documents to send ? Collector : A buffer that collects Solr update documents âą Time Triggers ( T ) âą Time based collector on the client-Ââside to batch document payloads to Solr âą Document Size Triggers ( S ) âą Document size based collector on the client-Ââside to batch document payloads to Solr âą Document Number Triggers ( N ) âą Number of documents based collector on the client-Ââside to batch document payloads to Solr The collectors are all simultaneously used in order of priority. The lower priority collectors act as a cut-Ââoff backups to safe guard from overflows.
- 20. Benchmarking Setup âą Client application sending data to 4-way replicated SolrCloud âą 5 node Zookeeper ensemble âą All tests done with a similar dataset ( machine generated text ) âą We synthesize a high throughput ingest stream, which serves as our input âą Soft commits set at 1sec
- 21. Benchmarking : Time Limit Tests docs/sec Time Triggers: Collection window in ms
- 22. Benchmarking : Document Limit Tests docs/sec Document Number Triggers: Collection window in number of documents
- 23. Benchmarking : Byte Limit Tests docs/sec Document Size Triggers: Collection window in bytes
- 24. Observations âą On an average we were able to observe 5x-7x increase in ingestion throughput âą Optimization parameters are dependent constantly changing factors âą The tuning variables need to be constantly adjusted for best performance âą How to use this now
- 25. Design for a better NRT indexer
- 26. PID Controller Proportional term ( P ) â present Output proportional to current error value Integral term ( I ) - past Sum of instantaneous error over time, and give accumulated offset that should have been corrected previously Derivative term ( D ) - future Calculated by determining the slope of previous error over time times the rate of change
- 27. PID implementation in the indexer Solr Cloud Solr response Sampling thread Process variable Docs/sec Client indexer process Pick one of the Triggers Time (T ) Control Variable PID controller implementa5on Indexing threads
- 28. Future Work
- 29. Future work âą Perfect the PID indexer âą Add it to the YCSB benchmarking framework âą Add other server side parameters on the PID indexer âą Use the PID indexer along with the YCSB framework to size hardware
- 30. Never Stop Exploring: Pushing the Limits of Solr Anirudha Jadhav , Bloomberg LP QUESTIONS ?