Solr sparse faceting

Editor's Notes

• #3: Highly scientific. Very happy with Solr faceting for general work. Principles primarily, applicable to non-Solr settings.
• #4: State & University + Royal Library. Harvest 4 times/year. Hardware: Save money. Static shards. Generation 1: 25 shards. Generation 2: Currently 9 shards. We'll use this. Apologies for small sample. 100GB free cache mem (0.5%-1.23% of index size) Danish Net Archive: http://netarkivet.dk/in-english/ Next: We need aggregations, so faceting. String faceting is focus here.
• #5: Bird's eye view of vanilla Solr fc-faceting. 1) Allocate counter 2) Update counter 3) Extract top-X DocValues all the way. Works really well for a few million values. Next: Performance: 200M values.
• #6: Not time series, but bucketed by result size. Most real-world results in the lower end. Non-warmed, random queries, limit=25. 3 concurrent searches. 200M values. Note the high variance for same-size result sets.
• #7: Standard JRE 1.7 garbage collector – no tuning. Full GC means delay for the client. Standard GC means higher CPU load.
• #8: Upon release, the counter is cleared by the pool. Faster than re-allocating. Optionally the clearing is done in the background by a dedicated thread.
• #9: Without and with reuse of counters. Probably possible to reduce full GC on reuse.
• #10: With and without reuse of counters. The last we see of vanilla Solr (the tricks are stacked)
• #11: Quartiles plot. Box is 75%, median and 25%. Lower median with reuse. Smaller variance with reuse. Downside: Static overhead. Next: Why the 500ms lower bound?
• #12: Vanilla Solr extraction of top-X terms scales linear to the number of unique terms. Previous chart: 200M terms = 200M counters. Next: Track the changes.
• #16: No change as ord 3 has already been incremented.
• #18: Problem: Random memory access. Slower beyond ~8% (so turn off automatically).
• #19: Non-sparse & sparse. Downside: 8% overhead. Next: Few Solr's stand alone - SolrCloud.
• #20: With distributed faceting, fine counting is necessary.
• #21: Pit of Pain coined by Andy Jackson, UKWA. 1K-10K very high, due to the highest chance of fine-counting. < 100: Every shard probably returned all know terms. > 100K: Most of the terms in top-X probably returned by most shards. 10M+: True load starts to show (rare). Note: 1M cardinality field. We count just skip finecounting, but we don't want to cheat that much. Next: What to do about it?
• #22: Simple strategy: As phase 1 was fast, just repeat it and lookup the specific terms.
• #23: Since we already did the counting in phase 1, we'll just re-use the filled counter structure. Cleanup problem handled by background thread.
• #24: Yes, re-use and sparse multipliers does stack. Downside: Not much. Maybe large result sets? Next: From performance to memory: Counter representation.
• #25: A typical shard from Net Archive Search v2. References are from documents to terms. High-cardinality in different ways. We want to facet on url and links to visualize connectivity hotspots. Next: All counters are not equal.
• #26: Common theme: Many with max count 1.
• #27: We flip the visualization. Squares are counter-bits needed.
• #28: int[ordinals] uses 4 bytes/counter. PackedInts uses as many bits per value as is needed to represent the highest value of any counter.This works well for fields with a low max count (such as url), but not so well for fields with a high max count (domain and links). Note: The int[ordinals] is not to scale. It really goes up to 32.
• #29: Black squares are counter value bits. Blue squares signify if the counter it belongs to is so large that it continues on the next plane. All previous blue squares must be counted. This requires a fast rank function to work (3% overhead). Creation is slow! Full term-iteration. Amortized plane-accesses for increments in n-plane is 2 (1 + 50% chance of p2 + 25% chance of p3 + ...). Each plane-shift requires rank lookup and overflow-bit lookup: 2 memory reads Details at https://sbdevel.wordpress.com/2015/03/13/n-plane-packed-counters-for-faceting/
• #30: Start with 0.
• #31: Set first bit. No overflow -> no problem.
• #32: Plane 1 overflows for L. Count blue overflow bits: There are 5. At position 5 in Plane 2, update the bits.
• #36: This works very well as most counters have a max of 1.
• #37: This works very well as most counters have a max of 1.
• #38: This works very well as most counters have a max of 1.
• #39: This works very well as most counters have a max of 1.
• #40: This works very well as most counters have a max of 1.
• #41: This works very well as most counters have a max of 1.
• #42: N-plane-z instantiations beyond the first re-uses the overflow bits. Lowest possible: 0.7MB (15%), 27.9MB (4%), 132.3MB (6%)
• #43: Single shard. 640M values. Obvious speed degradation that follows space savings. Insignificant with small result sets due to sparse. TODO: Worst case (*:*) array: packed: nplanez Next: Scale it up.
• #44: I can't see how this could possibly go wrong.
• #45: Worst case: About 50 billion references to 9 billion values must be processed. (10 minutes) Two strategies presents themselves: Threading for exact counts and sampling. Threading is currently implemented.
• #46: Screenshot taken immediately after kitchen sink test.
• #47: Available on GitHub. Currently Solr 4.8, promise to go for 4.10.4 Real Soon Now. 5.x later.