SlideShare a Scribd company logo
2
Most read
18
Most read
19
Most read
Solr on Docker - the Good, the Bad and the Ugly
Radu Gheorghe
Sematext Group, Inc.
01
Agenda
The Good (well, arguably). Why containers? Orchestration, configuration drift...
The Bad (actually, not so bad). How to do it? Hardware, heap size, shards...
The Ugly (and exciting). Why is it slow/crashing? Container limits, GC&OS settings
01
Clients
Sematext
Cloud
logs
metrics
...
Our own dockerizing (dockerization?)
01
Because Docker is the future!
01
*
* you’re not tied to the provider’s autoscaling
* you may get better deals with huge VMs
Orchestration
01
github.com/sematext/lucene-revolution-samples
Demo: Kubernetes
01
dev=test=prod; infrastructure as code. Sounds familiar? But:
○ light images
○ faster start&stop
○ hype ⇒ community
Efficiency (overhead vs isolation): (processes + VMs)/2 = containers
More on “the Good” of containerization
01
Zookeeper on separate hosts
nodes
Avoid hotspots:
Equal nodes per host
Equal shards per node
(per collection)
podAntiAffinity on k8s
Moving on to “how”
01
Overshard*. A bit.
time
logs1 logs2
logs3
*Moving shards creates load ⇒ be aware of spikes
Time series? Size-based indices
On scaling
01
volumes/StatefulSet for persistence
local > network (esp. for full-text search)
permissions
latency (mostly to Zookeeper)
AWS → enhanced networking
network storage on different interface
AWS → EBS-optimized
01
Not too small
OS caches are shared between containers
⇩
>1 Solr nodes per host?
Co-locate with less IO-intensive apps?
Not too big
Host failure will be really bad
Overhead (e.g. memory allocation)
Big vs small hosts
01
Many small Solr nodes ⇒ bigger cluster state, # of shards
Multithreaded indexing
Full text search is usually bound by IO latency
Facets are usually parallelized between shards/collections
Size usually limited by heap (can’t be too big due to GC)
or by recovery time
bigger = better
Big vs small containers/nodes
01
More data → more heap (terms, docValues, norms…)
Caches (generally, fieldValueCache is evil, use docValues)
Transient memory (serving requests)
→ add 50-100% headroom
Make sure to leave enough room for OS caches
How much heap?
01
@32GB → no more compressed object pointers
Depending on OS, >30GB → still compressed, but not 0-based → more CPU
Uncompressed pointers’ overhead varies on use-case, 5-10% is a good
Larger heaps → GC is a bigger problem
The 32GB heap problem
01
Defaults → should be good up to 30GB
Larger heaps need tuning for latency
100GB+ per node is doable.
CMS: NewRatio, SurvivorRatio, CMSInitiatingOccupancyFraction
G1 trades heap for latency and throughput:
■ Adaptive sizing depending on MaxGCPauseMillis
■ Compacts old gen (check G1HeapRegionSize)
More useful info: https://wiki.apache.org/solr/ShawnHeisey#GC_Tuning_for_Solr
usually jump
to 45GB+
typical cluster killer (timeouts)
GC Settings
01
GC-related
young: ParallelGCThreads
old: ConcGCThreads + G1ConcRefinementThreads
facet.threads
merges*: maxThreadCount & maxMergeCount
* also account for IO throughput&latency
<Java 9 defaults depend on host’s #CPUs
N nodes per host ⇒ threads
01
Memory: more than heap, but won’t include OS caches
CPU
Single NUMA node? --cpu-shares
Multiple NUMA nodes? --cpuset*
vm.zone_reclaim_mode to store caches only on local node?
* Docker isn’t NUMA aware: https://github.com/moby/moby/issues/9777
But kernel automatically balances threads by default
Container limits
01
Memory leak → OOM killer with a wide range of Java versions*
What helps:
Similar leaks (growing RSS) → NativeMemoryTracking
Don’t overbook memory + leave room for OS caches
Allocate on startup via AlwaysPreTouch
Increase vm.min_free_kbytes?
* https://bugs.openjdk.java.net/browse/JDK-8164293
JVM+Docker+Linux = love. Or not.
Newer kernels and Dockers are usually better
Open files and locked memory limits
Check dmesg and kswapd* CPU usage
Dare I say it:
Try smaller hosts
Try niofs? (if you trash the cache - and TLB - too much)
A bit of swap? (swappiness is configurable per container, too)
Play with mmap arenas and THP
01
* kernel’s (single-threaded) GC: https://linux-mm.org/PageOutKswapd
e.g. 4.4+ and 1.13+
More on that love
01
The Good:
Orchestration
Dynamic allocation of resources (works well for bigger boxes)
Might actually deliver the promise of dev=testing=prod, because
The Bad:
Pets → cattle requires good sizing, config, scaling practices
The Ugly:
Ecosystem is still young → exciting bugs
Docker is the future!
Summary
Thank You! And please check out:
Solr&Kubernetes cheatsheets:
sematext.com/resources/#publications
Openings:
sematext.com/jobs
@sematext @radu0gheorgheOur booth :)

More Related Content

PDF
Deploying PostgreSQL on Kubernetes
PPTX
Understanding DPDK
PDF
CloudStack and GitOps at Enterprise Scale - Alex Dometrius, Rene Glover - AT&T
PDF
Intel dpdk Tutorial
PDF
Enabling new protocol processing with DPDK using Dynamic Device Personalization
KEY
Hadoop, Pig, and Twitter (NoSQL East 2009)
PPTX
HBase in Practice
PPTX
Grafana Mimir and VictoriaMetrics_ Performance Tests.pptx
Deploying PostgreSQL on Kubernetes
Understanding DPDK
CloudStack and GitOps at Enterprise Scale - Alex Dometrius, Rene Glover - AT&T
Intel dpdk Tutorial
Enabling new protocol processing with DPDK using Dynamic Device Personalization
Hadoop, Pig, and Twitter (NoSQL East 2009)
HBase in Practice
Grafana Mimir and VictoriaMetrics_ Performance Tests.pptx

What's hot (20)

PDF
Room 3 - 6 - Nguyễn Văn Thắng & Dzung Nguyen - Ứng dụng openzfs làm lưu trữ t...
PDF
Linux Performance Analysis: New Tools and Old Secrets
PPTX
Room 1 - 3 - Lê Anh Tuấn - Build a High Performance Identification at GHTK wi...
PPTX
BTRİSK Web Uygulama Güvenliği Denetimi Eğitimi
PDF
Embedded Android : System Development - Part IV
ODP
Basics of boot-loader
PDF
Tutorial ceph-2
PPTX
Mod security
PDF
Security Monitoring with eBPF
PDF
Low Latency OLAP with Hadoop and HBase
PDF
The Linux Block Layer - Built for Fast Storage
PDF
La investigacion hermeneutica
PDF
Suricata: A Decade Under the Influence (of packet sniffing)
PDF
SANS 20 Kritik Siber Guvenlik Kontrolü
PPTX
Minio Cloud Storage
PDF
Embedded Linux Kernel - Build your custom kernel
PDF
Network Programming: Data Plane Development Kit (DPDK)
PPTX
Embedded Linux/ Debian with ARM64 Platform
PDF
Kafka to the Maxka - (Kafka Performance Tuning)
PPTX
Ceph Performance and Sizing Guide
Room 3 - 6 - Nguyễn Văn Thắng & Dzung Nguyen - Ứng dụng openzfs làm lưu trữ t...
Linux Performance Analysis: New Tools and Old Secrets
Room 1 - 3 - Lê Anh Tuấn - Build a High Performance Identification at GHTK wi...
BTRİSK Web Uygulama Güvenliği Denetimi Eğitimi
Embedded Android : System Development - Part IV
Basics of boot-loader
Tutorial ceph-2
Mod security
Security Monitoring with eBPF
Low Latency OLAP with Hadoop and HBase
The Linux Block Layer - Built for Fast Storage
La investigacion hermeneutica
Suricata: A Decade Under the Influence (of packet sniffing)
SANS 20 Kritik Siber Guvenlik Kontrolü
Minio Cloud Storage
Embedded Linux Kernel - Build your custom kernel
Network Programming: Data Plane Development Kit (DPDK)
Embedded Linux/ Debian with ARM64 Platform
Kafka to the Maxka - (Kafka Performance Tuning)
Ceph Performance and Sizing Guide
Ad

Viewers also liked (6)

PDF
Best practices for highly available and large scale SolrCloud
PPTX
Solr Search Engine: Optimize Is (Not) Bad for You
PDF
Effective Hive Queries
PDF
SolrCloud on Hadoop
PDF
Cross Datacenter Replication in Apache Solr 6
PDF
How to Run Solr on Docker and Why
Best practices for highly available and large scale SolrCloud
Solr Search Engine: Optimize Is (Not) Bad for You
Effective Hive Queries
SolrCloud on Hadoop
Cross Datacenter Replication in Apache Solr 6
How to Run Solr on Docker and Why
Ad

Similar to Solr on Docker - the Good, the Bad and the Ugly (20)

PDF
OOPs, OOMs, oh my! Containerizing JVM apps
PDF
Ceph Performance: Projects Leading Up to Jewel
PDF
Ceph Performance: Projects Leading up to Jewel
PDF
Cassandra Anti-Patterns
PPT
Mysql talk
PDF
IMCSummit 2015 - Day 2 IT Business Track - 4 Myths about In-Memory Databases ...
PDF
VMworld 2013: Just Because You Could, Doesn't Mean You Should: Lessons Learne...
PDF
Scaling Ceph at CERN - Ceph Day Frankfurt
PDF
strangeloop 2012 apache cassandra anti patterns
PDF
SUE 2018 - Migrating a 130TB Cluster from Elasticsearch 2 to 5 in 20 Hours Wi...
PPTX
Cassandra in Operation
PPTX
Sizing MongoDB on AWS with Wired Tiger-Patrick and Vigyan-Final
PDF
LXC Containers and AUFs
PPTX
How swift is your Swift - SD.pptx
PPTX
Hadoop World 2011: Hadoop and Performance - Todd Lipcon & Yanpei Chen, Cloudera
PDF
10 Devops-Friendly Database Must-Haves - Dor Laor, ScyllaDB - DevOpsDays Tel ...
PDF
Scale11x lxc talk
PPTX
Cephfs jewel mds performance benchmark
PPTX
Responding rapidly when you have 100+ GB data sets in Java
PPTX
Spark Tips & Tricks
OOPs, OOMs, oh my! Containerizing JVM apps
Ceph Performance: Projects Leading Up to Jewel
Ceph Performance: Projects Leading up to Jewel
Cassandra Anti-Patterns
Mysql talk
IMCSummit 2015 - Day 2 IT Business Track - 4 Myths about In-Memory Databases ...
VMworld 2013: Just Because You Could, Doesn't Mean You Should: Lessons Learne...
Scaling Ceph at CERN - Ceph Day Frankfurt
strangeloop 2012 apache cassandra anti patterns
SUE 2018 - Migrating a 130TB Cluster from Elasticsearch 2 to 5 in 20 Hours Wi...
Cassandra in Operation
Sizing MongoDB on AWS with Wired Tiger-Patrick and Vigyan-Final
LXC Containers and AUFs
How swift is your Swift - SD.pptx
Hadoop World 2011: Hadoop and Performance - Todd Lipcon & Yanpei Chen, Cloudera
10 Devops-Friendly Database Must-Haves - Dor Laor, ScyllaDB - DevOpsDays Tel ...
Scale11x lxc talk
Cephfs jewel mds performance benchmark
Responding rapidly when you have 100+ GB data sets in Java
Spark Tips & Tricks

More from Sematext Group, Inc. (20)

PDF
Tweaking the Base Score: Lucene/Solr Similarities Explained
PPTX
Is observability good for your brain?
PDF
Introducing log analysis to your organization
PDF
Monitoring and Log Management for
PDF
Introduction to solr
PDF
Building Resilient Log Aggregation Pipeline with Elasticsearch & Kafka
PDF
Elasticsearch for Logs & Metrics - a deep dive
PDF
Tuning Solr & Pipeline for Logs
PPTX
Running High Performance & Fault-tolerant Elasticsearch Clusters on Docker
PDF
Top Node.js Metrics to Watch
PPT
Running High Performance and Fault Tolerant Elasticsearch Clusters on Docker
PDF
Large Scale Log Analytics with Solr (from Lucene Revolution 2015)
PDF
From Zero to Production Hero: Log Analysis with Elasticsearch (from Velocity ...
PDF
Docker Logging Webinar
PDF
Docker Monitoring Webinar
PDF
Metrics, Logs, Transaction Traces, Anomaly Detection at Scale
PDF
Side by Side with Elasticsearch & Solr, Part 2
PPTX
Tuning Elasticsearch Indexing Pipeline for Logs
PDF
Solr Anti Patterns
PDF
Tuning Solr for Logs
Tweaking the Base Score: Lucene/Solr Similarities Explained
Is observability good for your brain?
Introducing log analysis to your organization
Monitoring and Log Management for
Introduction to solr
Building Resilient Log Aggregation Pipeline with Elasticsearch & Kafka
Elasticsearch for Logs & Metrics - a deep dive
Tuning Solr & Pipeline for Logs
Running High Performance & Fault-tolerant Elasticsearch Clusters on Docker
Top Node.js Metrics to Watch
Running High Performance and Fault Tolerant Elasticsearch Clusters on Docker
Large Scale Log Analytics with Solr (from Lucene Revolution 2015)
From Zero to Production Hero: Log Analysis with Elasticsearch (from Velocity ...
Docker Logging Webinar
Docker Monitoring Webinar
Metrics, Logs, Transaction Traces, Anomaly Detection at Scale
Side by Side with Elasticsearch & Solr, Part 2
Tuning Elasticsearch Indexing Pipeline for Logs
Solr Anti Patterns
Tuning Solr for Logs

Recently uploaded (20)

PDF
Spectral efficient network and resource selection model in 5G networks
PPTX
Big Data Technologies - Introduction.pptx
PDF
cuic standard and advanced reporting.pdf
PDF
Reach Out and Touch Someone: Haptics and Empathic Computing
PDF
7 ChatGPT Prompts to Help You Define Your Ideal Customer Profile.pdf
PDF
Review of recent advances in non-invasive hemoglobin estimation
PDF
Mobile App Security Testing_ A Comprehensive Guide.pdf
PPTX
VMware vSphere Foundation How to Sell Presentation-Ver1.4-2-14-2024.pptx
PDF
Electronic commerce courselecture one. Pdf
PPT
“AI and Expert System Decision Support & Business Intelligence Systems”
PDF
Blue Purple Modern Animated Computer Science Presentation.pdf.pdf
PPTX
Effective Security Operations Center (SOC) A Modern, Strategic, and Threat-In...
PDF
GamePlan Trading System Review: Professional Trader's Honest Take
PDF
Chapter 3 Spatial Domain Image Processing.pdf
PDF
Shreyas Phanse Resume: Experienced Backend Engineer | Java • Spring Boot • Ka...
PPTX
Cloud computing and distributed systems.
PDF
NewMind AI Weekly Chronicles - August'25 Week I
PPTX
Understanding_Digital_Forensics_Presentation.pptx
DOCX
The AUB Centre for AI in Media Proposal.docx
PDF
CIFDAQ's Market Insight: SEC Turns Pro Crypto
Spectral efficient network and resource selection model in 5G networks
Big Data Technologies - Introduction.pptx
cuic standard and advanced reporting.pdf
Reach Out and Touch Someone: Haptics and Empathic Computing
7 ChatGPT Prompts to Help You Define Your Ideal Customer Profile.pdf
Review of recent advances in non-invasive hemoglobin estimation
Mobile App Security Testing_ A Comprehensive Guide.pdf
VMware vSphere Foundation How to Sell Presentation-Ver1.4-2-14-2024.pptx
Electronic commerce courselecture one. Pdf
“AI and Expert System Decision Support & Business Intelligence Systems”
Blue Purple Modern Animated Computer Science Presentation.pdf.pdf
Effective Security Operations Center (SOC) A Modern, Strategic, and Threat-In...
GamePlan Trading System Review: Professional Trader's Honest Take
Chapter 3 Spatial Domain Image Processing.pdf
Shreyas Phanse Resume: Experienced Backend Engineer | Java • Spring Boot • Ka...
Cloud computing and distributed systems.
NewMind AI Weekly Chronicles - August'25 Week I
Understanding_Digital_Forensics_Presentation.pptx
The AUB Centre for AI in Media Proposal.docx
CIFDAQ's Market Insight: SEC Turns Pro Crypto

Solr on Docker - the Good, the Bad and the Ugly

  • 1. Solr on Docker - the Good, the Bad and the Ugly Radu Gheorghe Sematext Group, Inc.
  • 2. 01 Agenda The Good (well, arguably). Why containers? Orchestration, configuration drift... The Bad (actually, not so bad). How to do it? Hardware, heap size, shards... The Ugly (and exciting). Why is it slow/crashing? Container limits, GC&OS settings
  • 4. 01 Because Docker is the future!
  • 5. 01 * * you’re not tied to the provider’s autoscaling * you may get better deals with huge VMs Orchestration
  • 7. 01 dev=test=prod; infrastructure as code. Sounds familiar? But: ○ light images ○ faster start&stop ○ hype ⇒ community Efficiency (overhead vs isolation): (processes + VMs)/2 = containers More on “the Good” of containerization
  • 8. 01 Zookeeper on separate hosts nodes Avoid hotspots: Equal nodes per host Equal shards per node (per collection) podAntiAffinity on k8s Moving on to “how”
  • 9. 01 Overshard*. A bit. time logs1 logs2 logs3 *Moving shards creates load ⇒ be aware of spikes Time series? Size-based indices On scaling
  • 10. 01 volumes/StatefulSet for persistence local > network (esp. for full-text search) permissions latency (mostly to Zookeeper) AWS → enhanced networking network storage on different interface AWS → EBS-optimized
  • 11. 01 Not too small OS caches are shared between containers ⇩ >1 Solr nodes per host? Co-locate with less IO-intensive apps? Not too big Host failure will be really bad Overhead (e.g. memory allocation) Big vs small hosts
  • 12. 01 Many small Solr nodes ⇒ bigger cluster state, # of shards Multithreaded indexing Full text search is usually bound by IO latency Facets are usually parallelized between shards/collections Size usually limited by heap (can’t be too big due to GC) or by recovery time bigger = better Big vs small containers/nodes
  • 13. 01 More data → more heap (terms, docValues, norms…) Caches (generally, fieldValueCache is evil, use docValues) Transient memory (serving requests) → add 50-100% headroom Make sure to leave enough room for OS caches How much heap?
  • 14. 01 @32GB → no more compressed object pointers Depending on OS, >30GB → still compressed, but not 0-based → more CPU Uncompressed pointers’ overhead varies on use-case, 5-10% is a good Larger heaps → GC is a bigger problem The 32GB heap problem
  • 15. 01 Defaults → should be good up to 30GB Larger heaps need tuning for latency 100GB+ per node is doable. CMS: NewRatio, SurvivorRatio, CMSInitiatingOccupancyFraction G1 trades heap for latency and throughput: ■ Adaptive sizing depending on MaxGCPauseMillis ■ Compacts old gen (check G1HeapRegionSize) More useful info: https://wiki.apache.org/solr/ShawnHeisey#GC_Tuning_for_Solr usually jump to 45GB+ typical cluster killer (timeouts) GC Settings
  • 16. 01 GC-related young: ParallelGCThreads old: ConcGCThreads + G1ConcRefinementThreads facet.threads merges*: maxThreadCount & maxMergeCount * also account for IO throughput&latency <Java 9 defaults depend on host’s #CPUs N nodes per host ⇒ threads
  • 17. 01 Memory: more than heap, but won’t include OS caches CPU Single NUMA node? --cpu-shares Multiple NUMA nodes? --cpuset* vm.zone_reclaim_mode to store caches only on local node? * Docker isn’t NUMA aware: https://github.com/moby/moby/issues/9777 But kernel automatically balances threads by default Container limits
  • 18. 01 Memory leak → OOM killer with a wide range of Java versions* What helps: Similar leaks (growing RSS) → NativeMemoryTracking Don’t overbook memory + leave room for OS caches Allocate on startup via AlwaysPreTouch Increase vm.min_free_kbytes? * https://bugs.openjdk.java.net/browse/JDK-8164293 JVM+Docker+Linux = love. Or not.
  • 19. Newer kernels and Dockers are usually better Open files and locked memory limits Check dmesg and kswapd* CPU usage Dare I say it: Try smaller hosts Try niofs? (if you trash the cache - and TLB - too much) A bit of swap? (swappiness is configurable per container, too) Play with mmap arenas and THP 01 * kernel’s (single-threaded) GC: https://linux-mm.org/PageOutKswapd e.g. 4.4+ and 1.13+ More on that love
  • 20. 01 The Good: Orchestration Dynamic allocation of resources (works well for bigger boxes) Might actually deliver the promise of dev=testing=prod, because The Bad: Pets → cattle requires good sizing, config, scaling practices The Ugly: Ecosystem is still young → exciting bugs Docker is the future! Summary
  • 21. Thank You! And please check out: Solr&Kubernetes cheatsheets: sematext.com/resources/#publications Openings: sematext.com/jobs @sematext @radu0gheorgheOur booth :)