High Performance Solr and JVM Tuning Strategies used for MapQuest’s Search Ahead: Presented by Darren Spehr, MapQuest
20 likes13,158 views
MapQuest developed a search ahead feature for their mobile app to enable auto-complete searching across their large dataset. They used Solr and implemented various techniques to optimize performance, including custom routing, analysis during ETL, and extensive JVM tuning. Their architecture included multiple Solr clusters with different configurations. Through testing and monitoring, they were able to meet their sub-140ms response time requirement for queries.
High Performance Solr and JVM Tuning Strategies used for MapQuest’s Search Ahead: Presented by Darren Spehr, MapQuest
- 2. High Performance Solr and JVM Tuning Strategies used for MapQuest’s Search Ahead Darren Spehr – System Architect
- 3. MapQuest Going strong … since 1967 • Maps • Directions • Routing • Geocoding • Mobile • B2B
- 4. Every Adventure has a Beginning Our mobile client needs an overhaul … Oh, and we need an auto-correct feature … well, auto-complete … actually, search ahead
- 5. … Top Secret Meeting Minutes • How do we use auto-‐complete today? • What are we searching over? • How fast can a person type? • What are we going to say in response? • When do we have to launch this?
- 6. Characteristics • Searches march from le; to right • Expect the first term to be highly relevant • Term order and proximity are clues • Spaces are now really important • Expect mixed query types • AbbreviaCons and misspellings are common • People can type really fast (but generally less than 10 keystrokes per sec) • Users frequently want to browse
- 7. Requirements Fast, Like really fast 140 milliseconds maximum response time
- 8. Methodology Some opCmizaCons can be planned Others need to be discovered Test alternaCves – opCmize low hanging fruit early Finally: Take it to task
- 10. The Data: Categories Franchises Locations • Neighborhoods to Countries Points of Interest • Airports • Businesses • Landmarks Addresses • Individual • Block (Interpolated) In all – over 10 Billion unique documents
- 11. Architecture Solr API Clusters Mobile Client Mobile App API-‐East Targeted LocaCon Business Address 1 Address 2 API-‐West Targeted: 4 VMs 1 shard, 283,000 docs Frequent Low Volume Updates Loca7on: 3 VMs 1 shard, 4.3 million docs Frequent Low Volume Updates Business: 5 VMs 1 shard, 13.4 million docs Heavy Updates Address: 30 VMs 10 shards, 100 million docs No Updates Interpolated Address: 30 VMs 10 shards, 10 billion docs* No Updates
- 12. Special Cases Business data Ø Complex synonyms Ø Stemming needs Ø The memory factor Ø Complex query patterns Addresses Ø So many! Ø Nested structure Ø Interpolated positions Ø Updates an issue Airports Ø Airport codes Ø International issues Locations Ø International issues Ø Relevance
- 13. Move Analysis to the ETL A typical job includes: • Basic text processing / cleansing • Stemming • Synonyms and subsCtuCon • Cloning • Filtering • Various permutaCons • RegionalizaCon • Pre-‐calculaCng relevance
- 14. Custom Doc Routing Address data won’t fit in memory or perform well … Both collections are sharded so the size on disk is around 6-8 GB Initial, naïve balancing wasn’t nearly good enough Optimization problem that accounts for: - Size on disk - Predicted query volumes - FST load (entropy)
- 15. Setting Up the Indexes Clean up schema.xml and solrconfig.xml Exact and Fuzzy queries tested – String fields WIN! (Thank you FST and prefix queries!) Geo-sensitivity made easy using Spatial4J (Thank you David Smiley!) Optimization required No NRT functionality needed
- 16. Query-Time Considerations Jetty -‐ <New class=“java.uCl.concurrent.ArrayBlockingQueue”> -‐ Limit thread pool based on projected need Filters used judiciously Pull in a single field from the indexes for display. Shard/route aware clients used for Addresses Estimate caching needs
- 17. The API has to be Fast Too Pool as many resources as it makes sense A Note on connection pools: - The DefaultHttpClient avoids key registry overhead - Ask for keep-alive support - Balance pool according to use Thread level caching used to avoid ClassLoader overhead Take out some insurance with TTLs Solr Query HfpClient Executors
- 18. Keep Queries Simple Federate a larger number of queries Break queries out by type and expectation Use custom search handlers to move the burden of “tough” queries to Solr Special case: Ø Interpolated Addresses Ø Business Names Collec7on Query Count Category 3 Franchise 3 Airport 1 CriCcal Address 1 LocaCons 4 Businesses 3 Addresses (both) 2 each
- 19. At this point the service is up and running … but the fun has only begun
- 20. Getting Ready to Test Choose your tool set … Ø Test Suite (JMeter) Ø Application Monitoring (VisualVM) Ø GC Monitoring (VisualGC) Ø On Host tools (top, pidstat) Ø Runtime exposure (JMX, jsvc) Ø Offline analysis (JMeter, GCHisto)
- 21. Set Boundary Conditions Production Query Volume • What is the expected peak QPS • Estimate 50th, 75th percentiles Know what success looks like: • What availability are you looking for? • What about latency? • Caching success? Know what failure looks like: • When do you consider a machine maxed out?
- 22. So, Let’s Talk About …
- 23. Memory Settings Max Heap = anticipated index size in memory + delta for new gen Min Heap = Max Heap to limit HotSpot optimizations -Xmx = -Xms Sizing the new generation (-Xmn): Ø Start with around 1/3 of your heap size Ø Set the Survivor space (-XX:SurvivorRatio=15) Determine the Eden Space: eden = -Xmn - 2 * ( -Xmn / 15 )
- 24. Example 7 GB Index + 3 GB for new generation: -Xms10G -Xmx10G -Xmn3G -XX:SurvivorRatio=15 -XX:PermSize=64m -XX:MaxPermSize=64m Survivor Size: 3 GB / 15 = 205 MB Eden Space: 3 GB – 2 * 205 MB = 2.7 GB
- 25. Baseline JVM Settings Simple and verbose -verbose:gc -XX:HeapDumpPath=/logs/solr_heap.hprof -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCApplicationStoppedTime -Xloggc:/logs/gc.log -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled
- 26. Other Settings We Use TargetSurvivorRatio=70 MaxTenuringThreshold=5 PretenureSizeThreshold=64m CMSFullGCsBeforeCompaction=1 CMSInitiatingOccupancyFraction=70 CMSTriggerPermRatio=80 CMSMaxAbortablePrecleanTime=6000 +CMSScavengeBeforeRemark +UseCMSInitiatingOccupancyOnly +CMSParallelRemarkEnabled +ParallelRefProcEnabled
- 27. Establish Single Test Thread Settings <ConstantThroughputTimer guiclass="TestBeanGUI" testclass="ConstantThroughputTimer" testname="Constant Throughput Timer" enabled="true"> <intProp name="calcMode">0</intProp> <doubleProp> <name>throughput</name> <value>1600.0</value> <savedValue>0.0</savedValue> </doubleProp> </ConstantThroughputTimer>
- 28. Test Cycle Monitor Record Evaluate ? JVM Page Faults CPU GC Rates Threading Context Switches Locks Swapping Network Traffic Availability Throughput Latency Thread Count Have I met my exit condiCons? Add More Threads
- 29. Monitoring the JVM Watch your application come to life! Memory Steady States: • Old GeneraCon: ⅓ to ¼ the size of your sepngs • Permanent GeneraCon: ½ its size Tenure histogram sizes should drop off … this is your ideal level 14000 12000 10000 8000 6000 4000 2000 0 Tenure Size 1 2 3 4 5 Tenure Size
- 30. Monitoring Solr Caches The UI is a wealth of information! Cache Strategy Ø Size Ø Type Look at the hit and eviction statistics Use “binary sizing” to walk the sizes up until there are diminishing returns <filterCache class="solr.LRUCache" size="8384" initialSize="8384" autowarmCount="0"/> <documentCache class="solr.LRUCache" size="8384" initialSize="8384" autowarmCount="0"/>
- 31. JVM Tuning Strategies Smaller eden spaces result in: Ø more frequent minor GCs Ø a higher probability of premature promotion Ø the best performance Watch out for too much eager promotion and lengthening major GCs Mitigate major GC STW pauses by: Ø Keeping the old generation as small as possible Ø Maybe even a little smaller Ø Turn off swapping Ø Consider explicit GC
- 32. Bookkeeping Demo … VisualVM GCHisto Performance Stats
- 33. Planning for the Future What we used to do predictive expansion: 1) Target max VM capacity 2) Matching QPS 3) Breakdown of traffic load 4) Scaling factor
- 34. Conclusions 7 Habits of Highly Effective Tuners 1. Know where you’re going 2. Know where you’re starting from 3. Test incrementally 4. Monitor with intent 5. Make small changes 6. Know when to stop 7. Plan ahead
- 35. Questions? Darren Spehr darren.spehr@mapquest.com
- 36. Resources VisualVM VisualGC GCHisto Java Performance – Hunt and John The Garbage Collection Handbook – Jones, Hosking and Moss Solr In Action – Grainger and Potter