Thoth - Realtime Solr Monitor and Search Analysis Engine
Download as PPTX, PDF2 likes2,263 views
Thoth is an innovative project at Trulia designed to monitor and analyze search infrastructure using a modular system that collects and indexes search data in near real-time via Solr. It features a dashboard for visual insights, and employs machine learning for query time prediction and pattern recognition. The platform addresses performance issues and facilitates troubleshooting without directly accessing logs.
![Thoth Monitor
- Continuously monitoring for metrics
- Stateless
- Alerting through email or Nagios
- Examples: QTime, Number of Zero hits,
Predictor Model Health
- Possibility to implement custom monitors
- Reuse StatsComponent
[http://wiki.apache.org/solr/StatsComponent]
if possible](https://image.slidesharecdn.com/thothrealtimesolrmonitorandsearchanalysisengine-141114112830-conversion-gate01/85/Thoth-Realtime-Solr-Monitor-and-Search-Analysis-Engine-17-320.jpg)
Thoth - Realtime Solr Monitor and Search Analysis Engine
- 2. Thoth Real-time Solr Monitor Search Analysis Engine dbraga@trulia.com pmhatre@trulia.com Damiano Braga Sr. Software Engineer Praneet Mhatre Data Mining Engineer
- 3. Overview - What is Thoth ? - Data Collection and Thoth Core Indexing - Thoth API & Thoth Dashboard - Thoth Monitor - Thoth ML : Prediction and Topic Modeling - Special Thanks & Q/A Demo
- 4. What is Thoth? - Innovation project at Trulia - Understand our search infrastructure without touching logs - Troubleshoot search performance issues - Designed as a modular system - Set of tools that can help gather info, monitor, understand a search infrastructure - Open source project : thoth thoth-ml thoth-api thoth-dashboard thoth-monitor thoth-demo
- 5. Problem: Know Your Search Infrastructure - Solr logs are a good source. Sometimes partial information - Decentralized data (at least 1 log per search server) - Log rotation - Not searchable If we could index all the information .. Let’s use Solr ! - We can search on it - We have some handy features for free: facets, stats etc - It’s scalable
- 6. Thoth Document 1 Solr Request = 1 Thoth (Solr) Document Server Info hostname, port number, core name, pool name Query Info timestamp, actual query, qtime, hits, exception?
- 7. Data Collection (1/2) - Should be smooth. No traffic slowing down. - We care about near real-time data - We care about historical data - Dataset is growing fast - Interceptor on each search server - We use a SolrComponent attached to a Request Handler - Queue System (E.g: ActiveMQ) to facilitate and temporary store messages - Each search server has a manifest in the solrconfig.xml
- 8. Data Collection (2/2) <requestHandler name="select" class="com.solr2activemq.SolrToActiveMQHandler”> <arr name="last-components”> <str>solr2activemq</str> </arr> </requestHandler> <searchComponent name="solr2activemq” class="com.solr2activemq.SolrToActiveMQComponent" > <str name="activemq-broker-uri">localhost</str> <int name="activemq-broker-port">61616</int> <str name="activemq-broker-destination-type">queue</str> <str name="activemq-broker-destination-name">test-queue</str> <str name="solr-hostname">localhost</str> <int name="solr-port">8983</int> <str name="solr-poolname">default</str> <str name="solr-corename">collection</str> <int name="solr2activemq-buffer-size">1000</int> <int name="solr2activemq-dequeuing-buffer-polling">500</int> <int name="solr2activemq-check-activemq-polling">5000</int> </searchComponent>
- 9. Sizing of Data - Need for granular information for near real-time data - Less granularity for historical data Too much data = slow search, space problem - Shrinking feature: - Create Shrank Document - Real-time Core cleanup - Shrinking time is configurable
- 10. Thoth Index - Solr 4.7 - Soft commit for near real-time search - Soft commit maxTime set to 1s - Auto commit set to 15s - Update chain set to enforce UUID as PkID - Use of Solrj to index data and query
- 11. Thoth API - Abstraction for Thoth index and Thoth data - Read only REST-like API - JSON response - Written in Node.js to accommodate socket.io Example: {"numFound":95,"values":[{"timestamp":"2014-09- 16T18:00:02Z","value":45337},{"timestamp":"2014-09- 16T18:15:02Z","value":77325},{"timestamp":"2014-09- 16T18:30:02Z","value":109523},{"timestamp":"2014-09- 16T18:45:02Z","value":112279},{"timestamp":"2014-09- 16T19:00:02Z","value":115334} thoth:3001/api/server/foo/core/bar/port/portbar/start/NOW-1DAY/end/NOW/count/nqueries
- 12. Thoth Dashboard (1/5) - Visual insight on Thoth data - Useful graphs divided by server or pool - Handy list of slow queries and exceptions - Real-time view for server - Selecting data based on time - Sharable URLs (to OPS team, QA team, Release Eng. )
- 17. Thoth Monitor - Continuously monitoring for metrics - Stateless - Alerting through email or Nagios - Examples: QTime, Number of Zero hits, Predictor Model Health - Possibility to implement custom monitors - Reuse StatsComponent [http://wiki.apache.org/solr/StatsComponent] if possible
- 18. Thoth ML What can we do with all this data? • Rich source of information • Can we turn it into knowledge? • How about machine learning? 1. Query time prediction 2. Query pattern recognition 3. Server sizing and resource allocation
- 19. 1. Query Time Prediction (1/4) • Goal : appropriately route queries to slow/ fast pool • Look at query attributes • Query text • Start parameter • Facets, range queries, geo spatial searches etc • Train a supervised learning model • Use learned model to predict if a query will be slow v/s fast • H2O Machine Learning Library
- 20. 1. Query Time Prediction (2/4) Challenges • Imbalanced dataset • Frequency of model training • Type of model • Minimal delay requirement
- 21. 1. Query Time Prediction (3/4) Challenges Addressed • Imbalanced dataset • Stratified sampling • Frequency of model training • Auto identify relearning frequency • Type of model • Boolean, categorical features -> Tree based • High accuracy • Gradient Boosted Machine • Minimal delay requirement • User pool queries: 45-50 ms • Prediction: 1-3 ms
- 22. 1. Query Time Prediction (4/4) • 1000 Gradient Boosted Trees • Slow queries = (>100ms. Configurable) • Experimental Results • Training on ~3.1 million • Test on ~1.4 million • AUC: 0.94542 • Accuracy: 0.9202223
- 23. Query Time Prediction in Action (1/2) Performance on real time traffic at Trulia
- 24. Query Time Prediction in Action (2/2) Performance on real time traffic at Trulia
- 25. 2. Query Pattern Recognition • Exceptions, zero hit queries • Analyze and find out why • Probabilistic Topic Modeling • Using MALLET open source toolkit
- 28. Future Direction - Thoth ML improvements: • Predicting query time buckets • Regression v/s classification • Exceptions and zero hit query analysis • Sizing and resource allocation - Solr Cloud integration - Dashboard integration with Solr cloud - More standard metrics on Thoth Monitor - More data collection (load, GC)
- 29. Contributors and Special Thanks Damiano : dbraga@trulia.com Praneet: pmhatre@trulia.com Fork us on Github! github.com/trulia/thoth JD Cantrell ( API, Dashboard) Giulio Grillanda (API, Dashboard) Rajendra Shioramwar (Core) Ying Wang (Design) Girish Gudla (Monitor) Alexander Kanarsky Alex Burmester