SF ElasticSearch Meetup 2012.10.03
Download as PPTX, PDF1 like813 views
The document discusses the importance of a search engine for managing over 600,000 publishers, detailing the processes of monitoring, index building, and optimizing query performance. Key insights include the accelerated index building time from one week to three hours using mappers and the implementation of a bulk API for performance efficiency. The document also highlights learning outcomes related to query performance, including the benefits of utilizing replicas, refreshing indexes, and optimizing API usage.
SF ElasticSearch Meetup 2012.10.03
- 1. Scaling ElasticSearch SF Meetup 2012.10.03 Sushant Shankar sushant.shankar@33across.com
- 2. Agenda • Why we need a search engine • Monitoring • Index Building • Query Performance
- 3. Who is asdfas >600,000 Publishers Machine Learning and Graph algorithms to: - Build advertising segments - Extract insights out of social and interest data - Target via high-performance distributed systems that integrate with our advertising partners Website | Facebook | Twitter
- 4. Why we really need a search engine Batch! Good for complicated tasks (Machine Learning, Graph Algorithms, etc.) … …
- 5. INDEX BUILDING 1 WEEK → 3 HOURS
- 6. Mappers to build index 6 nodes, 24GB RAM 16GB for ES service 4 cores 3x 1.5TB drive >1TB/index Build index (replicated) using MR job ~300M documents and Bulk API ~5KB / document ~3 hours
- 9. Parameter Optimization Amount bulk indexed Time taken CPU util. Mem util. Disk I/O Network # Shards
- 10. Index Building: Learnings • Bulk API • No replicas • 2 shards / CPU • 10,000 documents (users) per indexing request • Refresh off (index.refresh_interval = -1)
- 11. QUERY PERFORMANCE 5 MINUTES 10 SECONDS
- 12. Query Performance: Learnings • 1-2 Replicas (and for reliability) • Turn refresh on again (5s default) • Warm up effect (Index Warm up API 0.20+) • Optimize API • Simulate multiple users
- 13. Warm Up: load into memory and cache
- 14. Other cool features • Custom Scoring functions • Scripts – MVEL, Python • Facets • Exploring: • Real-time indexing • Indexing images, files, etc. • Parent-child relationships
- 15. QUERIES?
- 16. Index Building over time
Editor's Notes
- #5: Collect information over 1B users internationally – text copied from over 600K publisher sites, images, searches, pages visitedDifferent slices of data – now!