Building Real-Time Search at MailChimp
1 like269 views
The document presents a technical overview of building a real-time search application at Mailchimp, detailing the challenges of querying and indexing search data. It explores the potential use of Elasticsearch, Kafka for event streaming, and the indexing process for various data models. The results highlight improvements in query response times and a successful implementation without support tickets post-launch.
Building Real-Time Search at MailChimp
- 1. Building Real-Time Search at Mailchimp Kevin Xu Software Engineer
- 3. 2018 JAN ‘17 OCT ‘17
- 4. 2018 MC Technical Overview US1 - US19
- 5. 2018 SEARCH APP (downstream) search queries indexing requests (every 5 min) SQLite Search
- 6. 2018 SEARCH APP Problem: Querying search queries
- 7. 2018 SEARCH APP Problem: Indexing indexing requests (every ???)
- 8. 2018 SEARCH APP search queries indexing requests (every ???) SQLite Search
- 9. 2018 Considering a Replacement - Use cases: full-text search, logging/log analysis, events and metrics - Fast queries - Scale horizontally
- 11. 2018 Querying ES
- 12. 2018 Capturing Events from MC US1 US19
- 13. 2018 Possible Solution: Direct Indexing?
- 14. 2018 Adding a Message Queue
- 16. Proprietary & Confidential 2018 16 Kafka: Topics
- 20. 2018 App-Layer Filtering config.searchable_model: “Contact”; config.searchable_model: “Campaign”; ... $contact = new Contact(“Ben”); $contact->save(); // onSave()
- 21. 2018 Write to File, Ship to Kafka
- 22. 2018 Indexing to ES PHP Consumers ?
- 25. 2018 No Order, No Problem
- 28. 2018 Queries > 1s 9/26/17 10/10/17 NumberofQueries release!
- 29. 2018 release! Queries > 2sNumberofQueries 9/26/17 10/10/17
- 30. 2018 30 250msMedian (p50) Query Response Time
- 31. 2018 31 400msp95 Query Response Time
- 32. 2018 32 19Number of Elasticsearch clusters
- 33. 2018 33 373 billion Total docs across all ES clusters
- 34. 2018 34 93,000 Total Changelog events per second (peak)
- 35. 2018 35 3.7 minutes Average Time to Index
- 36. 2018 36 0Support tickets post-launch
- 37. Proprietary & Confidential 2018 37 What Now? - Explore other applications of this infrastructure - Ongoing technical challenges - Data Drift, Consumer Lag
- 38. Proprietary & Confidential 2018 Thanks! 38 kevin.xu@mailchimp.com