SF ElasticSearch Meetup - How HipChat Scaled to 1B Messages
5 likes1,877 views
The document discusses the implementation of Elasticsearch in Hipchat for improved messaging services and performance, addressing challenges with CouchDB. It outlines technical specifications, performance benchmarks, and scaling issues encountered, along with insights on system configurations. The session appears to encourage monitoring and careful management to enhance user experience and system efficiency.
![# Elasticsearch
nodes = partial_search(:node, "role:elasticsearch_server AND
chef_environment:#{node.chef_environment}",
:keys => {
'hostname' => [ 'hostname' ]
})
if nodes.empty?
Chef::Log.error "site.json - No Elasticsearch servers found"
else
es_nodes = nodes.map {|n|
{
'host' => n['hostname'],
'port' => 9200
}
}
data["elasticsearch"] = {
"connections" => es_nodes.sort_by {|n| n['host'] }
}
end
Wednesday, November 20, 13](https://image.slidesharecdn.com/hipchatelasticsearchmeetupfin-131120143942-phpapp01/85/SF-ElasticSearch-Meetup-How-HipChat-Scaled-to-1B-Messages-38-320.jpg)
![site.json
"elasticsearch": {
"connections": [
{
“host": "esindex-c1",
"port": 9200
},
...
{
"host": "esindex-e3",
"port": 9200
}
]
}
Wednesday, November 20, 13](https://image.slidesharecdn.com/hipchatelasticsearchmeetupfin-131120143942-phpapp01/85/SF-ElasticSearch-Meetup-How-HipChat-Scaled-to-1B-Messages-39-320.jpg)
SF ElasticSearch Meetup - How HipChat Scaled to 1B Messages
- 1. Wednesday, November 20, 13
- 2. Heavy Lifting ZUHAIB SIDDIQUE Wednesday, November 20, 13 • @ZUHAIB • ENGINEER
- 3. Wednesday, November 20, 13
- 4. 1. Persistent chat rooms Wednesday, November 20, 13
- 5. 2. File sharing & Chat history Wednesday, November 20, 13
- 6. 3. @mentions + notifications Wednesday, November 20, 13
- 7. 4. Integrations and API Wednesday, November 20, 13
- 8. 5. Runs on everything Wednesday, November 20, 13
- 9. 5. Runs on everything Wednesday, November 20, 13
- 10. 5. Runs on everything Wednesday, November 20, 13
- 11. Group chat and IM for teams. Wednesday, November 20, 13
- 13. What is a HipChat Message? { } Wednesday, November 20, 13 "id": "b0952da8-faa1-48f7-b1df-a8b2fb24bdfe", "from": { "nick": "Zuhaib Siddique", "group_id": 1, "user_id": 1 }, "room": { "group_id": 1, "name": "testing", "id": 1 }, "stanza_type": "muc_message", "date": "2013-11-14T23:20:13Z 424967", "stanza_data": { "body": "ElasticSearch FTW" }
- 14. In the beginning there was … Wednesday, November 20, 13
- 15. Then we added... https://github.com/rnewson/couchdb-lucene Wednesday, November 20, 13
- 16. CouchDB + CouchDB-Lucene •Started to tip over at 200M Docs •m2.4xlarge Wednesday, November 20, 13
- 17. CouchDB + CouchDB-Lucene m2.4xlarge m2.4xlarge Worked well for the first 500M Wednesday, November 20, 13
- 18. m2.4xlarge Wednesday, November 20, 13 hs1.8xlarge
- 19. But it still wasn’t big enough. Wednesday, November 20, 13
- 20. Dont F*ck the Customer Wednesday, November 20, 13
- 21. First Attempt at Scaling Wednesday, November 20, 13
- 22. Wednesday, November 20, 13
- 23. Why ElasticSearch for HipChat Wednesday, November 20, 13
- 24. Why ElasticSearch for HipChat { "query_string" : { "default_field" : "content", "query" : "this AND that OR thus" } } Wednesday, November 20, 13
- 25. Why ElasticSearch for HipChat Wednesday, November 20, 13
- 26. Search Error Rate Wednesday, November 20, 13
- 27. Wednesday, November 20, 13
- 28. Don’t Forget about CouchDB AWS Max EBS size Wednesday, November 20, 13
- 29. Stop the CouchDB bleeding Wednesday, November 20, 13
- 30. CouchDB end of the road Wednesday, November 20, 13
- 31. Wednesday, November 20, 13
- 32. (allthethings) ElasticSearch ElasticSearch { "_index": "muc-2013.11", "_type": "document", "_id": "b0952da8-faa1-48f7-b1df-a8b2fb24bdfe", "_version": 1, "exists": true, "_source": { "from": { "nick": "Zuhaib Siddique", "group_id": 1, "user_id": 1 }, "room": { "group_id": 1, "name": "testing", "id": 1 }, "stanza_type": "muc_message", "date": "2013-11-14T23:20:13Z 424967", "_id": "b0952da8-faa1-48f7-b1df-a8b2fb24bdfe", "stanza_data": { "body": "ElasticSearch FTW" } } } Wednesday, November 20, 13 CouchDB { "id": "b0952da8-faa1-48f7-b1df-a8b2fb24bdfe", "from": { "nick": "Zuhaib Siddique", "group_id": 1, "user_id": 1 }, "room": { "group_id": 1, "name": "testing", "id": 1 }, "stanza_type": "muc_message", "date": "2013-11-14T23:20:13Z 424967", "stanza_data": { "body": "ElasticSearch FTW" } }
- 33. (allthethings) ElasticSearch ElasticSearch { "_index": "muc-2013.11", "_type": "document", "_id": "b0952da8-faa1-48f7-b1df-a8b2fb24bdfe", "_version": 1, "exists": true, "_source": { "from": { "nick": "Zuhaib Siddique", "group_id": 1, "user_id": 1 }, "room": { "group_id": 1, "name": "testing", "id": 1 }, "stanza_type": "muc_message", "date": "2013-11-14T23:20:13Z 424967", "_id": "b0952da8-faa1-48f7-b1df-a8b2fb24bdfe", "stanza_data": { "body": "ElasticSearch FTW" } } } Wednesday, November 20, 13 CouchDB { "id": "b0952da8-faa1-48f7-b1df-a8b2fb24bdfe", "from": { "nick": "Zuhaib Siddique", "group_id": 1, "user_id": 1 }, "room": { "group_id": 1, "name": "testing", "id": 1 }, "stanza_type": "muc_message", "date": "2013-11-14T23:20:13Z 424967", "stanza_data": { "body": "ElasticSearch FTW" } }
- 34. CouchDB vs ElasticSearch CouchDB Spike to 800ms! Avg 100ms with ES Wednesday, November 20, 13
- 35. Running ElasticSearch HipChat Customer master Wednesday, November 20, 13 clients Thanks Philip O’Toole - Loggly
- 36. $ fab create_esindex:hostname=esindex-d4 Wednesday, November 20, 13
- 37. Behind the Scenes • Spin up EC2 Instance • Install Oracle Java 7 • Create Provision IOPs EBS • Install and configure ElasticSearch • ??? • Profit! Wednesday, November 20, 13
- 38. # Elasticsearch nodes = partial_search(:node, "role:elasticsearch_server AND chef_environment:#{node.chef_environment}", :keys => { 'hostname' => [ 'hostname' ] }) if nodes.empty? Chef::Log.error "site.json - No Elasticsearch servers found" else es_nodes = nodes.map {|n| { 'host' => n['hostname'], 'port' => 9200 } } data["elasticsearch"] = { "connections" => es_nodes.sort_by {|n| n['host'] } } end Wednesday, November 20, 13
- 39. site.json "elasticsearch": { "connections": [ { “host": "esindex-c1", "port": 9200 }, ... { "host": "esindex-e3", "port": 9200 } ] } Wednesday, November 20, 13
- 40. You Wednesday, November 20, 13
- 41. Running Elasticsearch Wednesday, November 20, 13
- 42. Running ElasticSearch Single Node OOM! Wednesday, November 20, 13
- 43. Monitor, Monitor, Monitor! Wednesday, November 20, 13
- 45. 100,000,000,000? To infinity and beyond! Wednesday, November 20, 13