Elastic development. Implementing Big Data search Grzegorz Kołpuć
0 likes102 views
The document outlines the capabilities and infrastructure of Grzegorz Kolpuc's event platform, which includes advanced features such as big data search, analytics, and geo-search functionalities. It details the architecture of distributed search engines, scripting for performance enhancement, and alerting mechanisms to notify users of data changes. Additionally, it mentions the scale of operations, including the number of clusters, indices, and daily queries handled.
![standard highlighter
"highlight": {
"StrEvntBriefBlob": [
"continues to see <b>pressure</b> on
internal growth"
]
}
"snippets": [
{
"snippet": "continues to see
<b>pressure</b> on internal growth",
"startOff": 2082,
"endOff": 2126,
"pages": [0],
"snippetID": 1
}
]
fvhEx highlighter](https://image.slidesharecdn.com/wszystkootechnologiachduychdanychgrzegorzkopu-180116115100/85/Elastic-development-Implementing-Big-Data-search-Grzegorz-Kolpuc-23-320.jpg)
!["aggregations": {
"events_grouped": {
"terms": {"field": "group"},
"aggregations": {
"top_events": {
"top_hits": {
"size": 10,
"sort": [
{"date": {"order": "desc"}}
]
}
}
}
}
}
"aggs": {
"level2": {
"date_histogram": {
"field": "timestamp",
"interval": "day",
"order": {"_key": "desc"}
},
"aggs": {
"level3": {
"terms": {"field": "host"},
"aggs": {
"level4": {
"terms": {"field":
"apiRequest.appId"}
}
}
}}}}](https://image.slidesharecdn.com/wszystkootechnologiachduychdanychgrzegorzkopu-180116115100/85/Elastic-development-Implementing-Big-Data-search-Grzegorz-Kolpuc-27-320.jpg)
![‘total count of snippets’
{
"query": {
"function_score": {
"query": {
"query_string": {"query":
"Average "Running five""}
},
"script_score": {>>>>},
"boost_mode": "replace"
}
},
"sort": [{"_score": {}}]
}
"script_score": {
"script_id": "scount6",
"lang": "groovy",
"params": {
"terms": ["average"],
"phases": [
{
"distance": 1,
"phase": ["running","five"]
}
],
"fields": [{"field": "StrEvntTranscriptBlob",
"grp": 111}]
}
},
"boost_mode": "replace"](https://image.slidesharecdn.com/wszystkootechnologiachduychdanychgrzegorzkopu-180116115100/85/Elastic-development-Implementing-Big-Data-search-Grzegorz-Kolpuc-30-320.jpg)
![{
"error": "NoShardAvailableActionException
[[std_events_sem_021215][5] null]",
“status": 503
}](https://image.slidesharecdn.com/wszystkootechnologiachduychdanychgrzegorzkopu-180116115100/85/Elastic-development-Implementing-Big-Data-search-Grzegorz-Kolpuc-35-320.jpg)
Elastic development. Implementing Big Data search Grzegorz Kołpuć
- 6. Financial & Risk IP & Science Legal News Tax & Accounting Technology & OPS
- 9. 1200+ EMPLOYEES IN GDYNIA 150+ IN TECHNOLOGY
- 10. Eikon
- 12. Elastic Development Implementing big data search
- 13. Distributed Search Engine Full-text search Analytics
- 14. open source document-oriented based on Lucene distributed search analytics
- 15. full-text search filtering (exact matches, ranges, geo) cacheable
- 19. Marvel
- 21. Native client fluent API single request bulk requests bulk processor
- 23. standard highlighter "highlight": { "StrEvntBriefBlob": [ "continues to see <b>pressure</b> on internal growth" ] } "snippets": [ { "snippet": "continues to see <b>pressure</b> on internal growth", "startOff": 2082, "endOff": 2126, "pages": [0], "snippetID": 1 } ] fvhEx highlighter
- 24. 2. Cross cluster search cluster_threecluster_twocluster_one newsfilings research events
- 25. Tribe Node cluster_threecluster_twocluster_one newsfilings research events Tribe Node tribe: t1: cluster.name: cluster_one t2: cluster.name: cluster_two t2: cluster.name: cluster_three
- 26. 3. Analytics grouping search result statistics metrics
- 27. "aggregations": { "events_grouped": { "terms": {"field": "group"}, "aggregations": { "top_events": { "top_hits": { "size": 10, "sort": [ {"date": {"order": "desc"}} ] } } } } } "aggs": { "level2": { "date_histogram": { "field": "timestamp", "interval": "day", "order": {"_key": "desc"} }, "aggs": { "level3": { "terms": {"field": "host"}, "aggs": { "level4": { "terms": {"field": "apiRequest.appId"} } } }}}}
- 28. 4. Geo search distance filter polygon filter bounding box filter shape query "location": { "type": "geo_shape", "tree": "quadtree", "precision": "1m" } "position": { "type": "geo_point" }
- 29. 5. Scripting if nothing else works decrease performance
- 30. ‘total count of snippets’ { "query": { "function_score": { "query": { "query_string": {"query": "Average "Running five""} }, "script_score": {>>>>}, "boost_mode": "replace" } }, "sort": [{"_score": {}}] } "script_score": { "script_id": "scount6", "lang": "groovy", "params": { "terms": ["average"], "phases": [ { "distance": 1, "phase": ["running","five"] } ], "fields": [{"field": "StrEvntTranscriptBlob", "grp": 111}] } }, "boost_mode": "replace"
- 31. 6. Alerting users subscribed to data notify users when data changes
- 32. Elastic Search Percolate Elastic Search index documentsquery
- 33. Any Cons?
- 36. Support
- 37. Reindex Apply mapping changes online Updatable number of shards in the index Shard splitting
- 41. 16+ TB of indexed data 7,000,000,000+ documents
- 42. 100,000+ queries per day (EVENTS) 500,000+ queries per day (expected for research)
- 43. Q?