![POST crime/incidents
{
“location” : { “lat” : 41.12, “lon” : -71.34 }
}
24
insert
geo_point mapping
POST crime/incidents
{
“location” : “41.12, -71.34”
}
POST crime/incidents
{
“location” : [[-71.34, 41.12], [-71.32, 41.21]]
}](https://image.slidesharecdn.com/geo-230223084544-6d007b62/85/The-state-of-geo-in-ElasticSearch-24-320.jpg)
![26
insert
geo_shape mapping
POST police/precincts/
{
“coverage” : {
“type” : “polygon”,
“coordinates” : [[
[-73.9762134, 40.7538588],
[-73.9742356, 40.7526327],
[-73.9656733, 40.7516774],
[-73.9763236, 40.7521246],
[-73.9723788, 40.7516733],
[-73.9732423, 40.7523556],
[-73.9762134, 40.7538588]
]]
}
}](https://image.slidesharecdn.com/geo-230223084544-6d007b62/85/The-state-of-geo-in-ElasticSearch-26-320.jpg)
![‹#› 50
GeoDistance
Agg
{
"aggs" : {
“sf_rings" : {
"geo_distance" : {
"field" : "location",
"origin" : [32.95,
-96.82],
"ranges" : [
{ "to" : 50 },
{ "from" : 50,
"to" : 100 },
{ "from" : 100,
"to" : 300}
]
}
}
}
}](https://image.slidesharecdn.com/geo-230223084544-6d007b62/85/The-state-of-geo-in-ElasticSearch-50-320.jpg)
The state of geo in ElasticSearch
- 1. Elastic March 1, 2018 @nknize The State of Geo in Elasticsearch Nick Knize, Elasticsearch Software Engineer Thomas Neirynck, Kibana Visualization Area Lead
- 2. 2 Geospatial capabilities are becoming more popular among Elasticsearch Users
- 3. Topics 3 Geospatial Indexing, Search, and Visualization 1 Kibana / Elastic Maps Service 2 Geo Field Mappings 3 Geo Indexing, Search, and Lucene Data Structures 4 Geo Aggregations
- 4. Kibana / Elastic Maps Service
- 5. Kibana Visualizations 5 Out-of-the-box visualizations for geodata in Elasticsearch 2 types - Coordinate Maps - Region Maps Visualize is built on top of the Elasticsearch aggregations
- 6. Coordinate Map Visualization 6 Shows result of geohash_grid aggregations. Shows summary of all documents that belong to a single cell. Put location of “summarized” point in the “geo-centroid” (weighted middle). This gives a better approximate location. The more zoomed in, the more precise the location. Different marker-styles (bubbles, heatmap)
- 7. Example 1
- 8. Example 2
- 9. Region Maps 9 “Choropleth maps” Thematic maps: color intensity correspond to magnitude of metric Shows result of terms aggregations. “Client-side” join between the result of term aggregation and a reference shape layer. - Polygons/Multipolygons (simple feature) - Documents in elasticsearch need to have field that matches a property of the
- 11. Vega 1 1 Experimental feature Vega/VegaLite is a domain language in JSON to create visualizations. Vega has support for geographic projection.
- 12. Dashboard integration 1 2 - Use map for spatial filtering of data ... - … and have other filters applied to your map
- 14. Elastic Maps Service 1 4 Reference basemapping and reference data service hosted by Elastic. “Getting started” experience for mapping. (1) World base map - Base for Coordinate Map, Region Map (2) Shape layers - World countries, US States, Germany States, Canada Provinces, USA zip-codes - Number of identifier fields (name in one or more languages, and ISO-identifiers)
- 16. Custom base maps 1 6 - (1) Configure global base-map in kibana.yml by using Tile Map Service URL tilemap.url: https://tiles.elastic.co/v2/default/{z}/{x}/{y} - (2) Configure visualization-specific base-map using WMS (web map service) - Requires 3rd party geo-service - Geoserverb - ArcGIS Server - MapServer - ….
- 17. Custom maps examples 1 7 Image Removed
- 18. Custom shape layers 1 8 - geojson/topojson - Configure in kibana.yml -> available in region maps UI regionmap: includeElasticMapsService: false layers: - name: "Departments of France" url: "http://my.cors.enabled.server.org/france_departements.geojson" attribution: "INRAP" fields: - name: "department" description: "Full department name" - name: "INSEE" description: "INSEE numeric identifier" - Use any web-server - Make sure is CORS enabled so Kibana can download the data (!)
- 19. - customization - https://www.elastic.co/blog/kibana-and-a-custom-tile-server-for-nhl-data - https://www.elastic.co/blog/custom-region-maps-in-kibana-6-0 Useful blog posts
- 20. Future
- 21. Elastic Maps Service - More base layers (satellite, contours) - Different stylesheets - On-prem deployments Kibana - Elastic Maps Service integration with Vega - No restriction on number of layers - Support for geo_shape - Visualize individual documents/custom styling - Spatial filtering Upcoming
- 23. 23 PUT crime/incidents/_mapping { “properties” : { “location” : { “type” : “geo_point”, “ignore_malformed” : true, } } } define geo_point mapping
- 24. POST crime/incidents { “location” : { “lat” : 41.12, “lon” : -71.34 } } 24 insert geo_point mapping POST crime/incidents { “location” : “41.12, -71.34” } POST crime/incidents { “location” : [[-71.34, 41.12], [-71.32, 41.21]] }
- 25. 25 define geo_shape mapping PUT police/precincts/_mapping { “properties” : { “coverage” : { “type” : “geo_shape”, “ignore_malformed” : false, “tree” : ”quadtree”, “precision” : “5m”, “distance_error_pct“ : 0.025, “orientation” : “ccw”, “points_only” : false } } }
- 26. 26 insert geo_shape mapping POST police/precincts/ { “coverage” : { “type” : “polygon”, “coordinates” : [[ [-73.9762134, 40.7538588], [-73.9742356, 40.7526327], [-73.9656733, 40.7516774], [-73.9763236, 40.7521246], [-73.9723788, 40.7516733], [-73.9732423, 40.7523556], [-73.9762134, 40.7538588] ]] } }
- 27. • Shapes are parsed using OGC and ISO standards definitions • OGC Simple Feature Access • ISO Geographic information — Spatial Schema (19107:2003) • Supports the following geo_shape types • Point, MultiPoint • LineString, MultiLineString • Polygon (with holes), MultiPolygon (with holes) • Envelope (bbox) geo_shape mapping 27 insert
- 33. 33 geo_point indexing 2.x term/postings encoding term postings (doc ids) 1 1, 2, 3, 4, 5 10 1, 2, 4 11 3, 5 100 1 101 2, 4 111 3, 5 1000 2 1010 4 1011 3 1110 3 1111 5
- 34. 34 geo_point indexing 5.0 - “points” data structure - (Bkd-tree)
- 35. 35 geo_point indexing 5.0 - “points” data structure - (Bkd-tree)
- 37. 37 geo_shape indexing current - terms/postings encoding • Max tree_levels == 32 (2 bits / cell) • distance_error_pct • “slop” factor to manage transient memory usage • % of the diagonal distance (degrees) of the shape • Default == 0 if precision set (2.0) • points_only • optimization for points only shape index • short-circuits recursion
- 38. 38 geo_shape indexing 7.0+ - “ranges” encoding (Bkd-tree) • Dimensional Shapes represented using Minimum Bounding Ranges (MBR) ‒ Ranges (1D) - Available from 5.1+ for numerics, dates, and IP (v4 and v6) ‒ Rectangles (2D) - LatLonBoundingBox Available in Lucene 7.1+ ‒ Cubes (3D) ‒ Tesseract (4D) Quad Cells Indexed as LatLonBoundingBox
- 39. 39 geo_shape indexing performance - 1D Numerics
- 41. 41 geo_point search Pre 5.0 - terms/postings encoding • Spatial Queries • BoundingBox, Distance, DistanceRange, Polygon • PRECISION_STEP controls number of query terms (must match with index) • TwoPhaseIterator • Delays boundary confirmation so other query (filters, conjunctions) can pre-filter
- 42. 42 geo_point search 5.0+ - “points” encoding (Bkd-tree) Leaf cell is fully within polygon (salmon) - return all docs Leaf cell crosses the boundary (gray) - two-phase check 1 2
- 43. 43 geo_point search 5.0+ - performance improvements
- 44. 44 geo_shape search capabilities • Supports the following geo_shape types ‒ Point, MultiPoint ‒ LineString, MultiLineString ‒ Polygon (with holes), MultiPolygon (with holes) ‒ Envelope (bbox) • Supports relational queries ‒ INTERSECTS, DISJOINT, WITHIN, CONTAINS
- 45. 45 geo_shape search current - terms/postings encoding Recursively Traverse Query terms 1 2 Collect DocIDs from Postings based on requested relation
- 46. 46 geo_shape search 7.0+ - “points” encoding (B-kd Tree)
- 47. 47 geo_shape search 7.0+ - “points” encoding (B-kd Tree)
- 48. 48 geo_shape search 1D numeric range performance
- 50. ‹#› 50 GeoDistance Agg { "aggs" : { “sf_rings" : { "geo_distance" : { "field" : "location", "origin" : [32.95, -96.82], "ranges" : [ { "to" : 50 }, { "from" : 50, "to" : 100 }, { "from" : 100, "to" : 300} ] } } } }
- 52. ‹#› 52 GeoGrid Agg { "aggs" : { “crime_cells" : { "geohash_grid" : { "field" : "location", "precision" : 8 } } } }
- 54. ‹#› 54 GeoCentroid Agg "query" : { "match" : { "crime" : "burglary" } }, "aggs" : { "towns" : { "terms" : { "field" : "town" }, "aggs" : { "centroid" : { "geo_centroid" : { "field" : “location" } } } } }
- 57. 57 Geo Aggregations more available, and coming soon... • matrix_stats - (Matrix Aggs) plugin ‒ kurtosis/skewness ‒ variance-covariance matrix ‒ pearson’s product correlation matrix • geo_stats - Future? ‒ Moran’s I - measuring spatial auto-correlation ‒ Getis-Ord - spatial hot spot analysis
- 58. Questions? 19