SlideShare a Scribd company logo

Geospatial Indexing and Search at Scale with Apache Lucene

Download as PPTX, PDF
Nicholas Knize, Ph.D., GISP, profile picture
Nicholas Knize, Ph.D., GISP

The document discusses advancements in geospatial indexing and search using Lucene, detailing the evolution from prefix trees to block k-dimensional (BKD) trees for efficient geo searches. It covers the implementation of geo point and geo shape mappings in Elasticsearch, alongside techniques for indexing and searching geospatial data. Additionally, it highlights use cases for these technologies, including urban planning and virtual gaming, and introduces new features and performance improvements in recent updates.

Software
Related topics:
Data Structures
1 of 67
Downloaded 15 times
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
Geospatial Indexing & Search at Scale with Lucene Nick Knize, PhD Lucene PMC Member + Committer @nknize
2 Lets take a trip... ...through Geo advancements Lucene Geo Search, Then & Now: Prefix Trees vs. BKD Trees Improving Shape Search: BKD + Tessellation 2 23 Lucene Geo in Elasticsearch, an Introduction1
Mappings Geo Field Types
4 PUT crime/incidents/_mapping { “properties” : { “location” : { “type” : “geo_point”, “ignore_malformed” : true, } } } define geo_point mapping
POST crime/incidents { “location” : { “lat” : 41.12, “lon” : -71.34 } } 5 insert geo_point mapping POST crime/incidents { “location” : “41.12, -71.34” } POST crime/incidents { “location” : [[-71.34, 41.12], [-71.32, 41.21]] }
6 define geo_shape mapping PUT police/precincts/_mapping { “properties” : { “coverage” : { “type” : “geo_shape”, “ignore_malformed” : false, “tree” : ”quadtree”, “precision” : “5m”, “distance_error_pct“ : 0.025, “orientation” : “ccw”, “points_only” : false } } }
7 insert geo_shape mapping POST police/precincts/ { “coverage” : { “type” : “polygon”, “coordinates” : [[ [-73.9762134, 40.7538588], [-73.9742356, 40.7526327], [-73.9656733, 40.7516774], [-73.9763236, 40.7521246], [-73.9723788, 40.7516733], [-73.9732423, 40.7523556], [-73.9762134, 40.7538588] ]] } }
• Shapes are parsed using OGC and ISO standards definitions • OGC Simple Feature Access • ISO Geographic information — Spatial Schema (19107:2003) • Supports the following geo_shape types • Point, MultiPoint • LineString, MultiLineString • Polygon (with holes), MultiPolygon (with holes) • Envelope (bbox) geo_shape mapping 8 insert
Indexing Under the hood
10 A long, long time ago, in a galaxy far far away... ...the Inverted [text] Index terms dictionary (terms) postings list (doc ids) Fast 2 The 1 brown 1 dog 1 fox 1 jumping 2 jumps 1 lazy 1 over 1 quick 1 spiders 2 the 1
11 And the open source community went bananas! we Lucene text search!
12 But what about 1D numbers...? Prefix Trees; precision terms dictionary (terms) postings list (doc ids) 1 1, 2, 3, 4, 5 10 2, 3, 4 11 1, 5 100 2 101 3, 4 111 1, 5 1000 2 1010 4 1011 3 1110 1 1111 5 1 myINT: 3881553173 1110 0111 0101 1011 1100 1101 0001 0101 2 myINT: 2405166357 1000 1111 0101 1011 1110 1101 0001 0101 3 myINT: 3205335297 1011 1111 0000 1101 1000 1001 0000 0001 4 myINT: 2835679237 1010 1001 0000 0101 0000 1000 0000 0101 5 myINT: 4177856517 1111 1001 0000 1101 0000 1000 0000 0101
13 But what about numbers...? Prefix Trees; precision terms dictionary (terms) postings list (doc ids) 1 1, 2, 3, 4, 5 10 2, 3, 4 11 1, 5 100 2 101 3, 4 111 1, 5 1000 2 1010 4 1011 3 1110 1 1111 5 1 myINT: 3881553173 1110 0111 0101 1011 1100 1101 0001 0101 2 myINT: 2405166357 1000 1111 0101 1011 1110 1101 0001 0101 3 myINT: 3205335297 1011 1111 0000 1101 1000 1001 0000 0001 4 myINT: 2835679237 1010 1001 0000 0101 0000 1000 0000 0101 5 myINT: 4177856517 1111 1001 0000 1101 0000 1000 0000 0101
14 And the Lucene community went...bananas...again... we Lucene NUMERIC search!
15 But, but, what about Geo? …we’re cool too!
16 How about Geo...terms? Prefix Trees; round peg, square hole terms dictionary (terms) postings list (doc ids) 1 1, 2, 3, 4, 5 10 2, 3, 4 11 1, 5 100 2 101 3, 4 111 1, 5 1000 2 1010 4 1011 3 1110 1 1111 5
17 Searching for round pegs in square holes... Traversing the Prefix Tree... terms dictionary (terms) postings list (doc ids) 1 1, 2, 3, 4, 5 10 2, 3, 4 11 1, 5 100 2 101 3, 4 111 1, 5 1000 2 1010 4 1011 3 1110 1 1111 5 2 4 3 1 5
18 Searching for round pegs in square holes... Traversing the Prefix Tree... terms dictionary (terms) postings list (doc ids) 1 1, 2, 3, 4, 5 10 2, 3, 4 11 1, 5 100 2 101 3, 4 111 1, 5 1000 2 1010 4 1011 3 1110 1 1111 5 2 4 3 1 5
19 Searching for round pegs in square holes... Traversing the Prefix Tree... terms dictionary (terms) postings list (doc ids) 1 1, 2, 3, 4, 5 10 2, 3, 4 11 1, 5 100 2 101 3, 4 111 1, 5 1000 2 1010 4 1011 3 1110 1 1111 5 2 4 3 1 5
20 Searching for round pegs in square holes... Traversing the Prefix Tree... terms dictionary (terms) postings list (doc ids) 1 1, 2, 3, 4, 5 10 2, 3, 4 11 1, 5 100 2 101 3, 4 111 1, 5 1000 2 1010 4 1011 3 1110 1 1111 5 2 4 3 1 5
21 Searching for round pegs in square holes... Traversing the Prefix Tree... terms dictionary (terms) postings list (doc ids) 1 1, 2, 3, 4, 5 10 2, 3, 4 11 1, 5 100 2 101 3, 4 111 1, 5 1000 2 1010 4 1011 3 1110 1 1111 5 2 4 3 1 5
22 Searching for round pegs in square holes... Traversing the Prefix Tree...of death? terms dictionary (terms) postings list (doc ids) 1 1, 2, 3, 4, 5 10 2, 3, 4 11 1, 5 100 2 101 3, 4 111 1, 5 1000 2 1010 4 1011 3 1110 1 1111 5
23 Okay, cool! But what about GeoShapes?! Quad Trees!
24 Okay, cool! But what about GeoShapes?! Use the Inverted [geo] Index... terms dictionary (terms) postings list (doc ids) 1 1, 2, 3, 4, 5 10 1, 2, 4 11 3, 5 100 1 101 2, 4 111 3, 5 1000 2 1010 4 1011 3 1110 3 1111 5 DocID: 6
25 Okay, cool! But what about GeoShapes?! Use the Inverted [geo] Index... terms dictionary (terms) postings list (doc ids) 1 1, 2, 3, 4, 5, 6 10 1, 2, 4 11 3, 5, 6 100 1 101 2, 4 111 3, 5, 6 1000 2 1010 4 1011 3 1110 3, 6 1111 5, 6 DocID: 6
26 Okay, cool! But what about GeoShapes?! Melt with the Inverted [geo] Index...
27 Okay, cool! But what about GeoShapes?! Melt with the Inverted [geo] Index... • Max tree_levels == 32 (2 bits / cell) • distance_error_pct • “slop” factor to manage transient memory usage • % of the diagonal distance (degrees) of the shape • Default == 0 if precision set (2.0) • points_only • optimization for points only shape index • short-circuits recursion
28 geo search? pssh… it’s simple! ¯_(ツ)_/¯ ¯_(ツ)_/¯ ¯_(ツ)_/¯ ¯_(ツ)_/¯ ¯_(ツ)_/¯ ¯_(ツ)_/¯ ¯_(ツ)_/¯ ¯_(ツ)_/¯ ¯_(ツ)_/¯ ¯_(ツ)_/¯ ¯_(ツ)_/¯ ¯_(ツ)_/¯
29
30 For duty and humanity!
31 We’re doing it wrong
32
33
34 Block K Dimensional Trees (Bkd) to the rescue! ...the right tool!
35 Block K Dimensional Trees (Bkd) to the rescue! ...the right tool! Miny Minx Maxy Maxx m: internal node
36 Block K Dimensional Trees (Bkd) to the rescue! ...the right tool! Yval Xval leaf node
37 Block K Dimensional Trees (Bkd) to the rescue! multi-way...
38 Block K Dimensional Trees (Bkd) to the rescue! multi-way… perfectly balanced...
39 Block K Dimensional Trees (Bkd) to the rescue! multi-way… perfectly balanced...FAIR...
40 Block K Dimensional Trees (Bkd) to the rescue! ...the right tool!
41 Okay, cool! But what about GeoShapes?! Tessellation!
42 Quad Tree Decomposition Simple polygon example • 8 vertex polygon • 1º x 1º coverage area • 3m quad cell resolution • 1,105,889 terms
43 Tessellation Decomposition Simple polygon example • 8 vertex polygon • 1º x 1º coverage area • 3m quad cell resolution • 1,105,889 terms • 1.11 cm resolution • 8 terms • 138,236 : 1 term ratio • (•◡•) / smaller, faster index!
44 Tessellation + BKD a made in… Lucene 7.4 Miny Minx Maxy Maxx m: internal node ??? leaf node
45 Tessellation + BKD a seven dimension made in… Lucene 7.4 Index Dimensions Data Dimensions
46 Tessellation + BKD a made in… Lucene 7.4 Miny Minx Maxy Maxx m: internal node ??? leaf node
47 Smaller, faster, stronger... Searching triangle intersections
48 XYShape for Spatial...in Lucene 8.2+ Tessellating & Indexing Virtual Worlds
49 Smaller, faster, stronger… at scale Smaller index...
50 Smaller, faster, stronger… at scale Faster indexing - Nearly as fast as points!!
51 Smaller, faster, stronger… at scale Positive feedback... @imotov: "...test that was crashing a shard on a node with 16GB heap after running for 4.5 hours now finishes in less than 1sec" @esri: "results show that the insert operation is 25 to 30 times faster in Elasticsearch 6.6 compare to Elasticsearch 6.5.0/6.4.2. ...along with better performance we also get accurate results for spatial queries"
52 geo search? pssh… it’s simple! ¯_(ツ)_/¯ ¯_(ツ)_/¯ ¯_(ツ)_/¯ ¯_(ツ)_/¯ ¯_(ツ)_/¯ ¯_(ツ)_/¯
53 So, what’s next?? • Non-geo use cases (`XYShape` type) ‒ CAD Drawings (Design, County / City Planning) ‒ Venue Mapping (Conferences, Hotels, Theme Parks, Schools) ‒ Sports analysis (Chicago Cubs, Baseball Advanced Media) ‒ Virtual Gaming & Mapping (Blizzard, Cyber Security / SIEM) • Other coordinate systems (datum / projection support) ‒ Non terrestrial planet models ‒ Localized projections ‒ Custom projections • Beyond 2D ‒ Space - Time (OGC Moving Features) ‒ Elevation Modelling (DEM, DTM) ‒ LiDAR (3D Modelling) Use case application goodies... Available 8.2 In Work
Aggregations Geo & Analytics
‹#› 55 GeoDistance Agg { "aggs" : { “sf_rings" : { "geo_distance" : { "field" : "location", "origin" : [32.95, -96.82], "ranges" : [ { "to" : 50 }, { "from" : 50, "to" : 100 }, { "from" : 100, "to" : 300} ] } } } }
‹#› 56 GeoDistance Agg
‹#› 57 GeoGrid Agg { "aggs" : { “crime_cells" : { "geohash_grid" : { "field" : "location", "precision" : 8 } } } }
‹#› 58 GeoGrid Agg
59 geotile_grid Aggregation Finally, a grid designed for maps! 7.0 introduces geo_tile aggregation - matches the tiling scheme of well known tile maps in the Web Mercator Projection (EPSG:3857) - on web mercator maps, grid cells are - actually square - preserve an identical aspect ratio at all scales and latitudes
‹#› 60 GeoCentroid Agg "query" : { "match" : { "crime" : "burglary" } }, "aggs" : { "towns" : { "terms" : { "field" : "town" }, "aggs" : { "centroid" : { "geo_centroid" : { "field" : “location" } } } } }
‹#› 61 GeoCentroid Agg
‹#› 62 GeoCentroid Agg
‹#› 63 matrix_stats Agg { "aggs": { "statistics": { "matrix_stats": { "fields": ["poverty", "income"] } } } }
‹#› 64 matrix_stats Agg "aggregations": { "statistics": { "doc_count": 50, "fields": [{ "name": "income", "count": 50, "mean": 51985.1, "variance": 7.383377037755103E7, "skewness": 0.5595114003506483, "kurtosis": 2.5692365287787124, "covariance": { "income": 7.383377037755103E7, "poverty": -21093.65836734694 }, "correlation": { "income": 1.0, "poverty": -0.8352655256272504 } }, { "name": "poverty", "count": 50, "mean": 12.732000000000001, "variance": 8.637730612244896, "skewness": 0.4516049811903419, "kurtosis": 2.8615929677997767, "covariance": { "income": -21093.65836734694, "poverty": 8.637730612244896 }, "correlation": { "income": -0.8352655256272504, "poverty": 1.0 } }] }
65 Geo Aggregations more available, and coming soon... • pca - ML foundation plugin ‒ dimensionality reduction ‒ image analysis ‒ classification / recognition • geo_stats - In work... ‒ Moran’s I - measuring spatial auto-correlation ‒ Getis-Ord - spatial hot spot analysis
66 Now GA in Elastic 7.3….
THANK YOU Follow for updates: @nknize apache/lucene-solr

More Related Content

PPTX
You only look once (YOLO) : unified real time object detection
Entrepreneur / Startup
 
PDF
Object Detection Using R-CNN Deep Learning Framework
Nader Karimi
 
PDF
Uncertainty in Deep Learning
Roberto Pereira Silveira
 
PDF
Deep learning based object detection basics
Brodmann17
 
PDF
Lecture 4 Relationship between pixels
VARUN KUMAR
 
PDF
A Brief History of Object Detection / Tommi Kerola
Preferred Networks
 
PPTX
Edge Detection algorithm and code
Vaddi Manikanta
 
PDF
Deep sort and sort paper introduce presentation
경훈 김
 
You only look once (YOLO) : unified real time object detection
Entrepreneur / Startup
 
Object Detection Using R-CNN Deep Learning Framework
Nader Karimi
 
Uncertainty in Deep Learning
Roberto Pereira Silveira
 
Deep learning based object detection basics
Brodmann17
 
Lecture 4 Relationship between pixels
VARUN KUMAR
 
A Brief History of Object Detection / Tommi Kerola
Preferred Networks
 
Edge Detection algorithm and code
Vaddi Manikanta
 
Deep sort and sort paper introduce presentation
경훈 김
 

What's hot (20)

PDF
Optics ordering points to identify the clustering structure
Rajesh Piryani
 
PDF
Mukund Narasimhan, Engineer, Pinterest at MLconf Seattle 2017
MLconf
 
PPTX
Deep learning for object detection
Wenjing Chen
 
PDF
Neural Networks: Principal Component Analysis (PCA)
Mostafa G. M. Mostafa
 
PDF
ViT (Vision Transformer) Review [CDM]
Dongmin Choi
 
PDF
YOLOv4: optimal speed and accuracy of object detection review
LEE HOSEONG
 
PDF
PR-132: SSD: Single Shot MultiBox Detector
Jinwon Lee
 
PDF
Support Vector Machines for Classification
Prakash Pimpale
 
PPT
Image processing SaltPepper Noise
Ankush Srivastava
 
PPTX
Yolo
Bang Tsui Liou
 
PDF
Semantic Segmentation - Fully Convolutional Networks for Semantic Segmentation
岳華 杜
 
PDF
K - Nearest neighbor ( KNN )
Mohammad Junaid Khan
 
PPTX
Filtering and masking
amudhini
 
PPTX
Object Detection using Deep Neural Networks
Usman Qayyum
 
PDF
Faster R-CNN - PR012
Jinwon Lee
 
PDF
Harris corner detector and face recognition
Shih Wei Huang
 
PDF
Variational Autoencoders VAE - Santiago Pascual - UPC Barcelona 2018
Universitat Politècnica de Catalunya
 
PPTX
Faster R-CNN
anna8885
 
PPTX
Image compression
Bassam Kanber
 
PPT
Image segmentation
Amnaakhaan
 
Optics ordering points to identify the clustering structure
Rajesh Piryani
 
Mukund Narasimhan, Engineer, Pinterest at MLconf Seattle 2017
MLconf
 
Deep learning for object detection
Wenjing Chen
 
Neural Networks: Principal Component Analysis (PCA)
Mostafa G. M. Mostafa
 
ViT (Vision Transformer) Review [CDM]
Dongmin Choi
 
YOLOv4: optimal speed and accuracy of object detection review
LEE HOSEONG
 
PR-132: SSD: Single Shot MultiBox Detector
Jinwon Lee
 
Support Vector Machines for Classification
Prakash Pimpale
 
Image processing SaltPepper Noise
Ankush Srivastava
 
Yolo
Bang Tsui Liou
 
Semantic Segmentation - Fully Convolutional Networks for Semantic Segmentation
岳華 杜
 
K - Nearest neighbor ( KNN )
Mohammad Junaid Khan
 
Filtering and masking
amudhini
 
Object Detection using Deep Neural Networks
Usman Qayyum
 
Faster R-CNN - PR012
Jinwon Lee
 
Harris corner detector and face recognition
Shih Wei Huang
 
Variational Autoencoders VAE - Santiago Pascual - UPC Barcelona 2018
Universitat Politècnica de Catalunya
 
Faster R-CNN
anna8885
 
Image compression
Bassam Kanber
 
Image segmentation
Amnaakhaan
 
Ad

Similar to Geospatial Indexing and Search at Scale with Apache Lucene (20)

PDF
Geo exploration simplified with Elastic Maps
Elasticsearch
 
PDF
Geospatial Advancements in Elasticsearch
Elasticsearch
 
PDF
20170504 - Warp 10 Tour, 42 USA
Mathias Herberts
 
PDF
The state of geo in ElasticSearch
Fan Robbin
 
KEY
Mapping Flatland: Using MongoDB for an MMO Crossword Game (GDC Online 2011)
Grant Goodale
 
PPTX
No(Geo)SQL
Nicolasgmail.com Helleringer
 
PPTX
2014 11 lucene spatial temporal update
David Smiley
 
PDF
The Latest in Spatial & Temporal Search: Presented by David Smiley
Lucidworks
 
KEY
Handling Real-time Geostreams
Raffi Krikorian
 
KEY
Handling Real-time Geostreams
guest35660bc
 
PDF
Search with Polygons: Another Approach to Solr Geospatial Search
lucenerevolution
 
PPTX
Elasticsearch Field Data Types
Bo Andersen
 
PPTX
Lucene 4 spatial
David Smiley
 
PDF
Geo webinarjune2015
Nicholas Knize, Ph.D., GISP
 
PPTX
Geo data analytics
Daniel Marcous
 
PDF
OrientDB & Lucene
wolf4ood
 
PDF
Enhanced Methodology for supporting approximate string search in Geospatial ...
IJMER
 
PDF
Elastic Search
Lukas Vlcek
 
KEY
Geospatial Indexing and Querying with MongoDB
Grant Goodale
 
PPTX
RTree Spatial Indexing with MongoDB - MongoDC
Nicholas Knize, Ph.D., GISP
 
Geo exploration simplified with Elastic Maps
Elasticsearch
 
Geospatial Advancements in Elasticsearch
Elasticsearch
 
20170504 - Warp 10 Tour, 42 USA
Mathias Herberts
 
The state of geo in ElasticSearch
Fan Robbin
 
Mapping Flatland: Using MongoDB for an MMO Crossword Game (GDC Online 2011)
Grant Goodale
 
No(Geo)SQL
Nicolasgmail.com Helleringer
 
2014 11 lucene spatial temporal update
David Smiley
 
The Latest in Spatial & Temporal Search: Presented by David Smiley
Lucidworks
 
Handling Real-time Geostreams
Raffi Krikorian
 
Handling Real-time Geostreams
guest35660bc
 
Search with Polygons: Another Approach to Solr Geospatial Search
lucenerevolution
 
Elasticsearch Field Data Types
Bo Andersen
 
Lucene 4 spatial
David Smiley
 
Geo webinarjune2015
Nicholas Knize, Ph.D., GISP
 
Geo data analytics
Daniel Marcous
 
OrientDB & Lucene
wolf4ood
 
Enhanced Methodology for supporting approximate string search in Geospatial ...
IJMER
 
Elastic Search
Lukas Vlcek
 
Geospatial Indexing and Querying with MongoDB
Grant Goodale
 
RTree Spatial Indexing with MongoDB - MongoDC
Nicholas Knize, Ph.D., GISP
 
Ad

Recently uploaded (20)

PPTX
Transform Your Business with a Software ERP System
Canada Software Company
 
PPTX
L1 - Introduction to python Backend.pptx
MoizAhmed480282
 
PDF
Wondershare Filmora 15 Crack With Activation Key [2025
ghrom2211g
 
PPTX
ai tools demonstartion for schools and inter college
harshavardhanreddyka11
 
PDF
Internet Downloader Manager (IDM) Crack 6.42 Build 41
ghrom2211g
 
PPTX
Agentic AI : A Practical Guide. Undersating, Implementing and Scaling Autono...
IT IDOL Technologies
 
PDF
System and Network Administration Chapter 2
jaramharo
 
PDF
AI in Product Development-omnex systems
omnex systems
 
PDF
Odoo Companies in India – Driving Business Transformation.pdf
azhuhardeen1968
 
PPTX
Odoo POS Development Services by CandidRoot Solutions
CandidRoot Solutions Private Limited
 
PPTX
history of c programming in notes for students .pptx
Vijayakumari829030
 
PPTX
Lecture 3: Operating Systems Introduction to Computer Hardware Systems
hci7se
 
PDF
top salesforce developer skills in 2025.pdf
CloudMetic
 
PDF
How Creative Agencies Leverage Project Management Software.pdf
Orangescrum
 
PDF
SAP S4 Hana Brochure 3 (PTS SYSTEMS AND SOLUTIONS)
pts464036
 
PDF
Claude Code: Everyone is a 10x Developer - A Comprehensive AI-Powered CLI Tool
William Chong
 
PPTX
VVF-Customer-Presentation2025-Ver1.9.pptx
blackmambaettijean
 
PPTX
CHAPTER 12 - CYBER SECURITY AND FUTURE SKILLS (1) (1).pptx
AnujKumar530915
 
PDF
Softaken Excel to vCard Converter Software.pdf
markwillsonmw004
 
PDF
Understanding Forklifts - TECH EHS Solution
TECH EHS Solution
 
Transform Your Business with a Software ERP System
Canada Software Company
 
L1 - Introduction to python Backend.pptx
MoizAhmed480282
 
Wondershare Filmora 15 Crack With Activation Key [2025
ghrom2211g
 
ai tools demonstartion for schools and inter college
harshavardhanreddyka11
 
Internet Downloader Manager (IDM) Crack 6.42 Build 41
ghrom2211g
 
Agentic AI : A Practical Guide. Undersating, Implementing and Scaling Autono...
IT IDOL Technologies
 
System and Network Administration Chapter 2
jaramharo
 
AI in Product Development-omnex systems
omnex systems
 
Odoo Companies in India – Driving Business Transformation.pdf
azhuhardeen1968
 
Odoo POS Development Services by CandidRoot Solutions
CandidRoot Solutions Private Limited
 
history of c programming in notes for students .pptx
Vijayakumari829030
 
Lecture 3: Operating Systems Introduction to Computer Hardware Systems
hci7se
 
top salesforce developer skills in 2025.pdf
CloudMetic
 
How Creative Agencies Leverage Project Management Software.pdf
Orangescrum
 
SAP S4 Hana Brochure 3 (PTS SYSTEMS AND SOLUTIONS)
pts464036
 
Claude Code: Everyone is a 10x Developer - A Comprehensive AI-Powered CLI Tool
William Chong
 
VVF-Customer-Presentation2025-Ver1.9.pptx
blackmambaettijean
 
CHAPTER 12 - CYBER SECURITY AND FUTURE SKILLS (1) (1).pptx
AnujKumar530915
 
Softaken Excel to vCard Converter Software.pdf
markwillsonmw004
 
Understanding Forklifts - TECH EHS Solution
TECH EHS Solution
 

Geospatial Indexing and Search at Scale with Apache Lucene

  • 1. Geospatial Indexing & Search at Scale with Lucene Nick Knize, PhD Lucene PMC Member + Committer @nknize
  • 2. 2 Lets take a trip... ...through Geo advancements Lucene Geo Search, Then & Now: Prefix Trees vs. BKD Trees Improving Shape Search: BKD + Tessellation 2 23 Lucene Geo in Elasticsearch, an Introduction1
  • 4. 4 PUT crime/incidents/_mapping { “properties” : { “location” : { “type” : “geo_point”, “ignore_malformed” : true, } } } define geo_point mapping
  • 5. POST crime/incidents { “location” : { “lat” : 41.12, “lon” : -71.34 } } 5 insert geo_point mapping POST crime/incidents { “location” : “41.12, -71.34” } POST crime/incidents { “location” : [[-71.34, 41.12], [-71.32, 41.21]] }
  • 6. 6 define geo_shape mapping PUT police/precincts/_mapping { “properties” : { “coverage” : { “type” : “geo_shape”, “ignore_malformed” : false, “tree” : ”quadtree”, “precision” : “5m”, “distance_error_pct“ : 0.025, “orientation” : “ccw”, “points_only” : false } } }
  • 7. 7 insert geo_shape mapping POST police/precincts/ { “coverage” : { “type” : “polygon”, “coordinates” : [[ [-73.9762134, 40.7538588], [-73.9742356, 40.7526327], [-73.9656733, 40.7516774], [-73.9763236, 40.7521246], [-73.9723788, 40.7516733], [-73.9732423, 40.7523556], [-73.9762134, 40.7538588] ]] } }
  • 8. • Shapes are parsed using OGC and ISO standards definitions • OGC Simple Feature Access • ISO Geographic information — Spatial Schema (19107:2003) • Supports the following geo_shape types • Point, MultiPoint • LineString, MultiLineString • Polygon (with holes), MultiPolygon (with holes) • Envelope (bbox) geo_shape mapping 8 insert
  • 10. 10 A long, long time ago, in a galaxy far far away... ...the Inverted [text] Index terms dictionary (terms) postings list (doc ids) Fast 2 The 1 brown 1 dog 1 fox 1 jumping 2 jumps 1 lazy 1 over 1 quick 1 spiders 2 the 1
  • 11. 11 And the open source community went bananas! we Lucene text search!
  • 12. 12 But what about 1D numbers...? Prefix Trees; precision terms dictionary (terms) postings list (doc ids) 1 1, 2, 3, 4, 5 10 2, 3, 4 11 1, 5 100 2 101 3, 4 111 1, 5 1000 2 1010 4 1011 3 1110 1 1111 5 1 myINT: 3881553173 1110 0111 0101 1011 1100 1101 0001 0101 2 myINT: 2405166357 1000 1111 0101 1011 1110 1101 0001 0101 3 myINT: 3205335297 1011 1111 0000 1101 1000 1001 0000 0001 4 myINT: 2835679237 1010 1001 0000 0101 0000 1000 0000 0101 5 myINT: 4177856517 1111 1001 0000 1101 0000 1000 0000 0101
  • 13. 13 But what about numbers...? Prefix Trees; precision terms dictionary (terms) postings list (doc ids) 1 1, 2, 3, 4, 5 10 2, 3, 4 11 1, 5 100 2 101 3, 4 111 1, 5 1000 2 1010 4 1011 3 1110 1 1111 5 1 myINT: 3881553173 1110 0111 0101 1011 1100 1101 0001 0101 2 myINT: 2405166357 1000 1111 0101 1011 1110 1101 0001 0101 3 myINT: 3205335297 1011 1111 0000 1101 1000 1001 0000 0001 4 myINT: 2835679237 1010 1001 0000 0101 0000 1000 0000 0101 5 myINT: 4177856517 1111 1001 0000 1101 0000 1000 0000 0101
  • 14. 14 And the Lucene community went...bananas...again... we Lucene NUMERIC search!
  • 15. 15 But, but, what about Geo? …we’re cool too!
  • 16. 16 How about Geo...terms? Prefix Trees; round peg, square hole terms dictionary (terms) postings list (doc ids) 1 1, 2, 3, 4, 5 10 2, 3, 4 11 1, 5 100 2 101 3, 4 111 1, 5 1000 2 1010 4 1011 3 1110 1 1111 5
  • 17. 17 Searching for round pegs in square holes... Traversing the Prefix Tree... terms dictionary (terms) postings list (doc ids) 1 1, 2, 3, 4, 5 10 2, 3, 4 11 1, 5 100 2 101 3, 4 111 1, 5 1000 2 1010 4 1011 3 1110 1 1111 5 2 4 3 1 5
  • 18. 18 Searching for round pegs in square holes... Traversing the Prefix Tree... terms dictionary (terms) postings list (doc ids) 1 1, 2, 3, 4, 5 10 2, 3, 4 11 1, 5 100 2 101 3, 4 111 1, 5 1000 2 1010 4 1011 3 1110 1 1111 5 2 4 3 1 5
  • 19. 19 Searching for round pegs in square holes... Traversing the Prefix Tree... terms dictionary (terms) postings list (doc ids) 1 1, 2, 3, 4, 5 10 2, 3, 4 11 1, 5 100 2 101 3, 4 111 1, 5 1000 2 1010 4 1011 3 1110 1 1111 5 2 4 3 1 5
  • 20. 20 Searching for round pegs in square holes... Traversing the Prefix Tree... terms dictionary (terms) postings list (doc ids) 1 1, 2, 3, 4, 5 10 2, 3, 4 11 1, 5 100 2 101 3, 4 111 1, 5 1000 2 1010 4 1011 3 1110 1 1111 5 2 4 3 1 5
  • 21. 21 Searching for round pegs in square holes... Traversing the Prefix Tree... terms dictionary (terms) postings list (doc ids) 1 1, 2, 3, 4, 5 10 2, 3, 4 11 1, 5 100 2 101 3, 4 111 1, 5 1000 2 1010 4 1011 3 1110 1 1111 5 2 4 3 1 5
  • 22. 22 Searching for round pegs in square holes... Traversing the Prefix Tree...of death? terms dictionary (terms) postings list (doc ids) 1 1, 2, 3, 4, 5 10 2, 3, 4 11 1, 5 100 2 101 3, 4 111 1, 5 1000 2 1010 4 1011 3 1110 1 1111 5
  • 23. 23 Okay, cool! But what about GeoShapes?! Quad Trees!
  • 24. 24 Okay, cool! But what about GeoShapes?! Use the Inverted [geo] Index... terms dictionary (terms) postings list (doc ids) 1 1, 2, 3, 4, 5 10 1, 2, 4 11 3, 5 100 1 101 2, 4 111 3, 5 1000 2 1010 4 1011 3 1110 3 1111 5 DocID: 6
  • 25. 25 Okay, cool! But what about GeoShapes?! Use the Inverted [geo] Index... terms dictionary (terms) postings list (doc ids) 1 1, 2, 3, 4, 5, 6 10 1, 2, 4 11 3, 5, 6 100 1 101 2, 4 111 3, 5, 6 1000 2 1010 4 1011 3 1110 3, 6 1111 5, 6 DocID: 6
  • 26. 26 Okay, cool! But what about GeoShapes?! Melt with the Inverted [geo] Index...
  • 27. 27 Okay, cool! But what about GeoShapes?! Melt with the Inverted [geo] Index... • Max tree_levels == 32 (2 bits / cell) • distance_error_pct • “slop” factor to manage transient memory usage • % of the diagonal distance (degrees) of the shape • Default == 0 if precision set (2.0) • points_only • optimization for points only shape index • short-circuits recursion
  • 28. 28 geo search? pssh… it’s simple! ¯_(ツ)_/¯ ¯_(ツ)_/¯ ¯_(ツ)_/¯ ¯_(ツ)_/¯ ¯_(ツ)_/¯ ¯_(ツ)_/¯ ¯_(ツ)_/¯ ¯_(ツ)_/¯ ¯_(ツ)_/¯ ¯_(ツ)_/¯ ¯_(ツ)_/¯ ¯_(ツ)_/¯
  • 29. 29
  • 30. 30 For duty and humanity!
  • 32. 32
  • 33. 33
  • 34. 34 Block K Dimensional Trees (Bkd) to the rescue! ...the right tool!
  • 35. 35 Block K Dimensional Trees (Bkd) to the rescue! ...the right tool! Miny Minx Maxy Maxx m: internal node
  • 36. 36 Block K Dimensional Trees (Bkd) to the rescue! ...the right tool! Yval Xval leaf node
  • 37. 37 Block K Dimensional Trees (Bkd) to the rescue! multi-way...
  • 38. 38 Block K Dimensional Trees (Bkd) to the rescue! multi-way… perfectly balanced...
  • 39. 39 Block K Dimensional Trees (Bkd) to the rescue! multi-way… perfectly balanced...FAIR...
  • 40. 40 Block K Dimensional Trees (Bkd) to the rescue! ...the right tool!
  • 41. 41 Okay, cool! But what about GeoShapes?! Tessellation!
  • 42. 42 Quad Tree Decomposition Simple polygon example • 8 vertex polygon • 1º x 1º coverage area • 3m quad cell resolution • 1,105,889 terms
  • 43. 43 Tessellation Decomposition Simple polygon example • 8 vertex polygon • 1º x 1º coverage area • 3m quad cell resolution • 1,105,889 terms • 1.11 cm resolution • 8 terms • 138,236 : 1 term ratio • (•◡•) / smaller, faster index!
  • 44. 44 Tessellation + BKD a made in… Lucene 7.4 Miny Minx Maxy Maxx m: internal node ??? leaf node
  • 45. 45 Tessellation + BKD a seven dimension made in… Lucene 7.4 Index Dimensions Data Dimensions
  • 46. 46 Tessellation + BKD a made in… Lucene 7.4 Miny Minx Maxy Maxx m: internal node ??? leaf node
  • 48. 48 XYShape for Spatial...in Lucene 8.2+ Tessellating & Indexing Virtual Worlds
  • 49. 49 Smaller, faster, stronger… at scale Smaller index...
  • 50. 50 Smaller, faster, stronger… at scale Faster indexing - Nearly as fast as points!!
  • 51. 51 Smaller, faster, stronger… at scale Positive feedback... @imotov: "...test that was crashing a shard on a node with 16GB heap after running for 4.5 hours now finishes in less than 1sec" @esri: "results show that the insert operation is 25 to 30 times faster in Elasticsearch 6.6 compare to Elasticsearch 6.5.0/6.4.2. ...along with better performance we also get accurate results for spatial queries"
  • 52. 52 geo search? pssh… it’s simple! ¯_(ツ)_/¯ ¯_(ツ)_/¯ ¯_(ツ)_/¯ ¯_(ツ)_/¯ ¯_(ツ)_/¯ ¯_(ツ)_/¯
  • 53. 53 So, what’s next?? • Non-geo use cases (`XYShape` type) ‒ CAD Drawings (Design, County / City Planning) ‒ Venue Mapping (Conferences, Hotels, Theme Parks, Schools) ‒ Sports analysis (Chicago Cubs, Baseball Advanced Media) ‒ Virtual Gaming & Mapping (Blizzard, Cyber Security / SIEM) • Other coordinate systems (datum / projection support) ‒ Non terrestrial planet models ‒ Localized projections ‒ Custom projections • Beyond 2D ‒ Space - Time (OGC Moving Features) ‒ Elevation Modelling (DEM, DTM) ‒ LiDAR (3D Modelling) Use case application goodies... Available 8.2 In Work
  • 55. ‹#› 55 GeoDistance Agg { "aggs" : { “sf_rings" : { "geo_distance" : { "field" : "location", "origin" : [32.95, -96.82], "ranges" : [ { "to" : 50 }, { "from" : 50, "to" : 100 }, { "from" : 100, "to" : 300} ] } } } }
  • 57. ‹#› 57 GeoGrid Agg { "aggs" : { “crime_cells" : { "geohash_grid" : { "field" : "location", "precision" : 8 } } } }
  • 59. 59 geotile_grid Aggregation Finally, a grid designed for maps! 7.0 introduces geo_tile aggregation - matches the tiling scheme of well known tile maps in the Web Mercator Projection (EPSG:3857) - on web mercator maps, grid cells are - actually square - preserve an identical aspect ratio at all scales and latitudes
  • 60. ‹#› 60 GeoCentroid Agg "query" : { "match" : { "crime" : "burglary" } }, "aggs" : { "towns" : { "terms" : { "field" : "town" }, "aggs" : { "centroid" : { "geo_centroid" : { "field" : “location" } } } } }
  • 63. ‹#› 63 matrix_stats Agg { "aggs": { "statistics": { "matrix_stats": { "fields": ["poverty", "income"] } } } }
  • 64. ‹#› 64 matrix_stats Agg "aggregations": { "statistics": { "doc_count": 50, "fields": [{ "name": "income", "count": 50, "mean": 51985.1, "variance": 7.383377037755103E7, "skewness": 0.5595114003506483, "kurtosis": 2.5692365287787124, "covariance": { "income": 7.383377037755103E7, "poverty": -21093.65836734694 }, "correlation": { "income": 1.0, "poverty": -0.8352655256272504 } }, { "name": "poverty", "count": 50, "mean": 12.732000000000001, "variance": 8.637730612244896, "skewness": 0.4516049811903419, "kurtosis": 2.8615929677997767, "covariance": { "income": -21093.65836734694, "poverty": 8.637730612244896 }, "correlation": { "income": -0.8352655256272504, "poverty": 1.0 } }] }
  • 65. 65 Geo Aggregations more available, and coming soon... • pca - ML foundation plugin ‒ dimensionality reduction ‒ image analysis ‒ classification / recognition • geo_stats - In work... ‒ Moran’s I - measuring spatial auto-correlation ‒ Getis-Ord - spatial hot spot analysis
  • 66. 66 Now GA in Elastic 7.3….
  • 67. THANK YOU Follow for updates: @nknize apache/lucene-solr