Implementing Netezza Spatial
2 likes2,894 views
The document discusses spatial data and analysis. It defines spatial data as information that can be analyzed based on geographic context, such as locations, distances and boundaries. It then describes the three common types of spatial data - points, lines and polygons - and how they are used to answer questions about proximity and relationships between objects. Finally, it outlines some of the key sources for spatial data, challenges in working with spatial data, and provides a model for how to deliver spatial data and analysis.
Implementing Netezza Spatial
- 1. in conjunc(on with Data Management & Warehousing http://www.datamgmt.com
- 2. What is the Spa(al Module? • It’s the ability to analyse informa(on in a geographic context: – Where is the nearest petrol sta(on? – Which road am I on? – How many ATMs are in this area? • It’s not maps and images – These come later with tools that help present the informa(on Wednesday, July 28, 2010 © 2010 Data Management & Warehousing 2
- 3. The three types of data & many ques(ons • Points • How close are two – OS Grid points? – La(tude & Longitude • Does a point touch a • Lines line? – Pairs of points • Is a point inside or – e.g. Road Segments outside a polygon? • Polygons • Does a line cross a – A series of points that polygon? define a boundary • How many points are in – e.g. Postcode Boundaries a polygon? Wednesday, July 28, 2010 © 2010 Data Management & Warehousing 3
- 4. Using Spa(al Data Is Complex • Different distances between points at different longitudes and la(tudes • Measurement over a curved irregular surface • Mul(ple input and output formats • Mul(ple co-‐ordinate systems see: A Guide to Coordinate Systems in Great Britain Wednesday, July 28, 2010 © 2010 Data Management & Warehousing 4
- 5. Sources of Informa(on – GPS • In Car Device – Sends frequent data sets to processing centre – Point Data • Speed, Direc(on, Loca(on and G-‐force – Aggregate Data • Speed and Direc(on • Other Devices – Sat Nav Systems – Smart Phone Apps e.g. ‘GPS Tracker’ – Cameras Wednesday, July 28, 2010 © 2010 Data Management & Warehousing 5
- 6. Sources of Informa(on – Ordnance Survey • Integrated Road Network: A series of 3 million ‘linestrings’ and 17 million points that describe every road in the UK • Linestrings have between 2 and 655 points, most have less than 10 • 23 points for this picture Wednesday, July 28, 2010 © 2010 Data Management & Warehousing 6
- 7. Sources of Informa(on – Post Office/GAdm • Postal Address File: A series of c.1.75M UK postcodes – Postcode Boundaries – Over 28M complete addresses • Global Admin Boundaries – Na(onal and regional boundaries for c.245 countries – hgp://www.gadm.org Wednesday, July 28, 2010 © 2010 Data Management & Warehousing 7
- 8. Data Layers – Enriching what you have • Data Layers are sets of informa(on (ed to a geographic point – Road Speed for a given road segment – ATM Loca(on – House Price for a postcode • Where data has loca(on informa(on it is known as ‘Geo-‐tagged’ Wednesday, July 28, 2010 © 2010 Data Management & Warehousing 8
- 9. Data Layer Sources (1) • Ordnance Survey – Road Types, Limits, Closures, etc. • Government – UK Government now providing masses of geo-‐tagged info (hgp://data.gov.uk) • Met Office / HM Nau(cal Almanac Office – Weather, Daylight to Postcode Level Wednesday, July 28, 2010 © 2010 Data Management & Warehousing 9
- 10. Data Layer Sources (2) • Wikipedia – Geo-‐tag Access API – what’s nearby? • Google Maps – Road level photographic images • Commercial Sources – Fast Food Outlets, Supermarkets, Petrol Sta(ons, ATMs, etc. • Massive growth in both commercial and public domain geo-‐tagged data Wednesday, July 28, 2010 © 2010 Data Management & Warehousing 10
- 11. Issues with Geo-‐tagged data • Geo-‐tagging uses different formats – Longitude & La(tude, OS Grid Reference, etc • Geo-‐tagging at different levels – Data for a postcode or a an en(re county which makes it difficult to compare • Geo-‐tagging coverage is patchy and/or historic – Rate of change of fine detail data is very high – e.g. OS issues monthly updates to the UK mapping • Mul(ple standards and formats – XML & CSV, different file formats, etc. Wednesday, July 28, 2010 © 2010 Data Management & Warehousing 11
- 12. Our Model For Delivering Spa(al Data Source 1. Load Mul(ple File Formats Netezza 2. Standardise Geo-‐Tagging 3. Extract & Load CSVs (Proximity, Contains, Excludes) (Tableau, Google Maps, etc.) Query & Presenta(on Tools Source 4. Perform Spa(al Analysis (Sets of data with spa(al Spa(al Presenta(on 5. Create User Access Area Spa(al Analysis agributes) Source Source (Small) 1 Postgres 3 Database Source 2 Source 4 5 Wednesday, July 28, 2010 © 2010 Data Management & Warehousing 12
- 13. Netezza Spa(al Value Add • Netezza Spa(al is fast • Netezza Spa(al is easy – Analysis – Distance and proximity • Look up a typical 18 point calcula(ons are simple trip in the 3M linestrings to – ‘Touches’, ‘Overlaps’ & find the roads that the ‘Contains’ queries allow vehicle was on in less than 1 second instant value add • Overnight batch process of 300,000 points to matching • Netezza Spa(al integrates road names in under 30 minutes – Works well with Tableau – Presenta(on – Easy to generate KML for • Tools rely on fast query use with Google Earth and access to render any Google Maps queried map with sub-‐ second response (mes Wednesday, July 28, 2010 © 2010 Data Management & Warehousing 13
- 14. Netezza Spa(al Limita(ons • Fails the Slar(barpast Test: – Polygons for very detailed maps are too big to be loaded as Netezza limits the maximum block size to 64000 characters Norway – Named aqer the Hitch-‐Hikers Guide to the Galaxy coastline designer responsible for the twiddly bits around the Norwegian rords • Work-‐around: – Use regional boundaries (e.g. UK Coun(es, US States, etc.) and then aggregate into na(onal boundaries – If a point is in Berkshire then by Slar(barpast defini(on it is also in England Wednesday, July 28, 2010 © 2010 Data Management & Warehousing Page 14
- 15. Current Uses … • M/A/B road driving profiles • Time of day driving profiles • Speed Limits vs. Driven Speed • Matching GPS posi(ons to road names • Out of bounds driving • Customer Demographic Profiles … but this is only the start in a very short (me Wednesday, July 28, 2010 © 2010 Data Management & Warehousing 15
- 16. in conjunc(on with Data Management & Warehousing http://www.datamgmt.com