Multi-thematic spatial databases
1 like1,163 views
The document discusses multi-thematic spatial databases for efficiently storing, accessing, processing, and visualizing large volumes of geospatial data from multiple sources and sensors. It describes experience with designing databases to handle terabytes of temporal, multi-sensor data using spatial indexing. The goals are a unified approach for multi-thematic data storage, efficient data handling, and enabling searches across time, space and attributes while incorporating visualizations.
Multi-thematic spatial databases
- 1. Multi-Thematic Spatial Databases Experience designing and implementing Dr. Conor Mc Elhinney Dr. Paul Lewis Postdoctoral Researcher Mobile Mapping Group
- 2. What we do Store
- 3. What we do Store Access
- 4. What we do Store Access Process
- 5. What we do Store Access Process Visualise
- 6. What we do Store Access Voluminous Geospatial Data Process Visualise
- 7. Mobile Mapping Systems Group 1 Senior Researcher 2 Post-docs 2 PhDs Computer Science, GIS, Surveying
- 8. Mobile Mapping Systems Group
- 9. Mobile Mapping Systems Group
- 10. Mobile Mapping Systems Group
- 11. Mobile Mapping Systems Group
- 12. i2maps Group 1 Senior Researcher 1 Post-doc 2 PhDs +1 PHD GeoCrowd Computer Science, Maths, GIS Dr. Alexei Pozdnoukhov, Dr. Christian Kaisler Fergal Walsh
- 13. Learning from data Streams i2maps What we do at NCG: Scalable methods of spatial analytics
- 14. Learning from data Streams i2maps What we do at NCG: Scalable methods of spatial analytics Machine learning and data mining
- 15. Learning from data Streams i2maps What we do at NCG: Scalable methods of spatial analytics Machine learning and data mining Stream Handler {x, y} M f (.) i K (., xi ) Analysis and i 1 Modeling new { i} (.), ( xi ) H Dictionary of models MapReduce xOLD
- 16. Learning from data Streams i2maps What we do at NCG: Scalable methods of spatial analytics Machine learning and data mining Distributed approaches to spatial statistics Stream Handler {x, y} M f (.) i K (., xi ) Analysis and i 1 Modeling new { i} (.), ( xi ) H Dictionary of models MapReduce xOLD
- 17. What we have experience with Handling and proccessing of TBs of
- 18. What we have experience with Handling and proccessing of TBs of Multi-thematic data
- 19. What we have experience with Handling and proccessing of TBs of Multi-thematic data Temporal data
- 20. What we have experience with Handling and proccessing of TBs of Multi-thematic data Temporal data Multi-sensor data
- 21. What we have experience with Handling and proccessing of TBs of Multi-thematic data Temporal data Multi-sensor data using spatial information
- 22. Data Handling
- 23. Data Handling
- 24. LiDAR A laser scanning tech, laser radar
- 25. LiDAR A laser scanning tech, laser radar > 20Gbs an hour
- 26. LiDAR A laser scanning tech, laser radar > 30Gbs an hour after geocoding
- 27. LiDAR A laser scanning tech, laser radar > 30Gbs an hour after geocoding > 6 attributes output from scanner
- 28. LiDAR A laser scanning tech, laser radar > 30Gbs an hour after geocoding > 6 attributes output from scanner Between 1 pt/m2 and 2000 pt/m2
- 29. LiDAR
- 30. LiDAR
- 31. LiDAR
- 32. LiDAR
- 33. LiDAR
- 34. Imagery Developing trend to store as videos / blobs
- 35. Imagery Developing trend to store as videos / blobs Store metadata in SDB
- 36. Imagery Developing trend to store as videos / blobs Store metadata in SDB Using HTML5 or queries can link to frames
- 37. User Generated Comments
- 38. User Generated Comments Video / Imagery
- 39. User Generated Comments Video / Imagery Opinion
- 40. User Generated Comments Video / Imagery Opinion
- 41. Twitter
- 42. Twitter
- 43. Twitter 25% contain links
- 44. Facebook 500 Million Active Users
- 45. Facebook 250 Million Active Mobile Users
- 46. Facebook 10 Million pieces of content per day
- 48. Hard Drive Capacity Capacity is increasing linearly
- 49. CPU vs HD speed CS111 UCLA 2006
- 50. CPU vs HD speed We can process more than we can store CS111 UCLA 2006
- 51. CPU vs HD speed IDC - The Diverse and Exploding Digital Universe CS111 UCLA 2006
- 52. What next Process the data in real time
- 53. What next Process the data in real time Extract or compress to find a model for the relevant data
- 54. What next Process the data in real time Extract or compress to find a model for the relevant data Store this model for future processing
- 55. What next Process the data in real time Extract or compress to find a model for the relevant data Store this model for future processing Leads to the problem of what do we model and store?
- 57. Processing data streams Human activity on geo-referenced communication networks
- 58. Processing data streams Human activity on geo-referenced communication networks
- 59. Processing data streams Human activity on geo-referenced communication networks At least two categories we need to understand:
- 60. Processing data streams Human activity on geo-referenced communication networks At least two categories we need to understand: Dynamics of links
- 61. Processing data streams Human activity on geo-referenced communication networks At least two categories we need to understand: Dynamics of links Activity level at nodes
- 62. Enabling data speak for themselves Air Quality Sensor Weather Measurements VGI Feed (e.g. Twitter) Surveillance Camera SMS Web Page XML Video Push Polling Stream Stream Data Receiver Data Crawler Stream Handler Stream Handler Static Data Static Data Static Data Analysis and Spatial Database Modeling Spatio-Temporal Data i2maps Web Service KML/CSV/etc GeoJSON Spatio-Temporal Queries Interactive Spatio-Temporal Information Visualiser
- 63. Enabling data speak for themselves Air Quality Sensor Weather Measurements VGI Feed (e.g. Twitter) Surveillance Camera SMS Web Page XML Video Push Polling Stream Stream Data Receiver Data Crawler Stream Handler Stream Handler Spatial Static Data Static Data Static Data Analysis and Database Spatial Database Dictionary of models Modeling Spatio-Temporal Data i2maps Web Service KML/CSV/etc GeoJSON Spatio-Temporal Queries Interactive Spatio-Temporal Information Visualiser
- 64. Storage
- 65. What exists Files / DBs / SDBs
- 66. What exists Files / DBs / SDBs Files still extremely common
- 67. What exists Files / DBs / SDBs Files still extremely common SDBs are what is needed
- 68. What exists Files / DBs / SDBs Files still extremely common SDBs are what is needed Multi-source, sensor, type data
- 69. Our Aims Unified approach to storing multi- thematic data
- 70. Our Aims Unified approach to storing multi- thematic data Efficient data upload / access/ storage
- 71. Our Aims Unified approach to storing multi- thematic data Efficient data upload / access/ storage Searchable in Time/ Space / by Attributes
- 72. Our Aims Unified approach to storing multi- thematic data Efficient data upload / access/ storage Searchable in Time/ Space / by Attributes Incorporating Visualisations into all solutions
- 73. Our hardware 3 Processing Servers 8 Intel Xeons, 2.1- 2.8 GHz 72 GBs RAM
- 74. Our hardware 3 Processing Servers 8 Intel Xeons, 2.1- 2.8 GHz 72 GBs RAM 1 Storage Server 7TBs Raided Drives
- 75. Our Developed Systems LiDAR / Image based SDB
- 76. Our Developed Systems LiDAR / Image based SDB GeoComputation Platform
- 77. Database storage experience Optimisation of upload of large (GBs) spatial files to SDB.
- 78. Database storage experience Database optimisation to suit system architecture
- 79. Database storage experience Storage of multiple data types/sources
- 80. Watch out for Spatial Index size V RAM
- 81. Watch out for Spatial Index size V RAM Expected no. of concurrent users
- 82. Watch out for Spatial Index size V RAM Expected no. of concurrent users HD capacity V daily data throughput