GPUs, Cloud and Grids: Distributed Geoprocessing for Speed, Scalability and Better Living
Download as PPT, PDF1 like950 views
The document discusses advancements in distributed geoprocessing using GPUs, clouds, and grids to enhance speed and scalability for geospatial tasks. It highlights the challenges posed by growing geographic data and the need for optimized processing methods, including new programming technologies and data management strategies. The presentation underscores the significant improvements in calculation times and the potential of GPU geoprocessing for various applications, including urban planning and environmental forecasting.
![Additional [Experimental] Measures Tiling Pyramids EC2 for planned peaks – NYC Big Apps HTTP file caching - Varnish](https://image.slidesharecdn.com/20110218ncgisdistributedcomputing-v2-110219160124-phpapp01/85/GPUs-Cloud-and-Grids-Distributed-Geoprocessing-for-Speed-Scalability-and-Better-Living-42-320.jpg)
GPUs, Cloud and Grids: Distributed Geoprocessing for Speed, Scalability and Better Living
- 1. GPUs, Clouds and Grids: Distributed Geoprocessing for Speed, Scalability and Better Living Robert Cheetham 17 February 2011 NC GIS 2011
- 2. About Azavea Founded in 2000 27 people software engineers spatial analysts project managers Web & Mobile apps Spatial Analysis R&D High Performance Computing User Experience
- 3. B Corporation 10% Research Program Pro Bono Program Time-to-Give-Back Program Employee-focused Culture Projects with Social Value
- 5. Classic GIS Use Case ...
- 6. Close to Center City Walk to Grocery Store Nearby Restaurants Library Near a Park Biking / walking distance from our work Biking distance to fencing club somewhat important vital very important nice to have somewhat important very important somewhat important Robert’s Rules of Housing
- 7. Child Care Local School Rankings Farmer's Market PhillyCarShare Public Transit Your Factors might include…
- 8. Tax Incentives Commercial Corridor Health Public Transit Car Share Open Space Farmers’ Markets Street Network Density Recycling Participation Walkability Sustainability Factors
- 9. Not a new idea… Ian McHarg
- 10. Not a new idea … Design with Nature
- 11. Not a new idea … Map Algebra
- 12. Desktop GIS
- 13. x 5 x 2 x 3 x 1 + + + = Generate Output Heat Map
- 14. The Web is different from the Desktop Lots of simultaneous users Stateless environment HTML+JS+CSS Users are less skilled Users are less patient
- 15. ArcGIS Server Flex, Silverlight and JS API’s Publish tasks and models Caching Optimized MSD files
- 16. But wait … there’s a problem 10 – 60 second calculation time Multiple simultaneous users … … that are impatient
- 17. User Interface version 1
- 24. Reports
- 25. Reports
- 35. NYC Big Apps Submission
- 36. Specific Optimization Goals New Raster File format Distributed processing Binary messaging protocol
- 37. Optimization: File Format Simple - strip out metadata Limit data type and range 1D arrays are fast to read/write Assume Same extent Same cell size Same pixel data type Same cell alignment Same projection Azavea Raster Grid (ARG)
- 38. Optimization: Distributed Processing Parallelizable - Local Ops and Focal Ops Support multiple Threads Cores CPU’s Machines Considered Hadoop Amazon Map Reduce Beowolf
- 40. Binary Messaging Protocol Started with XML Binary Protocol Buffer is better simpler 3 to 10 times smaller 20 to 100 times faster less ambiguous a bit easier to use programmatically Considered AMF Google Protocol Buffer
- 41. Success!! Reduced from 10-60 seconds to <500 milliseconds
- 42. Additional [Experimental] Measures Tiling Pyramids EC2 for planned peaks – NYC Big Apps HTTP file caching - Varnish
- 43. Optimizing one process sub-optimizes others Complex to configure and maintain One type of operation No interpolation No mixing cell sizes No mixing extents No mixing projections No Map Algebra No ModelBuilder etc.
- 44. High Performance Geoprocessing 2.0 More generic Cache data – memory is cheaper New programming technology
- 45. OMB Watch: Federal Spending Equity
- 46. High Performance Geoprocessing 2.0 Reduced calculation time to ~40ms
- 47. GPU geoprocessing research But wait, there’s more…
- 50. New languages CUDA OpenCL DirectCompute Re-write every algorithm Hardware Diversity Challenges
- 51. We re-wrote a few Map Algebra operations: Local Neighborhood Zonal Viewshed etc. 15 – 120x speed improvement Large grids Large neighborhoods Results
- 52. Not for the faint of heart
- 53. Sea Level Rise
- 54. Crime Analysis, Early Warning and Forecasting
- 55. Hunch Helper
- 56. Risk Forecasting
- 57. Animation
- 61. Food, Culture and Sustainability
- 63. The Future Clouds of Processors - Google App Engine Faster is different
- 64. Summary Challenges Geographic data growth rates are exploding Size of data sets is growing Lidar Raster Social Media New form factors that are less powerful Distributed data sets Larger numbers of less technical users New Options Clouds of processors Clouds of virtual machines GPUs
- 65. Many Thanks! © Photo used with permission from Alphafish , via Flickr.com
- 66. GPUs, Clouds and Grids: Distributed Geoprocessing for Speed, Scalability and Better Living Robert Cheetham 17 February 2011 NC GIS 2011
Editor's Notes
- #6: So this story starts with my wife and I looking for a house and being frustrated by the type of information we had access to. We didn't know where to start. Each of the real estate agents we met knew a particular part of the city really well, so they tended to steer us toward those houses.
- #7: We didn't own a car, so we wanted to find a place that was * Close to Center City (somewhat important) * Walking distance to a grocery store (vital) * Near some restaurants (very important) * Walk to a library (nice to have) * Near a Park (yes!) * Close to fencing * Biking / walking distance from our work
- #8: The factors you consider important are probably not the same as ours. Maybe yours are related to child care or rankings for local schools or being near a farmer's market. Or maybe you want to be close to PhillyCarShare or public transit, but don't want to be too close to downtown.
- #9: We selected a variety of factors that contribute to sustainability, ranging from location in a state or federal tax incentive zone to environmental amenities like tree canopy to transit considerations like access to bus and regional rail lines. Retail businesses targeting markets may be interested in demographic factors like age and per capita income and proxies for environmental engagement like recycling participation.
- #10: Many of the ideas here are not new. Actually, they were developed here in Philadelphia by Ian McHarg, who was chair of the Landscape Architecture and Regional Planing department at the University of Pennsylvania
- #11: He wrote a book in 1969 called Design with Nature, and focused on sustainable and ecological design. Among other concepts, he described how a series of inputs drawn on transparent acetate sheets. Could be combined as a set of map overlays to identify the best site for a particular facility, road or whatever.
- #12: Now fast forward to the 1990’s. We’re interested in digital maps. This approach to compositing several digital maps was developed further by Dana Tomlin, who is now also a professor at University of Pennsylvania. Tomlin developed the computational vocabulary to perform this type of work – he called it Map Algebra. He’s also a really great teacher, so if you ever get a chance to take a class with him. Do it. He’s great.
- #13: So, these days people do this kind of work using desktop GIS systems. You are looking at the ArcMap application from ESRI
- #27: Export to KML
- #29: scenarios
- #30: Heat map
- #59: Export to KML
- #60: Export to KML
- #61: Export to KML