New Trends of Geospatial Services on AWS Cloud - Channy Yun :: ICGIS 2015 Seoul
12 likes5,179 views
This document discusses new trends in geospacial services based on AWS cloud. It covers GIS services, open GIS data, Landsat data, and planetary research available on AWS. Specifically, it discusses how ESRI's ArcGIS Server, global open data sources, the entire Landsat satellite image archive, and NASA planetary research projects are now supported and hosted on AWS cloud infrastructure. This wider availability of geospacial services and data on AWS is enabling more innovation and applications in fields like government GIS, natural resource monitoring, and space exploration.
New Trends of Geospatial Services on AWS Cloud - Channy Yun :: ICGIS 2015 Seoul
- 1. New Trends of Geospacial Services based on AWS Cloud Channy Yun ! ! Tech Evangelist, Amazon Web Services Korea! ! @channyun!
- 2. • Cloud Computing and AWS • New trends on GIS on Cloud 1. GIS Services on AWS 2. Open GIS Data on AWS 3. Landsat on AWS 4. Planetary Research on AWS Agenda
- 3. Cloud Computing and AWS
- 4. Utility Computing If computers of the kind I have advocated become the computers of the future, then computing may someday be organized as a public utility just as the telephone system is a public utility... The computer utility could become the basis of a new and important industry.! ! —John McCarthy, speaking at the MIT Centennial in 1961!
- 5. What is Cloud Computing? On-Demand! Uniform! Pay as you go! Available! http://aws.amazon.com/ko/what-is-cloud-computing/ ! "Cloud Computing", by definition, refers to the on-demand delivery of IT resources and applications via the Internet with pay-as- you-go pricing.
- 6. The Growth of AWS Over 1 million active customers! “Active customer” is defined as a non-Amazon customer with AWS account usage activity in the past month, including the free tier! Over 190 Countries Startup, Enterprise and Public Sectors 2008! 2009! 2010! 2011! 2012! 2013! 2014!
- 8. Why do researchers love using AWS? Time to Science! Access research! infrastructure in minutes! Low Cost! Pay-as-you-go pricing! Elastic! Easily add or remove capacity! Globally Accessible! Easily Collaborate with ! researchers around the world! Secure! A collection of tools to! protect data and privacy! Scalable! Access to effectively! limitless capacity!
- 9. US-WEST (Oregon) EU-WEST (Ireland) AP-NORTHEAST (Tokyo) US-WEST (N. California) SA-EAST (Sao Paulo) US-EAST (Virginia) AWS GovCloud (US) AP-SOUTHEAST (Sydney) AP- SOUTHEAST (Singapore) CHINA (Beijing) EU-CENTRAL (Frankfurt) 11 Regions 30 Availability Zones 53 Edge AWS Global Infra Structure
- 10. The AWS’ History of Innovation AWS has been continually expanding its services to support virtually any cloud workload, and it now has more than 50 services that range from compute, storage, networking, database, analytics, application services, deployment, management and mobile. 2009 Amazon RDS Amazon VPC Auto Scaling Elastic Load Balancing 2010 Amazon SNS AWS Identity & Access Management Amazon Route 53 2011 Amazon ElastiCache Amazon SES AWS CloudFormation AWS Direct Connect AWS Elastic Beanstalk GovCloud 2012 Amazon SWF Amazon Redshift Amazon Glacier Amazon Dynamo DB Amazon CloudSearch AWS Storage Gateway AWS Data Pipeline 2013 Amazon CloudTrail Amazon CloudHSM Amazon WorkSpaces Amazon Kinesis Amazon Elastic Transcoder Amazon AppStream AWS OpsWorks 2015 Amazon EFS Amazon API Gateway Amazon WorkMail Amazon Machine Learning AWS Device Farm 2014 AWS KMS Amazon Config Amazon Cognito Amazon Mobile Analytics Amazon ECS Amazon RDS for Aurora Amazon Lambda Amazon WorkDocs AWS Directory Service AWS CodeCommit AWS CodePipeline * As of 1 Sept 15!
- 11. AWS Services – Deep and Various workloads TECHNICAL & BUSINESS SUPPORT! Account Management! Support! Professional Services! Solutions Architects! Training & Certification! Security & Pricing Reports! Partner Ecosystem! AWS! MARKETPLACE! Backup! Big Data! & HPC! Business Apps! Databases! Development! Industry! Solutions! Security! MANAGEMENT! TOOLS! Queuing! Notifications! Search! Orchestration! Email! ENTERPRISE! APPS! Virtual! Desktops! Storage! Gateway! Sharing &! Collaboration! Email &! Calendaring! Directories! HYBRID CLOUD! MANAGEMENT! Backups! Deployment! Direct! Connect! Identity! Federation! Integrated! Management! SECURITY &! MANAGEMENT! Virtual Private! Networks! Identity &! Access! Encryption! Keys! Configuration! Monitoring! Dedicated! INFRASTRUCTURE! SERVICES! Regions! Availability! Zones! Compute! Storage Databases! SQL, NoSQL, Caching! CDN!Networking! PLATFORM! SERVICES! App! Mobile ! & Web! Front-end! Functions! Identity! Data Store! Real-time! Development! Containers! Source! Code! Build! Tools! Deployment! DevOps! Mobile! Sync! Identity! Push! Notifications! Mobile! Analytics! Mobile! Backend! Analytics! Data! Warehousing! Hadoop! Streaming! Data! Pipelines! Machine! Learning!
- 12. 24! 48! 61! 82! 159! 280! 516! 342! 2008! 2009! 2010! 2011! 2012! 2013! 2014! 2015! The AWS pace of Innovations +*! *July 9th 2015 AWS has launched a total of 431 new features and/or services to-date* in 2015, for a total of 1,604 new features and/or services since inception in 2006.
- 13. Gartner Magic Quadrant for Cloud Infrastructure as a Service, Worldwide Gartner “Magic Quadrant for Cloud Infrastructure as a Service, Worldwide,” Lydia Leong, Douglas Toombs, Bob Gill, May 18, 2015. This Magic Quadrant graphic was published by Gartner, Inc. as part of a larger research note and should be evaluated in the context of the entire report. The Gartner report is available at http://aws.amazon.com/resources/analyst-reports/. Gartner does not endorse any vendor, product or service depicted in its research publications, and does not advise technology users to select only those vendors with the highest ratings or other designation. Gartner research publications consist of the opinions of Gartner's research organization and should not be construed as statements of fact. Gartner disclaims all warranties, expressed or implied, with respect to this research, including any warranties of merchantability or fitness for a particular purpose.!
- 14. An Expansive Ecosystem on Cloud 5,000+ SIs & Consultants 3,000+ ISVs 22 Global Premier Tier partners 6 Enterprise-focused competencies 2,000+ products available for 1- click deployment across 23 distinct product categories Customers run over 70M hours of software per month
- 15. What Sets AWS Apart? Building and managing cloud since 2006 40+ services to support any cloud workload History of rapid, customer-driven releases 11 regions, 30 availability zones, 53 edge locations 49 proactive price reductions to date 8,000+ SIs and ISVs; 2,000+ Marketplace products Experience Service Breadth & Depth Pace of Innovation Global Footprint Pricing Philosophy Ecosystem
- 16. 1. GIS Services on AWS New trends on GIS on Cloud!
- 17. ArcGIS Server on AWS “ESRI users on Amazon Web Services get Access to unlimited support for high computing power volume sites with ideal environment of cost effectiveness for developers” (2012)! http://www.slideshare.net/AmazonWebServices/esri-12135108 !
- 18. Enterprise GIS Service 1) Setting EC2 Instance and Auto-scaling !
- 19. 3) Setting GeoDatabases! 2) Setting EC2 Instance type and Region! Enterprise GIS Service
- 20. https://media.amazonwebservices.com/AWS_ESRI_Mapping_GeoSpatial_Analysis_Using_ArcGIS.pdf ! Option 1: On-Premises and Cloud Hybrid! Option 2: Pure Cloud Solution! Enterprise GIS Service
- 21. Listing on AWS Marketplace Atlas comes bundled with global street and satellite layers and allows you to upload custom layers from Mapbox Studio. ! ! It includes Mapbox.js and Turf so you can build interactivity, add data feeds, and run GIS operations inside your mapping applications.!
- 22. 2. Open GIS Data on AWS New trends on GIS on Cloud!
- 23. Open data is data that can be used by anyone for any purpose for free. What is open data? Many of our customers, such as Esri, the Weather Company, and the Climate Corporation, rely on quality open data as much as they rely on our computing, storage, and other web services.
- 24. Amazon Web Services provides a comprehensive toolkit for gathering, storing, analyzing, and working with data at any scale. Open data on AWS Amazon Elastic MapReduce (Amazon EMR) provides the Apache Hadoop analytics framework as an easy-to-use managed service.! Amazon S3 lets you store and retrieve any amount of data, at any time, from anywhere on the web. ! Amazon DynamoDB is a fully-managed NoSQL database service that makes it cost-effective to store and retrieve any amount of data.!
- 25. Making data open on AWS enables more innovation by making data available for rapid access to our flexible and low-cost computing resources. The power of open data on AWS Amazon! EC2! Amazon! EMR! Amazon! Redshift! Amazon! DynamoDB! AWS! Lambda!
- 26. The Weather Company saves $1 million per year running its forecasting application on AWS The Weather Company provides millions of people with the world’s best weather forecasts, content and data, every day. Using AWS, TWC can scale as necessary to handle constantly changing workloads and maintain our 11-millisecond response time. Bryson Koehler EVP, CTO, CIO, The Weather Company ” “ • Needed a cost-effective, scalable alternative to operating 13 data centers with legacy systems. • TWC ingests, stores, and analyzes ingests 4 GB of weather data per second from over 800 sources. • Designed to handle more than 15 billion API calls each day, at a rate of 150,000 per second. • Reduced its on-premises IT environment form 13 to 6 data centers.
- 27. Data Enrichment Sensemaking Data at Rest (Object storage) Basic APIs Complex APIs Consumer applications Algorithmic policy Data-driven journalism Data Catalogs Focused data dashboards Predictive modeling Visualizations Lower cost of knowledge (Efficiency) Open data as a platform
- 28. Data Creation Data Enrichment Sensemaking Data at Rest (Object storage) Basic APIs Complex APIs Consumer applications Algorithmic policy Data-driven journalism Data Catalogs Focused data dashboards Predictive modeling Visualizations Efficiency Open data as a platform
- 29. Data Enrichment Sensemaking Amazon Kinesis Amazon EC2 Amazon EC2 AWS Data Pipeline Amazon S3 Amazon RDS Amazon EMR Amazon Redshift Amazon DynamoDB AWS Lambda Open data as a platform Many of our customers, such as ESRI, the Weather Company, and the Climate Corporation rely on quality open data as much as they rely on our computing, storage, and other services.!
- 30. • Mobile app turns bus and train riders into real-time sensors for city government • Integrates with city back-end systems to improve both service and rider experience • Powered worldwide by the AWS cloud Moovit: Smart Public Transportation
- 31. • First government-wide national intelligent map portal • Integrated map system for government agencies to deliver location-based services and information to government agencies and citizens • Powers over 100 government GIS websites and applications • Reduced costs by 60% Singapore government “AWS has helped my organization to provide better service availability and handle higher traffic load at a lower cost.” ! —Chan Chin Wai, Chief Information Officer Singapore Land Authority!
- 32. 3. Landsat on AWS New trends on GIS on Cloud!
- 33. To enable more innovation, AWS hosts a selection of datasets that anyone can access for free. Data in our public datasets is available for rapid access to our flexible and low-cost computing resources. Public datasets on AWS Earth Science! NASA Earth Exchange! (NASA NEX)! Life Sciences! 1000 Genomes Project! Internet Science! Common Crawl Corpus!
- 34. The Landsat program is a joint effort of the U.S. Geological Survey and NASA. It is the longest running program to gather Earth imagery from space and is considered the gold standard for natural resources satellite imagery. Landsat
- 35. The Landsat program is a joint effort of the U.S. Geological Survey and NASA. It is the longest running program to gather Earth imagery from space and is considered the gold standard for natural resources satellite imagery. It has traditionally been time- consuming and expensive to acquire, store, and analyze Landsat data. Landsat is big open data
- 36. We have committed to making up to 1 petabyte of Landsat imagery readily available as objects on Amazon S3. Now, anyone can analyze Landsat data at web scale with no significant up-front investment of time or capital expense. Landsat on AWS
- 37. Esri has created a tool to show how ArcGIS Online can quickly visualize Landsat data for live visualization and analysis within the browser. “These are not pre-generated cache services limited to just visualization— they are dynamic, high-performance image services that perform on-the-fly processing and dynamic mosaicking of Landsat’s multi-spectral and multi- temporal imagery.” ESRI —Unlock Earth’s Secrets http://www.esri.com/landsatonaws !
- 38. Landsat on AWS helped Development Seed make optimizations that make landsat- util over 2× faster and allow for more functionality. landsat-util https://developmentseed.org/blog/2015/03/19/aws-landsat-archive/ ! !
- 39. MathWorks created a freely downloadable MATLAB based tool for accessing, processing, and visualizing Landsat 8 data. The tool allows MATLAB users to find Landsat 8 scenes, analyze them, and combine them with other sources of GIS data for new visualizations. MATLAB—Landsat8 Data Explorer http://blogs.mathworks.com/steve/2015/03/19/matlab-landsat-8-aws/ !
- 40. Mapbox created Landsat-live, a map that is constantly refreshed with the latest satellite imagery from NASA’s Landsat 8 satellite. Creating a live Earth imagery pipeline is possible because Landsat imagery is available on Amazon S3 within hours of creation. Rendering a Seamless Satellite Map https://www.mapbox.com/blog/landsat-live-live/ ! !
- 41. • Quickly process massive amounts of data • Distribute processed data to users around the world quickly and reliably • Processing ü Massive storage for raw and processed data ü Massive computing that we can spin up and down in minutes ü Everything must be fully automated ü Low cost Rendering a Seamless Satellite Map In the past 30 days we have served 9.8 billion requests!
- 42. Amazon EC2 Low-cost, scalable computing! Amazon S3 Data storage for input data and processed output! Auto Scaling Controls the number of worker EC2 instances! Amazon SQS Manages the queue of work! Rendering a Seamless Satellite Map
- 43. Rendering a Seamless Satellite Map NASA Server Source S3 Bucket Watcher Instance Auto Scaling group SQS Queue Worker Instances Destination S3 Bucket Processed Outputs Running 200 c3.xlarge SPOT instances $25,618 ! in savings per month! Instancereservations! Buy computing up front for long-running instances ! Save up to 60% or more over the course of a year!
- 44. 4. Planetary Research on AWS New trends on GIS on Cloud!
- 45. NASA JPL Web Sites Mars Exploration Program! Mars.jpl.nasa.gov! Eyes on the Solar System! Eyes.jpl.nasa.gov! Night Sky Network! Nightsky.jpl.nasa.gov!
- 46. Massive Data Storage for Disaster Recovery Mars Exploration Rovers! Station Fires !
- 47. Mars Science Laboratory – Live “The 42 pack”!
- 48. Mars Science Laboratory – Global Web
- 50. Opportunity Pancam mosaic of total size 11280×4280 pixels containing 77 color images!
- 51. Mastcam instrument on Mars Science Laboratory consist of up to 1296 images and have a resolution of almost 2 Gigapixels!
- 52. Massive Image Processing on AWS
- 53. • Cloud Computing and the world of GeoApps https://aws.amazon.com/blogs/aws/cloud-computing-and-the-world-of-geoapps/ • ESRI and Amazon Web Services https://aws.amazon.com/solutions/global-solution-providers/esri/ • Mapbox’s use cases on AWS Cloud • https://www.mapbox.com/blog/tags/aws/ • OSGeo and AWS • https://sites.google.com/site/geoecodocs/osgeo-and-aws • GEO865 Cloud and Server GIS (PennState) https://www.e-education.psu.edu/cloudGIS/node/20 What’s Next?
- 54. AWS Research Grants AWS provides free usage credits to help researchers: • Explore new projects • Create resources for the scientific community https://aws.amazon.com/grants
- 55. AWS Educate Program AWS provides free usage credits and training resource to help educators and students: • Make a course for students • Teach cloud based GIS training https://awseducate.com
- 56. We’re waiting for your feedback! channyun@amazon.com @AWSKorea AmazonWebServices AWSKorea !