SlideShare a Scribd company logo
MIT 
Cambridge, September 2014 
A Spatial Platform 
to Support Research and Collaboration 
Ben Lewis 
blewis@cga.harvard.edu 
http://worldmap.harvard.edu
Outline 
• Why lat/long is valuable and what research 
support operations should do about it 
• WorldMap as straw man 
– Overview of current system 
– New projects 
• Global services registry (OGP integration?) 
• Dataverse integration 
• HarvardX integration 
• Neatline integration 
• Discussion
Lat/long (location) as organizing facet 
(http://www.architectmagazine.com/technology/meet-the-geodesigner.aspx
Spatial search 
• Improve access to both spatial and non spatial 
materials with a “spatial visualization facet” 
– Books, documents 
– Paper maps 
– Local spatial data 
– Remote spatial data 
– Local map display and processing services 
– Remote map display and processing services 
– Wikipedia 
– Social media – tweets, etc 
– The web
Build platforms that lower barriers to: 
• Finding spatial materials which reside inside or 
outside the academy 
• Visualizing this data 
• Using this data 
• Sharing this data and views of it 
• Mashing up private and public data 
• Spatializing materials that are not yet spatial
Support spatial and spatio-temporal 
visualization 
• Make it easy to visualize datasets on a map 
(A great way to see temporal patterns is on a map 
http://worldmap.harvard.edu/tweetmap ) 
• Make it easy to mashup private and public data 
• Make it easy to share spatial data and views of data 
• Make it easy to crowd curate data
Support the research life cycle 
* Research Lifecycle: 
• Scoping 
• Data gathering / exploration 
• Synthesis / analysis 
• Writing / communication 
• Publishing / delivery 
Archiving?
WorldMap In a Nutshell 
• Designed to lower barriers for researchers 
who wish to use spatial technology 
• Web-based, cloud hosted 
• Open source software 
• Service oriented architecture
Allows researchers to… 
• Organize: their own (large) mapping datasets and 
share them 
• Visualize: maps with data-driven symbology 
• Publish: data to the world or to just a few 
collaborators 
• Mashup / Combine: one’s own data with data 
provided by others 
• Analysis: basic but can be easily extended 
• Collaborate: by letting several people edit the 
same map
12,000 Registered Users 
14,000 Data Layers 
3,500 Map Collections 
800,000 Visits 
Statistics
Traffic By City
Built on Open Source Software
Loosely Coupled Approach 
for Adding Major Capabilities 
CSW 
MapWarper Neatline 
Dataverse 
EdX 
New 
WMS WMS 
http 
Service Registry
Contributing Organizations 
(WorldMap and GeoNode) 
• Boston Area Research Initiative - BostonMap 
• UNICEF – Education Access in Cameroon 
• Cornell University – Global Health Map 
• UN University – Wildlife Enforcement Monitoring 
• Virtue Foundation – Women in the World 
• Amazon - Hardware 
• Others… 
• World Bank – GFDRR, Dominode, Risiko 
• U.S. State Dept. – ROGUE, HIU, Syria Damage Assessment 
• NOAA - GeoCloud 
• UN World Food Program – WFP Geonode 
• Australian Govt. - AIFDR, TsuDAT 
• MapStory Foundation – MapStory App 
• Others… 
WorldMap 
GeoNode
Niche between desktop and web 
Ease of Use  
WorldMap 
web apps 
desktop apps
Openness is key 
• Open registration 
• Open access to data 
• Open service protocols (WMS, WFS, ESRI Rest) 
• Open data formats (Shape, GeoTIFF, GeoRSS, 
KML, Json, CSV) 
• Open source code (GPL on github) 
• Runs on open source operating systems 
(Linux) and could run on Windows
Data Connectors to… 
• Google maps, Open Street Maps, Bing, 
MapQuest, ESRI 
• Geo-tweets 
• Google Street View 
• Google Earth 
• Flickr, GeoRSS 
• Geonames, Google Places, Yahoo Places 
• Social Explorer, Yelp 
• WMS, WFS, ESRI REST 
• More to come…
Examples… 
• Map applications people and organizations 
have created using WorldMap…
WORLDMAP: A SPATIAL INFRASTRUCTURE TO SUPPORT TEACHING AND RESEARCH (BROWN BAG TALK)
Reischauer Institute 
Japan Earthquake Archive
WORLDMAP: A SPATIAL INFRASTRUCTURE TO SUPPORT TEACHING AND RESEARCH (BROWN BAG TALK)
Boston Area Research Initiative 
Boston Data
University of Barcelona 
Historic Planning Data
Professor Colin Gordon 
Mapping Decline, St. Louis
WORLDMAP: A SPATIAL INFRASTRUCTURE TO SUPPORT TEACHING AND RESEARCH (BROWN BAG TALK)
Virtue Foundation 
Women-run NGOs
WORLDMAP: A SPATIAL INFRASTRUCTURE TO SUPPORT TEACHING AND RESEARCH (BROWN BAG TALK)
WORLDMAP: A SPATIAL INFRASTRUCTURE TO SUPPORT TEACHING AND RESEARCH (BROWN BAG TALK)
WORLDMAP: A SPATIAL INFRASTRUCTURE TO SUPPORT TEACHING AND RESEARCH (BROWN BAG TALK)
WORLDMAP: A SPATIAL INFRASTRUCTURE TO SUPPORT TEACHING AND RESEARCH (BROWN BAG TALK)
WORLDMAP: A SPATIAL INFRASTRUCTURE TO SUPPORT TEACHING AND RESEARCH (BROWN BAG TALK)
WORLDMAP: A SPATIAL INFRASTRUCTURE TO SUPPORT TEACHING AND RESEARCH (BROWN BAG TALK)
WORLDMAP: A SPATIAL INFRASTRUCTURE TO SUPPORT TEACHING AND RESEARCH (BROWN BAG TALK)
What’s Coming in WorldMap 
• Services Registry 
• HarvardX Integration 
• Dataverse Integration 
• Neatline Integration 
• Geo-tweet Archive
WorldMap and the Services Registry 
• WorldMap is a web-based, open source, 
collaborative mapping platform developed at 
Harvard CGA since 2012 
• The National Endowment for the Humanities 
Implementation Grant. 
– Key objective: create a comprehensive and 
sustainable map service registry which 
researchers and the public can use to discover, 
create and share any work that can be 
represented spatially.
Lower the Barrier to Geodata 
Access, Across Disciplines 
Commercial 
Systems 
Internet 
local data 
Government 
Systems 
Other 
Institutions 
WorldMap Data 
and Service 
Registry
Key Goals 
• Support discovery of the millions of web maps 
that are exposed but not easy to find (the 
“dark” geoweb) 
• To allow anyone to mashup content from any 
source from within any mapping application 
• Enable non-IT professionals to create their 
own map services without IT support 
• Crowd-source map data curation in a 
metadata-weak environment
Creating a Global Service Registry 
-- A basic piece of geo-infrastructure that doesn’t exist 
• Build registry of web map services (millions of map 
layers) 
• Make API available so any system can use it 
• Provide a fast, faceted search interface 
• Allow anyone to add new services to the registry 
• Maintain uptime statistics on each service 
• Use WorldMap usage statistics to improve search 
(eventually bring in stats from systems outside 
WorldMap which use API)
How Many Services Are Out There? 
• We estimate millions, each containing many map layers 
totaling petabytes of data which is currently VERY hard 
for the average researcher to find and use. 
• Try this to estimate number of Esri REST servers 
(15million) 
– allinurl: http "arcgis rest services" mapserver -test -kml - 
kmz -sitemap -query 
• Try this to estimate number of WMS servers (47 
thousand) 
– allinurl: http "?request getcapabilities" -test
Service Registry Challenges 
• Metadata - tagging and usage statistics 
• Projections - cascading 
• Persistence - caching 
• Discovery - central index, usage statistics 
• Performance - caching
Open API to Registry 
• Public, RESTful API 
• Access all (public) map layers within 
WorldMap 
• Access all service layers outside WorldMap 
• Access all Maps (collections of layers) within 
WorldMap 
• Search on information: 
– Metadata 
– Usage statistics 
– Attribute info (for local layers)
Distributed 
Map 
Services 
S 
e 
r 
v 
i 
c 
e 
s 
A 
P 
I 
Service 
Registry 
Distributed 
Users 
OpenLayers, 
Leaflet 
Esri 
clients 
Any 
map 
client 
WorldMap 
Find and bind 
to layers 
Crowd curation, user 
submitted services 
WorldMap 
Service 
Crawler 
Uptime 
Checker 
Service 
caching, 
reprojection 
WorldMap 
Local 
Services 
WorldMap 
Core 
*Common Crawl 
*Start with hadoop search of 
Common Crawl dataset 
http://commoncrawl.org/
Faceted Services Search UI Mockup (draft)
More information… 
WorldMap 
http://worldmap.harvard.edu 
Center for Geographic Analysis 
http://gis.harvard.edu 
Ben Lewis 
blewis@cga.harvard.edu
WORLDMAP: A SPATIAL INFRASTRUCTURE TO SUPPORT TEACHING AND RESEARCH (BROWN BAG TALK)
HarvardX Integration
Dataverse Integration 
(Social Science Archive)
Neatline Integration 
map-based story telling platform
TweetMap Archive
Demo of Service Registry Test Server 
• http://107.22.231.227/ 
– Show services 
– Add service 
– Find service layers as layers 
– Metadata for services 
– Layer page for services 
– Statistics for services 
– Saving map with service layers
Service Registry Architecture
End
Information Exists Within Spectrum of 
Publicness and Curatedness 
• Publicness 
Private - Group – University – Consortium - Public 
• Curatedness 
Data created – Reviewed by expert – Reviewed by 
crowd
Our job 
• Create platforms (of people and software and 
objects) to lower barriers to data access: 
– Data hosting 
– Data formatting 
– Metadata creation 
– Data and view sharing 
– Search tools – local and remote resources 
– Connecting systems where relevant 
– Analytical services 
– Publishing

More Related Content

PDF
New Discovery Tools for Digital Humanities and Spatial Data (Summary of the J...
PDF
New tasks, new roles: Libraries in the tension between Digital Humanities, Re...
PDF
2014_WWW_BTOR
PDF
20170501 Distributed Network of Digital Heritage Information
PDF
Csdh sbg clariah_intr01
PPTX
Making social science more reproducible by encapsulating access to linked data
PDF
Wehc - Linked Data for Economic-Social historians
PPTX
Chaos&Order: Using visualization as a means to
 explore large heritage collec...
New Discovery Tools for Digital Humanities and Spatial Data (Summary of the J...
New tasks, new roles: Libraries in the tension between Digital Humanities, Re...
2014_WWW_BTOR
20170501 Distributed Network of Digital Heritage Information
Csdh sbg clariah_intr01
Making social science more reproducible by encapsulating access to linked data
Wehc - Linked Data for Economic-Social historians
Chaos&Order: Using visualization as a means to
 explore large heritage collec...

What's hot (20)

PPTX
QB'er demonstration
PDF
2016 05-20-clariah-wp4
PDF
Linked Data
PPTX
DSpace for Cultural Heritage: adding support for images visualization,audio/v...
PPT
Rdf and open linked data a first approach
PPTX
Wednesday 6 May: Hand me the data! What you should know as a humanities resea...
PDF
Nanopublications and Decentralized Publishing
PPTX
A Comparative Kalendar - DH2013 Presentation
PPTX
Esshc presentation ashkan
PPTX
Elephant in the Room: Scaling Storage for the HathiTrust Research Center
PPTX
Humanities Users in the Digital Age: Library Needs Assessment
PPTX
Connecting Heterogeneous Collections using Linked Data
PPTX
Linked Geo-Data and Early Geospatial Documents
PDF
co:op-READ-Convention Marburg - Sebastian Colutto
PDF
co:op-READ-Convention Marburg - Günter Mühlberger
PPTX
COAR Venice 2017 Next Generation Repository Session: What can be done, right ...
PPTX
Repository technologies
PDF
DBpedia Mappings Wiki, SMWCon Fall 2013, Berlin
PPTX
Sharing data
PDF
SSHA 2019: Reconstructring a country
QB'er demonstration
2016 05-20-clariah-wp4
Linked Data
DSpace for Cultural Heritage: adding support for images visualization,audio/v...
Rdf and open linked data a first approach
Wednesday 6 May: Hand me the data! What you should know as a humanities resea...
Nanopublications and Decentralized Publishing
A Comparative Kalendar - DH2013 Presentation
Esshc presentation ashkan
Elephant in the Room: Scaling Storage for the HathiTrust Research Center
Humanities Users in the Digital Age: Library Needs Assessment
Connecting Heterogeneous Collections using Linked Data
Linked Geo-Data and Early Geospatial Documents
co:op-READ-Convention Marburg - Sebastian Colutto
co:op-READ-Convention Marburg - Günter Mühlberger
COAR Venice 2017 Next Generation Repository Session: What can be done, right ...
Repository technologies
DBpedia Mappings Wiki, SMWCon Fall 2013, Berlin
Sharing data
SSHA 2019: Reconstructring a country
Ad

Viewers also liked (12)

PPTX
10 SIMPLE STEPS TO BUILDING A REPUTATION AS A RESEARCHER, IN YOUR EARLY CAREER
PPTX
Complicating the Question of Access (and Value) with University Press Publica...
PPTX
Redistricting and Voting Technology
PPTX
Brown Bag: New Models of Scholarly Communication for Digital Scholarship, by ...
PPTX
June2014 brownbag privacy
PPTX
Brown Bag: DMCA §1201 and Video Game Preservation Institutions: A Case Study ...
PDF
BROWN BAG: THE VISUAL COMPONENT: MORE THAN PRETTY PICTURES - WITH FELICE FRANKEL
PDF
Can computers be feminist? Program on Information Science Talk by Gillian Smith
PPTX
Gary Price, MIT Program on Information Science
PPTX
Data Citation Rewards and Incentives
PDF
The Open Access Network: Rebecca Kennison’s Talk for the MIT Prorgam on Infor...
PDF
Making Decisions in a World Awash in Data: We’re going to need a different bo...
10 SIMPLE STEPS TO BUILDING A REPUTATION AS A RESEARCHER, IN YOUR EARLY CAREER
Complicating the Question of Access (and Value) with University Press Publica...
Redistricting and Voting Technology
Brown Bag: New Models of Scholarly Communication for Digital Scholarship, by ...
June2014 brownbag privacy
Brown Bag: DMCA §1201 and Video Game Preservation Institutions: A Case Study ...
BROWN BAG: THE VISUAL COMPONENT: MORE THAN PRETTY PICTURES - WITH FELICE FRANKEL
Can computers be feminist? Program on Information Science Talk by Gillian Smith
Gary Price, MIT Program on Information Science
Data Citation Rewards and Incentives
The Open Access Network: Rebecca Kennison’s Talk for the MIT Prorgam on Infor...
Making Decisions in a World Awash in Data: We’re going to need a different bo...
Ad

Similar to WORLDMAP: A SPATIAL INFRASTRUCTURE TO SUPPORT TEACHING AND RESEARCH (BROWN BAG TALK) (20)

PPTX
I ♥ Maps: Quantum GIS + Python
PDF
Cartography and Web GIS - Jack Dangermond
PPTX
Going Mobile with HTML5
PDF
LoCloud Geolocation enrichment tools, Siri Slettvag, Asplan Viak Internet (Av...
PDF
neurisa_11_09_rosenthal
PDF
neurisa_11_09_rosenthal
PPTX
Introduction to GIS
PDF
Geocart workshop
PPTX
ArcGIS and Multi-D: Tools & Roadmap
PDF
Free remote sensing and GIS data
PPTX
Dublinked tech workshop_15_dec2011
PPTX
Linked services: Connecting services to the Web of Data
PPTX
Crowd-Sourcing Approach of Building Ground Truth Database for Global Urban Ar...
PDF
Building Maps with Leaflet
PDF
What is GIS (Course Presentation).pdf
PDF
CartoHeritage 2011: Georeferencer & MapRank Search
PDF
Building an Open Source, Real-Time, Billion Object Spatio-Temporal Search Pla...
PDF
From Big Data to Fast Data
PPTX
Lecture1
PPT
NSDI_ Concepts and Components
I ♥ Maps: Quantum GIS + Python
Cartography and Web GIS - Jack Dangermond
Going Mobile with HTML5
LoCloud Geolocation enrichment tools, Siri Slettvag, Asplan Viak Internet (Av...
neurisa_11_09_rosenthal
neurisa_11_09_rosenthal
Introduction to GIS
Geocart workshop
ArcGIS and Multi-D: Tools & Roadmap
Free remote sensing and GIS data
Dublinked tech workshop_15_dec2011
Linked services: Connecting services to the Web of Data
Crowd-Sourcing Approach of Building Ground Truth Database for Global Urban Ar...
Building Maps with Leaflet
What is GIS (Course Presentation).pdf
CartoHeritage 2011: Georeferencer & MapRank Search
Building an Open Source, Real-Time, Billion Object Spatio-Temporal Search Pla...
From Big Data to Fast Data
Lecture1
NSDI_ Concepts and Components

More from Micah Altman (20)

PPTX
Selecting efficient and reliable preservation strategies
PPTX
Well-Being - A Sunset Conversation
PPTX
Matching Uses and Protections for Government Data Releases: Presentation at t...
PPTX
Privacy Gaps in Mediated Library Services: Presentation at NERCOMP2019
PPTX
Well-being A Sunset Conversation
PPTX
Can We Fix Peer Review
PDF
Academy Owned Peer Review
PPTX
Redistricting in the US -- An Overview
PPTX
A Future for Electoral Districting
PPTX
A History of the Internet :Scott Bradner’s Program on Information Science Talk
PPTX
SAFETY NETS: RESCUE AND REVIVAL FOR ENDANGERED BORN-DIGITAL RECORDS- Program ...
PPTX
Labor And Reward In Science: Commentary on Cassidy Sugimoto’s Program on Info...
PPTX
Utilizing VR and AR in the Library Space:
PPTX
Creative Data Literacy: Bridging the Gap Between Data-Haves and Have-Nots
PPTX
SOLARSPELL: THE SOLAR POWERED EDUCATIONAL LEARNING LIBRARY - EXPERIENTIAL LEA...
PDF
Ndsa 2016 opening plenary
PPTX
Software Repositories for Research-- An Environmental Scan
PDF
Attribution from a Research Library Perspective, on NISO Webinar: How Librari...
PPTX
Agenda's for Preservation Research
PPTX
Software Repositories for Research -- An Environmental Scan
Selecting efficient and reliable preservation strategies
Well-Being - A Sunset Conversation
Matching Uses and Protections for Government Data Releases: Presentation at t...
Privacy Gaps in Mediated Library Services: Presentation at NERCOMP2019
Well-being A Sunset Conversation
Can We Fix Peer Review
Academy Owned Peer Review
Redistricting in the US -- An Overview
A Future for Electoral Districting
A History of the Internet :Scott Bradner’s Program on Information Science Talk
SAFETY NETS: RESCUE AND REVIVAL FOR ENDANGERED BORN-DIGITAL RECORDS- Program ...
Labor And Reward In Science: Commentary on Cassidy Sugimoto’s Program on Info...
Utilizing VR and AR in the Library Space:
Creative Data Literacy: Bridging the Gap Between Data-Haves and Have-Nots
SOLARSPELL: THE SOLAR POWERED EDUCATIONAL LEARNING LIBRARY - EXPERIENTIAL LEA...
Ndsa 2016 opening plenary
Software Repositories for Research-- An Environmental Scan
Attribution from a Research Library Perspective, on NISO Webinar: How Librari...
Agenda's for Preservation Research
Software Repositories for Research -- An Environmental Scan

Recently uploaded (20)

PPTX
PA Analog/Digital System: The Backbone of Modern Surveillance and Communication
PDF
How UI/UX Design Impacts User Retention in Mobile Apps.pdf
PPTX
Digital-Transformation-Roadmap-for-Companies.pptx
PDF
Approach and Philosophy of On baking technology
PDF
Review of recent advances in non-invasive hemoglobin estimation
PDF
Machine learning based COVID-19 study performance prediction
PDF
Bridging biosciences and deep learning for revolutionary discoveries: a compr...
PDF
KodekX | Application Modernization Development
PDF
Agricultural_Statistics_at_a_Glance_2022_0.pdf
PPT
Teaching material agriculture food technology
PDF
7 ChatGPT Prompts to Help You Define Your Ideal Customer Profile.pdf
PDF
Blue Purple Modern Animated Computer Science Presentation.pdf.pdf
PPTX
Detection-First SIEM: Rule Types, Dashboards, and Threat-Informed Strategy
PPTX
Cloud computing and distributed systems.
PDF
Advanced methodologies resolving dimensionality complications for autism neur...
PDF
Building Integrated photovoltaic BIPV_UPV.pdf
PPTX
Understanding_Digital_Forensics_Presentation.pptx
PDF
Encapsulation theory and applications.pdf
PDF
CIFDAQ's Market Insight: SEC Turns Pro Crypto
PDF
Encapsulation_ Review paper, used for researhc scholars
PA Analog/Digital System: The Backbone of Modern Surveillance and Communication
How UI/UX Design Impacts User Retention in Mobile Apps.pdf
Digital-Transformation-Roadmap-for-Companies.pptx
Approach and Philosophy of On baking technology
Review of recent advances in non-invasive hemoglobin estimation
Machine learning based COVID-19 study performance prediction
Bridging biosciences and deep learning for revolutionary discoveries: a compr...
KodekX | Application Modernization Development
Agricultural_Statistics_at_a_Glance_2022_0.pdf
Teaching material agriculture food technology
7 ChatGPT Prompts to Help You Define Your Ideal Customer Profile.pdf
Blue Purple Modern Animated Computer Science Presentation.pdf.pdf
Detection-First SIEM: Rule Types, Dashboards, and Threat-Informed Strategy
Cloud computing and distributed systems.
Advanced methodologies resolving dimensionality complications for autism neur...
Building Integrated photovoltaic BIPV_UPV.pdf
Understanding_Digital_Forensics_Presentation.pptx
Encapsulation theory and applications.pdf
CIFDAQ's Market Insight: SEC Turns Pro Crypto
Encapsulation_ Review paper, used for researhc scholars

WORLDMAP: A SPATIAL INFRASTRUCTURE TO SUPPORT TEACHING AND RESEARCH (BROWN BAG TALK)

  • 1. MIT Cambridge, September 2014 A Spatial Platform to Support Research and Collaboration Ben Lewis blewis@cga.harvard.edu http://worldmap.harvard.edu
  • 2. Outline • Why lat/long is valuable and what research support operations should do about it • WorldMap as straw man – Overview of current system – New projects • Global services registry (OGP integration?) • Dataverse integration • HarvardX integration • Neatline integration • Discussion
  • 3. Lat/long (location) as organizing facet (http://www.architectmagazine.com/technology/meet-the-geodesigner.aspx
  • 4. Spatial search • Improve access to both spatial and non spatial materials with a “spatial visualization facet” – Books, documents – Paper maps – Local spatial data – Remote spatial data – Local map display and processing services – Remote map display and processing services – Wikipedia – Social media – tweets, etc – The web
  • 5. Build platforms that lower barriers to: • Finding spatial materials which reside inside or outside the academy • Visualizing this data • Using this data • Sharing this data and views of it • Mashing up private and public data • Spatializing materials that are not yet spatial
  • 6. Support spatial and spatio-temporal visualization • Make it easy to visualize datasets on a map (A great way to see temporal patterns is on a map http://worldmap.harvard.edu/tweetmap ) • Make it easy to mashup private and public data • Make it easy to share spatial data and views of data • Make it easy to crowd curate data
  • 7. Support the research life cycle * Research Lifecycle: • Scoping • Data gathering / exploration • Synthesis / analysis • Writing / communication • Publishing / delivery Archiving?
  • 8. WorldMap In a Nutshell • Designed to lower barriers for researchers who wish to use spatial technology • Web-based, cloud hosted • Open source software • Service oriented architecture
  • 9. Allows researchers to… • Organize: their own (large) mapping datasets and share them • Visualize: maps with data-driven symbology • Publish: data to the world or to just a few collaborators • Mashup / Combine: one’s own data with data provided by others • Analysis: basic but can be easily extended • Collaborate: by letting several people edit the same map
  • 10. 12,000 Registered Users 14,000 Data Layers 3,500 Map Collections 800,000 Visits Statistics
  • 12. Built on Open Source Software
  • 13. Loosely Coupled Approach for Adding Major Capabilities CSW MapWarper Neatline Dataverse EdX New WMS WMS http Service Registry
  • 14. Contributing Organizations (WorldMap and GeoNode) • Boston Area Research Initiative - BostonMap • UNICEF – Education Access in Cameroon • Cornell University – Global Health Map • UN University – Wildlife Enforcement Monitoring • Virtue Foundation – Women in the World • Amazon - Hardware • Others… • World Bank – GFDRR, Dominode, Risiko • U.S. State Dept. – ROGUE, HIU, Syria Damage Assessment • NOAA - GeoCloud • UN World Food Program – WFP Geonode • Australian Govt. - AIFDR, TsuDAT • MapStory Foundation – MapStory App • Others… WorldMap GeoNode
  • 15. Niche between desktop and web Ease of Use  WorldMap web apps desktop apps
  • 16. Openness is key • Open registration • Open access to data • Open service protocols (WMS, WFS, ESRI Rest) • Open data formats (Shape, GeoTIFF, GeoRSS, KML, Json, CSV) • Open source code (GPL on github) • Runs on open source operating systems (Linux) and could run on Windows
  • 17. Data Connectors to… • Google maps, Open Street Maps, Bing, MapQuest, ESRI • Geo-tweets • Google Street View • Google Earth • Flickr, GeoRSS • Geonames, Google Places, Yahoo Places • Social Explorer, Yelp • WMS, WFS, ESRI REST • More to come…
  • 18. Examples… • Map applications people and organizations have created using WorldMap…
  • 20. Reischauer Institute Japan Earthquake Archive
  • 22. Boston Area Research Initiative Boston Data
  • 23. University of Barcelona Historic Planning Data
  • 24. Professor Colin Gordon Mapping Decline, St. Louis
  • 34. What’s Coming in WorldMap • Services Registry • HarvardX Integration • Dataverse Integration • Neatline Integration • Geo-tweet Archive
  • 35. WorldMap and the Services Registry • WorldMap is a web-based, open source, collaborative mapping platform developed at Harvard CGA since 2012 • The National Endowment for the Humanities Implementation Grant. – Key objective: create a comprehensive and sustainable map service registry which researchers and the public can use to discover, create and share any work that can be represented spatially.
  • 36. Lower the Barrier to Geodata Access, Across Disciplines Commercial Systems Internet local data Government Systems Other Institutions WorldMap Data and Service Registry
  • 37. Key Goals • Support discovery of the millions of web maps that are exposed but not easy to find (the “dark” geoweb) • To allow anyone to mashup content from any source from within any mapping application • Enable non-IT professionals to create their own map services without IT support • Crowd-source map data curation in a metadata-weak environment
  • 38. Creating a Global Service Registry -- A basic piece of geo-infrastructure that doesn’t exist • Build registry of web map services (millions of map layers) • Make API available so any system can use it • Provide a fast, faceted search interface • Allow anyone to add new services to the registry • Maintain uptime statistics on each service • Use WorldMap usage statistics to improve search (eventually bring in stats from systems outside WorldMap which use API)
  • 39. How Many Services Are Out There? • We estimate millions, each containing many map layers totaling petabytes of data which is currently VERY hard for the average researcher to find and use. • Try this to estimate number of Esri REST servers (15million) – allinurl: http "arcgis rest services" mapserver -test -kml - kmz -sitemap -query • Try this to estimate number of WMS servers (47 thousand) – allinurl: http "?request getcapabilities" -test
  • 40. Service Registry Challenges • Metadata - tagging and usage statistics • Projections - cascading • Persistence - caching • Discovery - central index, usage statistics • Performance - caching
  • 41. Open API to Registry • Public, RESTful API • Access all (public) map layers within WorldMap • Access all service layers outside WorldMap • Access all Maps (collections of layers) within WorldMap • Search on information: – Metadata – Usage statistics – Attribute info (for local layers)
  • 42. Distributed Map Services S e r v i c e s A P I Service Registry Distributed Users OpenLayers, Leaflet Esri clients Any map client WorldMap Find and bind to layers Crowd curation, user submitted services WorldMap Service Crawler Uptime Checker Service caching, reprojection WorldMap Local Services WorldMap Core *Common Crawl *Start with hadoop search of Common Crawl dataset http://commoncrawl.org/
  • 43. Faceted Services Search UI Mockup (draft)
  • 44. More information… WorldMap http://worldmap.harvard.edu Center for Geographic Analysis http://gis.harvard.edu Ben Lewis blewis@cga.harvard.edu
  • 47. Dataverse Integration (Social Science Archive)
  • 48. Neatline Integration map-based story telling platform
  • 50. Demo of Service Registry Test Server • http://107.22.231.227/ – Show services – Add service – Find service layers as layers – Metadata for services – Layer page for services – Statistics for services – Saving map with service layers
  • 52. End
  • 53. Information Exists Within Spectrum of Publicness and Curatedness • Publicness Private - Group – University – Consortium - Public • Curatedness Data created – Reviewed by expert – Reviewed by crowd
  • 54. Our job • Create platforms (of people and software and objects) to lower barriers to data access: – Data hosting – Data formatting – Metadata creation – Data and view sharing – Search tools – local and remote resources – Connecting systems where relevant – Analytical services – Publishing

Editor's Notes

  • #10: So clearly people can load and organize ones one mapping datasets Go over slide Then: In other words….
  • #15: Organizations making major investments in GeoNode, the underlying platform: Harvard WorldBank GFDRR Dominode, Project NOAH Red Cross – Yolanda http://reliefweb.int/report/world/gfdrr-and-american-red-cross-launch-online-data-sharing-platform-aftermath-typhoon U.S. State Department ROUGE Project Malawi National Spatial Data Center University of West Indies GEM MapStory WFP JRC HIU NOAA
  • #16: WorldMap aims to fill a gap between desktop-bound mapping application and lightweight web apps
  • #20: Here is data brought together from various sources around the topic of fracking. This is a map of the bakken shale, one of the best places in the U.S. to Frack. Developed in a collaboration with Harvard School of Public Health
  • #21: Here is a collection of data developed after the Japan earthquake as part of building the Japan Data archive by the Reischauer institute. Developed for the Japan Daishinsai Archive, with support from the Reischauer Institute for Japanese Studies Edited by Ray Kameda and Merrick Lex Berman
  • #22: New Jersery flood hazard visualization All elevations are in feet and tenths of feet and refer to the North American Vertical Datum of 1988 (NAVD88).  To convert from NAVD88 to mean lower low water (m.l.l.w) add 2.43’.  To convert from NAVD88 to mean sea level (MSL) add 0.21’.   Please be advised that the hurricane storm surge inundation areas are based on a worst case scenario for each hurricane category, assuming a direct hit, maximum forward speed and striking at precisely high tide. Actual storm surge inundation areas will likely vary, so please utilize the latest National Weather Service forecasts and guidance from local emergency management officials.
  • #23: BostonMap developed by Chris Winship and others
  • #24: Historic planning data for barcelona (Universitat Autonoma de Barcelona) Another - This map visualizes the historic spatial footprints of hospital demand in barcelona And look at the shape of demand for Hospitals in Barcelona… Barcelona Historic Plan
  • #26: Philadelphia Neighborhoods color coded by parcel type
  • #27: A map created by the Virtue foundation to map women run NGOs around the world
  • #28: This map shows nearby wellness programs
  • #29: Developed by Dr. Mirza of the University of Umm Al-Qura in Makkah Current road data for Makkah combined with city boundaries of 1880 and 1945 To see how morphology of Mekkah changed
  • #30: Macrozoneamento Ecológico e Econômico de Goiás O Estado de Goiás possui como características marcantes o seu expressivo capital natural e riqueza cultural, por isso tem se tornado centro de interesses estratégicos que movimentam a política e a economia regional, nacional e mundial, devido ao potencial hídrico, mineral, genético, energético, fundiário, de produção de alimentos, e mais recentemente a produção de biocombustíveis. Os notáveis conflitos entre o desenvolvimento socioeconômico, e a necessidade de preservação da biodiversidade, além do respeito com as áreas naturalmente vulneráveis, coloca em evidência a necessidade do estabelecimento de um novo paradigma para o desenvolvimento do estado de Goiás. Esse novo modelo de desenvolvimento sustentável deve ser capaz de implementar a utilização do inestimável capital natural e a riqueza cultural do estado de Goiás, visto que a intensa ocupação de seu território, sem o devido planejamento que considerasse os aspectos ambientais, culminou em uma grande perda da biodiversidade, e de recursos naturais na região. Neste contexto, registram-se importantes conflitos sociais e ambientais na disputa pela destinação e uso da terra e dos recursos naturais, ao mesmo tempo em que novas formas e relações de produção são introduzidas no estado, com registro de parcerias internacionais acopladas a projetos domésticos de diversas ordens. Na ausência de um padrão de desenvolvimento adequado à especificidade do estado, por ele avançaram e ainda avançam rapidamente atividades predatórias, apesar de se ter em plena vigência, no âmbito global, novos modos de produzir, baseados na ciência e na tecnologia e que buscam otimizar o uso dos recursos naturais, e a ocupação do solo respeitando as vulnerabilidades naturais do mesmo. O uso sustentável do território do estado de Goiás, a partir do planejamento de seus usos múltiplos e integrados, poderá gerar mais trabalho e riqueza, considerando suas potencialidades, tais como recursos hídricos, solos, topografia, infraestrutura instalada, diversidade biológica, entre outros. As mudanças sociais dos últimos anos, indicam que a competitividade ocorrerá principalmente por meio de soluções inovadoras e sustentáveis no uso de recursos naturais, e do território. Salienta-se que o planejamento na implantação de novas infraestruturas, e no uso sustentável do capital natural, são indispensáveis para um projeto de desenvolvimento que concilie as funções estratégicas das forças produtivas do estado de Goiás, com os parâmetros da sustentabilidade ambiental que ocorre no estado. No entanto, para que seja possível o estabelecimento de políticas públicas que direcionem o desenvolvimento sustentável do estado de Goiás, faz se necessário, antes de tudo, as macrozonas ecológicas e econômicas do estado de Goiás, que são obtidas pela integração espacial entre o desempenho social e econômico dos municípios goianos, com suas vulnerabilidades naturais. Sendo assim, por meio do Decreto no 6.707/2007, o governo de Goiás instituiu a Comissão Coordenadora do ZAEE-GO, e deu outras providências. Sendo também, foi firmado o Convênio 44045/2009, entre MMA/SECIDADES/SEMARH/SEAGRO, com o intuito de se elaborar o Macrozoneamento Ecológico e Econômico (MacroZEE) do estado de Goiás. Mais que um instrumento para a gestão, o MacroZEE torna viável a implementação de regras que conduzam à organização eficaz da sociedade e de sua base econômica, em conformidade com os princípios e práticas da sustentabilidade. Além do seu caráter técnico, o MacroZEE é o ensaio para a implantação do Zoneamento Ecológico e Econômico, que é um instrumento político, de negociação entre os diversos setores da sociedade. Um instrumento não de exclusão de qualquer ator, mas, sim, de compatibilização entre eles, respeitando as vulnerabilidades naturais do território goiano. O ZEE se apresenta como uma ferramenta clara, para que a sociedade civil e os empreendedores conheçam as vulnerabilidades e potencialidades de cada local ou região, pois promove a integração espacial entre as vulnerabilidades ambientais e os desempenhos sociais e econômicas do território. Nesse contexto ele torna possível que os empreendedores tenham o prévio conhecimento das peculiaridades e as exigências ambientais para se instalarem num dado local. Por outro lado, o ZEE possibilita que o estado se aparelhe, utilizando critérios essencialmente técnicos, ao estabelecer novos e impessoais procedimentos para análise de projetos. O estabelecimento de novos procedimentos sobre como os processos serão avaliados, permitirá aos empreendedores, pequenos ou grandes, não serem surpreendidos. Portanto, além de procurar fornecer bases para o desenvolvimento sustentável de cada local, município ou região, o ZEE também é um estímulo para que os empreendedores busquem o estado de Goiás para ampliar seus negócios com a segurança ambiental necessária, gerando assim renda, emprego e bem-estar social. Os adventos científicos e tecnológicos atualmente disponíveis, fornecem meios para a produção de informações geograficamente referenciadas a respeito das características sociais, econômicas, institucionais e ambientais do estado de Goiás. Essas informações são subsídios imprescindíveis para o desenvolvimento sustentável do estado de Goiás no século XXI, minimizando as diferenças sociais e econômicas, estabelecendo o fortalecimento e segurança institucional, e finalmente promovendo a eficiência na gestão dos recursos ambientais, explicitando os locais de alta vulnerabilidade natural, que necessitam ser protegidas, além de promover o uso racional de serviços ambientais, dos recursos hídricos, do solo, do relevo e da biodiversidade.
  • #31: Air pollution in Eastern Massachusetts PM 2.5 Measurements in Eastern Massachusetts micrograms per cubic meter
  • #32: Giza Map (Anthropology)
  • #33: Introduction In 2009, 5.7 million people were living with HIV and AIDS in South Africa, a number higher than any other country. In order to improve the national collective response to HIV, the Virtue Foundation has partnered with Charlize Theron’s Africa Outreach Program and the Women in the World Foundation to launch Get on the Map! for HIV Prevention in South Africa.   This new, interactive online tool maps national, regional, local and institutional data on HIV/AIDS and showcases South African HIV/AIDS-related organizations and projects. Launched on April 5, 2013 at the Women in the World Summit, the Get on the Map! for HIV Prevention in South Africa advances evidence-based, high-impact HIV/AIDS policy and programming across the country.   Once fully populated, this tool will serve as a transformative development resource. Whether you are an individual interested in learning how you can contribute or an organization planning a project in a new area and want to know what is already going on and how to match resource to need, Get On The Map! for HIV Prevention in South Africa is a resource you can use.  By widening access to data on HIV/AIDS and to information about the HIV/AIDS-related work of organizations, the map aims to drive collaboration between groups with similar goals in South Africa. We invite you to submit your projects, use the interactive mapping tools, and Get on the Map! for HIV Prevention in South Africa.
  • #40: A proverbial dark web of data which is not easy to find. We used this Google query to estimate the number of WMS services existing on the web. Here is the URL we used in our search.   allinurl: http "?request getcapabilities" -test   The term “getcapabilities” is the initial request a client makes in attempting to connect to map services from a WMS server. This is preceded by a “?” in actual URL constructions which allows us to weed out most discussion posts talking about WMS services. This search returns 192,000 pages. Because WMS works somewhat differently than ESRI REST, these pages point to web servers; the count does not include the layers within the services. Thus the total number of web map services is likely to be much higher than the number of search results. In response to the getcapabilities request, a (sometimes long) list of mapping services is returned along with the information the client will need to connect to them as live maps and load them to its viewer. Again we looked at 25 pages of results and tested 2 results on each page for a total of 50 tests. Of the results returned, 75% were valid WMS URLs which pointed to WMS servers which would translate to a total of 144 thousand servers. The number of layers per server ranged from 2 to 285 with an average of 22. Multiplying 144 thousand servers by an average of 22 layers per server gives an estimated potential total of 3,168,000 WMS layers. When tested, 35% of these servers returned a live map. Our estimate of the number of live WMS layers is therefore about a million. We looked for these live WMS services in ArcGIS Online and did not find any of them which would suggest that at least a million WMS layers are missing from ArcGIS Online’s catalog. We also looked for these live WMS layers in MapMatters and found just 10 percent of them. On the MapMatters site they list the number of services in their index at 2146 and the number of layers at 89,523, an average of 41 layers per service. The 10% match with layers in MapMatters is about what one would expect if one had a list of a million (from the Google search) and randomly selected from it to search a subset of that list (in MapMatters) which contained about one tenth as many records (89,523). A map service index that was last updated in 2009, by Skylab Mobilesystems (http://www.skylab-mobilesystems.com/en/wms_serverlist.html) says it contained 339,254 WMS layers from 994 servers. This would be an average of 341 layers per server. In 2006 MapDex (http://www.mapdex.org) had an index of 1,781 servers serving 32,433 map services, both ESRI and WMS. That would be 18 layers per service on average. Clearly there is uncertainly in how many WMS layers there are per service on average. To summarize, between the most common types of map services, WMS and ESRI REST, we estimate that there are several million web map services not listed in a major index and we know that no index covers both ESRI REST and WMS well.
  • #47: This work is in its early stages. Bob Light and Samantha Earp
  • #50: About 4 billion tweets harvested to date Developing fast query platform
  • #51: I would like give you a chance to shape the system