SlideShare a Scribd company logo
Shipping and weather data 
CLIWOC ships captains logs 1662-1855 
A Question on Joining Look-up Tables 
Andrew Zolnai 
Cambridge UK 
aiz@zolnai.ca 
Image: Royal Museum Greenwich
Historic Climatology 
• Ships captains logs data from 1662 to 1855 
• Ships and routing details and location 
• Weather recording of almost 120 parameters 
• Release 1.x (2004) and Release 2.x (2007)
Location Metadata 
• First added to Esri file geodatabase 
– The complete (weather and location) dataset has 
almost 290,000 points, and is almost 250Mb database 
– The locations were extracted into a geodatabase (70 
Mb), then into a compressed file geodatabase (10 Mb) 
• Post map package for Release 2.0 
– layer files showing them in 25-year time-slices 
– and for Captains Cook and de la Perouse 
• Posted as ArcGIS Online map service 
– Use time stamps to aggregate layers 
– But stock server has performance issues 
– So Filter data by nationality and decade 
– Limits web fetches to 1000s not 100,000s
(Visio Pro data model reverse-engineered 
from Dbase extracts of File Geodatabase)
Climate Metadata 
• CLIWOC posted all shipboard data it had 
– from British, Dutch, French and Spanish sources 
– with look-up tables for each different source 
• Merged wind force and direction data almost 
doubles the number of feature classes as observed 
– Started with Wind Force 
– Then joined Wind Direction 
– Finally merged Force & Direction 
• Map package and layer template
(Dbase extract 8 char limit: 
add prefix of 
CLIWOC21_Features_ 
to all except look-ups 
Lookup_{ }_WindDirection_)
(Dbase extract 8 char limit: 
add prefix of 
CLIWOC21_Features_ 
to all except look-ups 
Lookup_{ }_WindForce_)
Wind Direction from each Maritime Agency Look-up to Merged 
Wind Force from each Maritime Agency Look-up to Merged
Question 
• Joining four lookup tables for each Nationality / 
Maritime Agency created redundant attributes 
• Performing a merge in Geoprocessing collapsed the 
redundancies into single computable columns 
• Why does it almost double the feature count? 
• GIS data Visio diagrams
CLIWOC Attributes

More Related Content

PPT
Reading HDF family of formats via NetCDF-Java / CDM
PPTX
Use FME To Efficiently Create National-Scale Vector Contours From High-Resolu...
PPTX
WMTS Performance Tests
PPTX
SmartMet Server OSGeo
PPTX
Advancing Scientific Data Support in ArcGIS
DOC
CORS96 ADJUSTMENT REPORT
PPT
The Homogenization and Reporting of Groundbased Atmospheric Datasets for the ...
PDF
1Spatial: FME World Tour London: Digital surveying with FME
Reading HDF family of formats via NetCDF-Java / CDM
Use FME To Efficiently Create National-Scale Vector Contours From High-Resolu...
WMTS Performance Tests
SmartMet Server OSGeo
Advancing Scientific Data Support in ArcGIS
CORS96 ADJUSTMENT REPORT
The Homogenization and Reporting of Groundbased Atmospheric Datasets for the ...
1Spatial: FME World Tour London: Digital surveying with FME

What's hot (20)

PPTX
Making data storage more efficient
PDF
Open Source Routing Machine - FOSS4G 2016 Bonn
PDF
Ronan Kerr: Exploring the Debris Disk Around Beta Pictoris
PDF
GIS on Rails by Oleksandr Kychun
PDF
Cstp project
PPT
Rcm tracking for teams
PDF
State of OSRM - SOTM 2016
PPTX
Localisation network
PPTX
FMI Open Data Interface and Usage
PPT
Using HDF5 Archive Information Package to preserve HDF-EOS2 data
PPTX
Hexagon binning for petroleum data
PDF
Ccsds based file delivery protocol (cfdp) v1p3
PPTX
FME World Tour 2016: Your Data in Motion (Safe Software)
PDF
Btp presentation
PDF
Meteo I/O Introduction
PDF
In-Car Navigation with OSRM - Wherecamp Berlin 2016
PPTX
Llnl talk
PDF
Converting between HDF4 and HDF5
PPT
Rural Payments Agency usage of Ordnance Survey data
PDF
Shi_2015_present
Making data storage more efficient
Open Source Routing Machine - FOSS4G 2016 Bonn
Ronan Kerr: Exploring the Debris Disk Around Beta Pictoris
GIS on Rails by Oleksandr Kychun
Cstp project
Rcm tracking for teams
State of OSRM - SOTM 2016
Localisation network
FMI Open Data Interface and Usage
Using HDF5 Archive Information Package to preserve HDF-EOS2 data
Hexagon binning for petroleum data
Ccsds based file delivery protocol (cfdp) v1p3
FME World Tour 2016: Your Data in Motion (Safe Software)
Btp presentation
Meteo I/O Introduction
In-Car Navigation with OSRM - Wherecamp Berlin 2016
Llnl talk
Converting between HDF4 and HDF5
Rural Payments Agency usage of Ordnance Survey data
Shi_2015_present
Ad

More from Andrew Zolnai (20)

PDF
LINQ 101
PDF
Group list be age via pivot table
PPTX
Reverse engineer data to match cums
PDF
nCOVID-19 pivot-and-fan map
PDF
Fire archive modis6_n50
PPTX
New Millennium London Fireworks
PDF
Mind maps for Cottenham Open
PDF
"Welcome back to the new Middle Ages"
PPTX
3D GIS time travel
PDF
Zolnai geobyte manuscript
PDF
Azolnai geocom2017
PPTX
Hurricane harvey copy
PDF
Mind Maps 4 GDPR
PDF
Geohipster Calendar, May 2018
PDF
Agi conf thanks_andrew_zolnai
PDF
To grid or not to grid
DOCX
Geobase
PDF
Info Supply Chain for Decommissioning
PPTX
Petex 2016 Future Working Zone
PPTX
Unified online dashboards to preserve business ip
LINQ 101
Group list be age via pivot table
Reverse engineer data to match cums
nCOVID-19 pivot-and-fan map
Fire archive modis6_n50
New Millennium London Fireworks
Mind maps for Cottenham Open
"Welcome back to the new Middle Ages"
3D GIS time travel
Zolnai geobyte manuscript
Azolnai geocom2017
Hurricane harvey copy
Mind Maps 4 GDPR
Geohipster Calendar, May 2018
Agi conf thanks_andrew_zolnai
To grid or not to grid
Geobase
Info Supply Chain for Decommissioning
Petex 2016 Future Working Zone
Unified online dashboards to preserve business ip
Ad

Recently uploaded (20)

PDF
[EN] Industrial Machine Downtime Prediction
PPTX
retention in jsjsksksksnbsndjddjdnFPD.pptx
PDF
Systems Analysis and Design, 12th Edition by Scott Tilley Test Bank.pdf
PPTX
Copy of 16 Timeline & Flowchart Templates – HubSpot.pptx
PPTX
Topic 5 Presentation 5 Lesson 5 Corporate Fin
PDF
Introduction to Data Science and Data Analysis
PPTX
Introduction to Inferential Statistics.pptx
PPTX
IBA_Chapter_11_Slides_Final_Accessible.pptx
PDF
How to run a consulting project- client discovery
PDF
Microsoft Core Cloud Services powerpoint
PPTX
QUANTUM_COMPUTING_AND_ITS_POTENTIAL_APPLICATIONS[2].pptx
PPTX
modul_python (1).pptx for professional and student
PDF
Data Engineering Interview Questions & Answers Cloud Data Stacks (AWS, Azure,...
PPTX
A Complete Guide to Streamlining Business Processes
PPTX
importance of Data-Visualization-in-Data-Science. for mba studnts
PPT
ISS -ESG Data flows What is ESG and HowHow
PPTX
mbdjdhjjodule 5-1 rhfhhfjtjjhafbrhfnfbbfnb
DOCX
Factor Analysis Word Document Presentation
PPTX
Database Infoormation System (DBIS).pptx
PDF
Transcultural that can help you someday.
[EN] Industrial Machine Downtime Prediction
retention in jsjsksksksnbsndjddjdnFPD.pptx
Systems Analysis and Design, 12th Edition by Scott Tilley Test Bank.pdf
Copy of 16 Timeline & Flowchart Templates – HubSpot.pptx
Topic 5 Presentation 5 Lesson 5 Corporate Fin
Introduction to Data Science and Data Analysis
Introduction to Inferential Statistics.pptx
IBA_Chapter_11_Slides_Final_Accessible.pptx
How to run a consulting project- client discovery
Microsoft Core Cloud Services powerpoint
QUANTUM_COMPUTING_AND_ITS_POTENTIAL_APPLICATIONS[2].pptx
modul_python (1).pptx for professional and student
Data Engineering Interview Questions & Answers Cloud Data Stacks (AWS, Azure,...
A Complete Guide to Streamlining Business Processes
importance of Data-Visualization-in-Data-Science. for mba studnts
ISS -ESG Data flows What is ESG and HowHow
mbdjdhjjodule 5-1 rhfhhfjtjjhafbrhfnfbbfnb
Factor Analysis Word Document Presentation
Database Infoormation System (DBIS).pptx
Transcultural that can help you someday.

CLIWOC Attributes

  • 1. Shipping and weather data CLIWOC ships captains logs 1662-1855 A Question on Joining Look-up Tables Andrew Zolnai Cambridge UK aiz@zolnai.ca Image: Royal Museum Greenwich
  • 2. Historic Climatology • Ships captains logs data from 1662 to 1855 • Ships and routing details and location • Weather recording of almost 120 parameters • Release 1.x (2004) and Release 2.x (2007)
  • 3. Location Metadata • First added to Esri file geodatabase – The complete (weather and location) dataset has almost 290,000 points, and is almost 250Mb database – The locations were extracted into a geodatabase (70 Mb), then into a compressed file geodatabase (10 Mb) • Post map package for Release 2.0 – layer files showing them in 25-year time-slices – and for Captains Cook and de la Perouse • Posted as ArcGIS Online map service – Use time stamps to aggregate layers – But stock server has performance issues – So Filter data by nationality and decade – Limits web fetches to 1000s not 100,000s
  • 4. (Visio Pro data model reverse-engineered from Dbase extracts of File Geodatabase)
  • 5. Climate Metadata • CLIWOC posted all shipboard data it had – from British, Dutch, French and Spanish sources – with look-up tables for each different source • Merged wind force and direction data almost doubles the number of feature classes as observed – Started with Wind Force – Then joined Wind Direction – Finally merged Force & Direction • Map package and layer template
  • 6. (Dbase extract 8 char limit: add prefix of CLIWOC21_Features_ to all except look-ups Lookup_{ }_WindDirection_)
  • 7. (Dbase extract 8 char limit: add prefix of CLIWOC21_Features_ to all except look-ups Lookup_{ }_WindForce_)
  • 8. Wind Direction from each Maritime Agency Look-up to Merged Wind Force from each Maritime Agency Look-up to Merged
  • 9. Question • Joining four lookup tables for each Nationality / Maritime Agency created redundant attributes • Performing a merge in Geoprocessing collapsed the redundancies into single computable columns • Why does it almost double the feature count? • GIS data Visio diagrams