Application packaging and systematic processing in earth observation exploitation platforms

Application Packaging and
Systematic Processing in EO
Exploitation Platforms
Earth Observation
Exploitation Platform ad hoc
OGC Technical Committee
#OGC18FR, Orleans, France 20 March 2018

Thematic Exploitation Platforms

■ Specific fields in the domain
○ Geohazards domain
■ Earthquakes
■ Volcanoes
■ Landslides
○ Hydrology domain
■ Hydrological modelling
■ Flood mapping
■ Water quality monitoring
■ Data-intensive research
○ Targeted data
○ Targeted software
THEMATIC Exploitation Platforms

■ Very fast access to large
volumes of data
○ EO data
○ In-situ data
■ Distributed processing
○ Hybrid Cloud infrastructure
○ Access to GRID resources
Thematic EXPLOITATION Platforms

■ Collaborative tools
○ Referencing Publications
○ Sharing processing results
■ Different roles
○ Community scientists
○ Data providers
○ Resource providers
Thematic Exploitation PLATFORMS

An Ecosystem of Resources Providers
APIs of
Other Cloud
Services
APIs of
Other Cloud
Services
Data Providers
Data Providers
Tools & Services
for Exploitation
TEP
resources
Tools & Services
for Integration
EO Data
Exploitation
New EO Service
Development
New EO Product
Development
Data
Provisioning
Cloud
Provisioning
APIs of
Other Cloud
Services
Results sharing

EO Data
Interferometric search
Processing Tools
Inputs Selection & Job Execution
EO Data
Discovery and selection
Processing Tools
Listing
Usage Scenarios

Processing Tools
Visualize Results
Processing Tools
Monitor Progress
Social Web
Promote results
Usage Scenarios

Platform Data Agency
Metadata and Data Brokering
Sentinel-2 products
Amazon S3
Sentinel-1/2/3
products
Copernicus Scihub
Sentinel-1 products
Alaska Satellite
Facility
Data Agency
Terradue
Sentinel-1 products
(Sentinel 2/3
upcoming)
CNES PEPS
SHB
AS
F
PEPS
AWS
T2
Coll
GS
Landsat-8 products
EarthExplorer
USGS
Sentinel-1/2/3
products
National hubs (AT, GR, FI)

Cloud Processing Capacities
Federation of data centres
■ EGI.eu (BELNET, INFN RECAS BARI)
■ CNR IREA Computing Centre
■ Poznań Supercomputing and
Networking Center
Federation and Cloud Bursting
… and Cloud providers:
■ Hetzner / Terradue Private Cloud
■ EO IPT.PL
■ Amazon AWS
EC2 - powered by jclouds
Amazon Web Services
AWS
OCCI
EGI Federated Cloud
EGI
Openstack API -
powered by libcloud
IPT.PL
IPT
Openstack API -
powered by jclouds
Poznań Supercomputing
and Networking Center
PSNC
OpenNebula
registered partner
Terradue
T2
Apache CloudStack - powered
by *cloudmonkey*
Interoute
IR
Cloudsigma API
[upcoming]
CloudSigma
CS

Systematic SAR Data Processing
● Systematic processing using the InSAR Browse of DLR.
Each tile has the extent of an S-1 footprint.
● Providing amplitude (cyan) and
coherence (red) of all consecutive
S-1 pairs over the world tectonic mask.
● Current production rate: 150 pairs/day
● About 1TB data cached and processed daily
● Published results available freely to any user
Systematic production of S-1 InSAR pairs

Systematic Optical Data Processing
Surface Temperature Map developed and integrated by INGV,
processes systematically Landsat-8 data
to measure the temperature for areas
identified as «thermally active» in the
volcanic areas (22 volcanoes worldwide)
STEMP L8 – Surface Temperature Map

Processing Services deployed
S-1 InSAR Browse 100m, 50m spacing (DLR)
S-1 DIAPASON Displacements (CNES, TRE-Alta) S-1 SNAP Displacements (RSS)
S-1 InSAR Browse 50m, 25m spacing (DLR)
L-8/S-3/S-2 Surface Temperature Map (INGV)
S-2 Vegetation Vigor Maps (NOVELTIS) S-2 Hot Spot Maps (NOVELTIS)
L-8/S-2 Landslide Deformation (UNISTRA)
Up & East-West terrain motion (CNR IREA)
S-1 FASTVEL Terrain motion velocity map (TRE)
S-1 SBAS on-demand service (CNR IREA)
S-1 SBAS Surveillance (CNR IREA)
S-2/SPOT/Pléiades DSM Generation (UNISTRA) Landslide risk maps (CNR IRPI)
Conventional
InSAR
Advanced InSAR
Optical
Others ENVISAT SISTEM (INGV Catania)
TIO - Optical NSBAS (ISTERRE)
L-8/S-2/SPOT/Pléiades Landslide Deformations & Inventories (UNISTRA)
PSI Post-Processing (TRE)
2018
(*) partial list, additional 22 services are upcoming or planned
S-1 SNAP COherence and INtensity (RSS)
GEP Systematic Processing
GEP On-demand Processing

Monitoring Systematic Productions
Monthly consumption of 7K CPU/hours for processing generating more
than 15K products per month (even before entering in pre-operations)
Full-fledged dashboard with custom metrics to monitor processing

Monitoring Systematic Productions
Routine monitoring for processing data access
Data Gateway Access in 2018
before starting the pre-operations phase

GEP and EPOS
A long-term plan to facilitate integrated
use of data, data products, and facilities
from distributed research infrastructures
for solid Earth science in Europe
● GEP is providing EO data processing expertise in exchange for
access to the EPOS in-situ data network
● GEP will be available to EPOS IP users and provide the interface
for the Satellite Data Thematic Core Services towards the EPOS
central hub.
EPOS - European Plate Observing System

The GEP and international initiatives
● Taking lessons from the GEP experience, a broader approach
developed jointly with space agencies ESA, CNES, DLR and ASI
○ aims to address priorities of the Sendai Framework for Disaster Risk Reduction 2015-
2030 taking advantage of satellite EO
○ goal: provide an EO processing & e-collaboration environment to exploit EO data to
assess geohazards and their impact
○ elaborated in the framework of CEOS WG Disasters and accepted by CEOS
○ a resource to support CEOS WG Disasters Pilots (e.g. Seismic Hazards, Landslides,
Volcanoes, Recovery Observatory) and GEO DARMA
● The GEP is also selected by the CEOS WGISS alongside the Data
Cube framework
Geohazards Lab Initiative

GEP & Disaster Risk Management Services
● International Charter Space and Major Disasters
selected a GEP Thematic Application for testing the
support to the Project Manager and Value Adding
partners of Charter activations:
○ Faster access to data and ancillary datasets (e.g. DEM, OSV, etc.)
○ On-line pre-processing of images (co-registering multi-sensor collections),
○ On-line image analysis (e.g. change detection, a classifier to generalize some
processing)
● ESA initiated a special contract to develop the GEP based prototype
to evaluate Exploitation Platform potential contribution to disaster
response
Charter Platform prototype

Ongoing Activities
● Thematic Apps: rule-based bundling of data collections and
processing services
○ Targeting specific usage scenarios
○ Processing services accessed on the GEP Portal via “Discover Thematic Apps”
○ My data uploads App: for user uploads on Data Gateway, with automated cataloguing
● Terradue Data Agency
○ Automatic multi-sourcing
○ Programmed cache
○ Personal storage (repository)
○ Production staging
○ Data usage accounting
New exploitation platform capabilities agreed upon with ESA

User Communities
Applying Exploitation Platform concept and technology in other domains
ESA Exploitation Platforms
(2014 onwards)
60+
EC FP7 MELODIES
(2014-2016)
14
EC FP7 SenSyF
(2013-2015)
8
EC FP7 GEOWOW
(2011-2013)
5
Processing
Campaigns
(2013-onwards)
30+
H2020 Co-ReSyF
(2016-2018)15+
10+
H2020 NextGEOSS
(2016-2020)
↗
H2020 ECOPOTENTIAL
(2015-2018)5+
↗
H2020 EVER-EST
(2015-2018)4+
↗
H2020 INTAROS
(2016-2021)2+
↗
H2020 Copernicus App Lab
(2016-2018)3+
H2020 BETTER
(2018-2021)
100+

Open Cloud Strategy
■ Open APIs
Embrace Cloud bursting APIs that can
be easily plugged into the Platform’s
codebase, so to expand the Platform
offering with Providers offering
complementary strategic advantages for
different user communities.
■ Developers community
Support and nurture Cloud communities
that collaborate on evolving open
source technologies, including at the
level of the Platform engineering team,
when it comes to deliver modular
extensions.
Roadmap for interoperability
■ Self-service provisioning &
management of resources
The Platform’s end-users are able to self-
provision their required ICT resources
and to work autonomously.
■ Users rights to move data
as needed
By supporting distributed instances of
its EO Data management layer, the
Platform delivers the required level of
data locality to ensure high performance
processing with optimized costs.

Earth Observation Exploitation Platform
● This Domain Working Group charter defines the role for OGC
activities within the Earth Observation Data Exploitation platform
domain
● Provide an open forum for the discussion
● Presentation of interoperability requirements, use cases, pilots,
and implementations of OGC standards in this domain
● Promote OGC standards
It will not define a new platform or a new architecture
OGC Domain Working Group

Earth Observation Exploitation Platform
Proponents:
Nuno Catarino (DEIMOS)
Chris Holmes (PLANET)
Stan Tillman (Hexagon)
Pedro Gonçalves (Terradue)
OGC Domain Working Group

Thanks !
▪ Pedro Gonçalves
pedro@terradue.com

Application packaging and systematic processing in earth observation exploitation platforms

More Related Content

What's hot (20)

Similar to Application packaging and systematic processing in earth observation exploitation platforms (20)

More from terradue (20)

Recently uploaded (20)

Application packaging and systematic processing in earth observation exploitation platforms

Editor's Notes