BETTER Session, Excercise 1 @ EO Joint Big Data Hackathon
- 1. Data pipelines for inter-communal conflict impact analysis Terradue BETTER - Nigeria change detection 7-8 Novembre 2019 - Frascati Italy H2020 EARTH OBSERVATION BIG DATA HACKATHON
- 3. Earth Observation dataset: Sentinel-1 Search parameters: productType: SLC start: 2018-09-09 end: 2018-09-21 box: 5.1,10.8,8.1,13.7 Sentinel-1 data discovery over Nigeria Goal: discover a pair of Sentinel-1 SLC for change detection with the Coherence and Intensity change for Sentinel-1 (COIN) processing service https://ellip.terradue.com/geobrowser/?id=better-eobd-hack/better-hack-nigeria-app#!
- 4. Select the product that “covers the area of interest” Platform: S1A Sensor: SAR Product Type: SLC Start Time: 2018-09-16T17:46:57.0540000Z End Time: 2018-09-16T17:47:24.0030000Z Orbit:23727 ASCENDING Track:30 Mode: IW_DP Swath: IW1 IW2 IW3 Sentinel-1 data discovery over Nigeria Goal: discover a pair of Sentinel-1 SLC for change detection with the Coherence and Intensity change for Sentinel-1 (COIN) processing service
- 5. Sentinel-1 data discovery over Nigeria Earth Observation dataset: Sentinel-1 Search parameters: productType: SLC start: 2019-09-09 end: 2019-09-21 orbitDirection: ASCENDING track: 30 Platform: S1A box: 5.1,10.8,8.1,13.7 Goal: discover a pair of Sentinel-1 SLC for change detection with the Coherence and Intensity change for Sentinel-1 (COIN) processing service
- 6. Coherence and backscatter change detection COIN Processing service Use the discovered pair of Sentinel-1 SLC procuts to fill the processing parameters: - Master - Slave Then set the parameter - Pixel spacing in meters to 10.0 Run the job Goal: process Sentinel-1 SLC pair for change detection with the Coherence and Intensity change for Sentinel-1 (COIN) processing service
- 7. Coherence and backscatter change detection COIN Processing service Use the discovered pair of Sentinel-1 SLC procuts to fill the processing parameters: - Master - Slave Then set the parameter - Pixel spacing in meters to 10.0 Run the job Goal: process Sentinel-1 SLC pair for change detection with the Coherence and Intensity change for Sentinel-1 (COIN) processing service
- 9. Hands-on: data discovery, processing and data pipelines Steps: 1. Set-up your credentials 2. Data discovery 3. COIN processing 4. COIN results 5. Sentinel-2 RGB composites data processing pipeline 6. SNAC data processing pipeline 7. COIN data processing pipeline 8. SNAC exploitation Hackathon kit available at: https://gitlab.com/ellip/training/wfp-nigeria-training
- 10. Hands-on: access and setup Open a Web Browser (best experience with Chrome) Type the address: https://notebooks.terradue.com You’ll be redirected to a login page. Use the credentials found on the card given to you
- 11. Hands-on: access and setup Open the folder: wfp-nigeria-training Double-click on the notebook: 1 Setup.ipybnb Run the cells and fill the information requested: username: same as in card API key: click on the link shown copy/paste the API key Goal: set-up the authentication/authorization information for all notebooks
- 12. Hands-on: data discovery Double-click on the notebook: 2 Data discovery.ipybnb Run the cells to repeat the data discovery steps done on the Geobrowser Goal: discover the pair of Sentinel-1 SLC acquisitions with the Ellip APIs following the same approach done using the Geobrowser
- 13. Hands-on: data discovery Goal: discover the pair of Sentinel-1 SLC acquisitions with the Ellip APIs following the same approach done using the Geobrowser
- 14. Hands-on: Sentinel-1 COIN processing Double-click on the notebook: 3 COIN processing.ipybnb Run the cells to submit the COIN processing request as done on the Geobrowser Goal: discover the COIN processing service and submit a processing request with the discovered pair of Sentinel-1 SLC acquisitions
- 15. Hands-on: RGB composites data processing pipeline Double-click on the notebook: 4 COIN results.ipybnb Run the cells to inspect the processing results Goal: explore the results of the COIN processing service
- 16. Data processing pipelines rely on the ATC model with: - Application - data driven application - Trigger - date or event driven application - Coordinator - schedule the execution of the Trigger instances periodically at fixed times, dates, or intervals Data processing pipelines
- 17. The ATC model starts with a data driven application taking as inputs one or more references to EO catalogue entries: - A single input reference - A pair of input references (e.g. interferometry) - A stack of input references A(pplication)TC
- 18. The Trigger is a date and/or event driven application that, when there’s new data available or an event: - Creates Data Items - Queues Data Items - Pipes Data Items AT(rigger)C
- 19. OWS Context (*) documents encapsulating all the information for processing the application and get it through the data processing pipeline AT(rigger)C - Data items (*) https://www.opengeospatial.org/standards/owc
- 20. The Coordinator submits Trigger instances periodically at fixed times, dates, or intervals ATC(oordinator)
- 21. Hands-on: RGB composites data processing pipeline Double-click on the notebook: 5 RGB composites data processing pipeline.ipybnb Run the cells to create and queue Data Items containing Sentinel-2 RGB Composites processing requests The data processing pipeline will produce, for a set of areas of interest, several RGB composites for all Sentinel-2 acquisitions during the period 2019-08-15 to 09-15. Goal: create a data processing pipeline for Sentinel-2 RGB composites over a few areas of interest in Nigeria
- 22. Hands-on: SNAC data processing pipeline Double-click on the notebook: 6 SNAC data processing pipeline.ipybnb Run the cells to create and queue Data Items containing SNAC processing requests Goal: create a data processing pipeline for Sentinel-1 SNAC over a large area in Nigeria
- 23. Hands-on: SNAC data processing pipeline
- 24. Hands-on: COIN data processing pipeline Double-click on the notebook: 7 COIN data processing pipeline.ipybnb Run the cells to create and queue Data Items containing COIN processing requests Goal: create a data processing pipeline for Sentinel-2 RGB composites over a few areas of interest in NigeriaGoal: create a data processing pipeline for Sentinel-1 COIN over a large area in Nigeria
- 25. Hands-on: COIN data processing pipeline
- 26. Hands-on: monitor a data processing pipeline Double-click on the notebook: 8 SNAC exploitation.ipybnb Monitor the Data Items status and access the associated results when done Goal: monitor and exploit the results of the Sentinel-1 SNAC data processing pipeline over a large area in Nigeria
- 27. ▪ Get in touch with Terradue: support(at)terradue.com www.terradue.com ▪ Become a BETTER data challenge promoter: https://www.ec-better.eu/ Way forward Horizon 2020 Research and Innovation Programme under grant agreement no 776280