DI4R 2018 - Ellip: a collaborative workplace for EO Open Science
0 likes296 views
The document outlines the features and goals of Terradue's collaborative platform, ellip, designed to enhance earth observation (EO) data usage for various scientific applications. It emphasizes the importance of value co-creation and partnerships to drive innovation in the Earth sciences community, highlighting ongoing projects and available resources. The platform aims to foster user-driven collaborative efforts while ensuring data accessibility, interoperability, and sustainability through robust cloud infrastructure.
DI4R 2018 - Ellip: a collaborative workplace for EO Open Science
- 1. Ellip: a collaborative workplace for EO Open Science Lisbon, October 9th , 2018 Hervé Caumont Platform Operations, Terradue
- 2. Copernicus Sentinels as the nextgen Earth Observation satellites Typical EO data volumes 500MB (S-2) < a single product < 8GB (S-1)
- 3. ~5000 ~400
- 4. Why Terradue ? ▪ Terradue is an ESA spin-off (2006) ▪ From day one providing support to application builders in earth sciences, to use satellite EO data as information source ▪ Early days Grid Computing. Nowadays, Cloud PaaS, complemented with APIs for Cloud bursting, aimed at the transfer in production of released applications ▪ Business model: Platform-centered, a collaborative workplace for value adders to interact & co-createNow opening new Ellip solutions to beta testers, who will help us improve our Platform services
- 5. Why value co-creation ? ▪ There is still need for significant capacity building within the earth sciences community in exploiting EO data ▪ We believe the value co-creation model is the way forward to unleash innovation ▪ Since 2014 we’re evolving Terradue Cloud Platform, user-driven, via ESA & EU funding ▪ A steady and careful process, as well as a learning process Now opening new Ellip solutions to beta testers, who will help us improve our Platform services
- 6. Funding & User communities
- 7. Hosted Communities ↗ H2020 Copernicus App Lab (2016-2018) ↗ H2020 INTAROS (2016-2021) ESA Exploitation Platforms (2014 onwards) 70+ EC FP7 MELODIES (2014-2016) 14 EC FP7 SenSyF (2013-2015) 8 EC FP7 GEOWOW (2011-2013) 5 Processing Campaigns (2013-onwards) 35+ ▪ Climate Change ▪ Carbon market ▪ Oceanography ▪ Climatology ▪ Biodiversity ▪ Volcanology ▪ Ship logistics ▪ Water quality ▪ Desertification ▪ Seismic analysis ▪ Urban accounting ▪ Land management ▪ Disaster Risk Reduction H2020 Co-ReSyF (2016-2018)15+ 10+ H2020 NextGEOSS (2016-2020) ↗ H2020 ECOPOTENTIAL (2015-2019)5+ ↗ H2020 EVER-EST (2015-2018)4+ 4+ 3+
- 8. ▪ EO data is a unique source of global measurements over decades ▪ When calibrated and combined with other sources, it empowers validation & interpolation models ▪ Collaboration process: a feedback loop of providers & consumers ▪ More providers of pre-processed data layers ▪ New demands for structured information The growth model is pulled by the communities Earth Observation Platform Producers Consumers
- 9. Partnerships We are always seeking for new Partnerships to expand our Platform services to users : ▪ Data providers ▪ e-Infra / IaaS providers ▪ Technology providers ▪ Scientific Toolbox/Algorithm/Model providers A particular focus from community requirements is to boost the joint use of EO & in-situ data
- 10. EGI.eu & Terradue Partnership M2M communications realised initially via OCCI and then via native APIs for cloud computing, ensuring integration, portability, and interoperability. Use of standard interfaces We can rely on well-defined operational processes (e.g. FitSM), having the EGI Foundation acting as single point of contact of the EGI Federated Cloud. Reliable operations We use Cloud resources within the European boundaries. In fact, it is mandatory for certain processing services due to their organisations’ policies. European cloud resources The partnership with EGI Foundation enhances the sustainability of the involved projects, via SLAs / OLAs helping to sustain operational lifetimes. Project(s) sustainability Oct. 2018 https://www.egi.eu/about/newsletters/egi-terradue-a-long-lasting-partnership/ Nov. 2015
- 11. Integration roadmap 2016/09 - 1st MoU 2015 2016 2017 2018 Feasibility Setup Dev Acceptance Pilots(cntd) Monitor (cntd) Monitor Pilots Acceptance (cntd) 2018/09 - 2nd MoU
- 12. Other partnership examples Science providers: ● research and commercial (GEP, HEP, NextGEOSS) Data providers: ● EPOS RI on Geohazards TEP for data uploads and processing ● Open Data policies in the EO data world ● (cf. RDA interview) Technology providers: ● OpenNebula (since before ESA TEPs early stage, circa 2012) ● OGC standards (Terradue as standard editor) Cloud IaaS providers: ● EGI.eu (NextGEOSS) ● EOSC.eu (EOSC-hub) ● DIAS (AppLab, GEP) https://www.rd-alliance.org/interview-terradue
- 14. ● Since Terradue’s inception 12 years ago, our company DNA pushes us to innovate ● We believe technology serves new ways to design products, that people can use to better work collaboratively ● Platform Operations goals ○ Manage well defined usage scenarii as ‘products’ ○ Pilot a unified foundation for software solutions ○ Innovate smoother, end-to-end business processes matching customer needs ○ Guarantee the ability to share data among applications ○ ... Why do Platform Operations exist ?
- 15. ● Platform Operations require a thoughtful mix of stability & innovation ● We’re embarking on a new journey to learn how to tune this mix, based on: ○ The market needs ○ Our own skills and capacities to deliver software of high quality ○ The cost / benefit of the solutions that we decide to market ○ The market recognition of what we do How to run Platform Operations ? Stability Timely Innovation
- 17. How to work with Ellip ?
- 18. Federated IdPs & Terradue SSO Possibility to keep your preferred Identity Provider OpenID Connect preferred Under work: User Managed Access (UMA) profile of OAuth 2.0
- 20. • Business model for sustainability of hosted processing operations • Public funding, commercial revenues, and costs-sharing • Cloud providers, • Processors developers, • Applications Integrators • Scientific users Subscription Plans EXPLORER Low cost App. development For Data science users SCALER Low cost Cluster processing For Producer users PREMIUM PARTNER Higher cost Innovation & platform evolutions For Integrator users 4 Partner Programs ◆ Technology Alliance ◆ Virtual Archive ◆ Data Science ◆ Cloud Appliance
- 21. Ellip Dashboard
- 22. ● Ellip Notebooks ○ “Create interactive laboratory notebooks” ● Ellip Workflows ○ “Design scalable processing chains” ● Ellip Launchpads ○ “Select a data processor and deploy it” ● Ellip Infohubs ○ “Connect a WPS to a custom Geobrowser” Ellip Solutions Vestibulum congue Vestibulum congue Vestibulum congue Vestibulum congue Ellip Infohubs Ellip N otebooks Ellip Launchpads Ellip W orkflow s
- 23. Ellip Solutions in a nutshell I want to build and maintain a processing service Interactively prototype Integrate for scalability Run a scale Monitor and collaborate
- 24. Vestibulum congue Vestibulum congue Vestibulum congueVestibulum congue Ellip Exchange Ellip Infohubs Ellip N otebooks Ellip Launchpads Ellip W orkflow s The Ellip Exchange as the core enabler of Ellip as a ‘collaborative workplace’ . Application code . Packaged application . Tutorial . Running WPS at a selected Cloud Provider . New information layers . Reproducible experiment as code . Interactive data visualisations . OWS Context template . ad-hoc Community Portal
- 26. So, what does a platform mean nowadays? The Ellip Solutions are built for earth sciences practitioners: ▪ to share about their findings, streamline their creation of new data products and make these accessible, interoperable, ▪ and document their developments on the PaaS as reproducible experiments In line with the FAIR guiding principles to make data findable, accessible, interoperable and reusable
- 27. BuildIntegrate & Test Package Dockerize Code Commit Scan Repository Production Environments Continuous Integration & Deployment Code Upload Application Release Integration Environment
- 28. Hybrid Cloud Openstack API - powered by jclouds Poznań Supercomputing and Networking Center EC2 - powered by jclouds Amazon Web Services OCCI, native cloud APIs EGI Federated Cloud Opennebula registered partner Terradue Openstack API - powered by libcloud IPT.PL AWS EGI IPT PSNC T2
- 29. www.terradue.com Scheduled Data Processing 29 24 processing nodes provided by EGI.eu (NextGEOSS partner) Growing season 2016-2017 Biopar products Terradue Cloud Platform Biopar Application Burst to EGI.eu
- 30. www.terradue.com Scheduled Data Processing 30 It’s a data driven systematic processing. The service will follow a ramp-up period starting from September 2016 until end 2017: ■ EU Tectonic area ■ World tectonic area (40%) ■ World tectonic area (70%) It currently processes 150+ Sentinel-1 SLC pairs per day. DLR InSAR Browse Medium Resolution Service Supported by BELNET-BEGRID (Belgium)
- 31. www.terradue.com User-Driven Data Processing 31 This service provides the coherence and intensity changes for Sentinel-1 TOPSAR IW data pairs performed through SNAP. SNAP is a common architecture for all Sentinel Toolboxes, which is ideal for Earth Observation processing and analysis. ESA SNAP Sentinel-1 COherence and INtensity (COIN) Supported by GOEGRID-GWGD (Germany)
- 34. All of it: an Open Cloud strategy ■ Open APIs Embrace Cloud bursting APIs that can be easily plugged into the Platform’s codebase, so to expand the Platform offering with Providers offering complementary strategic advantages for different user communities. ■ Developers community Support and nurture Cloud communities that collaborate on evolving open source technologies, including at the level of the Platform engineering team, when it comes to deliver modular extensions. ■ Self-service provisioning and management of resources The Platform’s end-users are able to self-provision their required ICT resources and to work autonomously. ■ Users rights to move data as needed By supporting distributed instances of its EO Data management layer, the Platform delivers the required level of data locality to ensure high performance processing with optimized costs.
- 35. All of it: an Open Cloud strategy ▪ The Ellip subscribers remain the owners of their created assets, and decide how to share these on the platform, for other users to find and reuse these assets. ▪ They are also empowered to decide on the target production environment of their packaged applications (no lock in with Cloud provider) Producers Consumers
- 36. That’s it! ▪ Last message: we’re always looking for new collaborators ▪ Join our team in Rome La Città Eterna 15 members, 5 EU nationalities ▪ We are welcoming experts in EO applications and cloud technologies ▪ Exciting new challenges rolling out Ellip for Earth Science
- 37. Looking forward hearing from you! https://www.terradue.com Hervé Caumont, Platform Operations herve.caumont@terradue.com