Big Data Solutions for the Climate Community
0 likes95 views
1) The document discusses interoperability between climate data systems like ESGF and computing infrastructures like EGI and GEF to more efficiently access and analyze large climate datasets. 2) It demonstrates running a climate analysis job on EGI infrastructure using the GEF framework and quantifies performance gains from accessing remote data and computing resources. 3) Future work is proposed to further optimize performance through parallelization across multiple VMs, and to develop tighter integration between data repositories and computing systems like ESGF-EGI-GEF.
Big Data Solutions for the Climate Community
- 1. www.eudat.euEUDAT receives funding from the European Union's Horizon 2020 programme - DG CONNECT e-Infrastructures. Contract No. 654065 Big Data Solutions for the Climate Community ESGF – ENES – GEF – EGI interoperability Xavier Pivan, CERFACS Christian Page, CERFACS EUDAT Porto’s Conference Porto 22 – 25 Janvier 2018
- 2. Summary Context & motivation. Introduction to the Generic Executive Framework (GEF) GEF – EGI Federated Cloud interoperability ENES usecase - Live demo or Video Performance quantification ESGF – ENES – (GEF – EGI) interoperability
- 3. Context & Motivation GEF presentation
- 4. Current situation Climate Research Community u Data available for scientific analysis: a very large trend § Limitations in data access means limitations in data analytics and scientific results u Download locally then Analyze: a workflow that cannot be sustained § Climate researchers § Impact researchers
- 5. Current situation Practical Example: Climate Community Federation Service • Temperature at 850 hPa field • 10 climate models • 1960-1990 & 2040-2070 = 60 years = 21 915 days • Daily fields = 1 field per day • Global spatial scale 100 km resolution TOTAL: 6 754 500 fields to download ~100 Kb per 2D field = 626 Gb After the analysis post-processing • Anomaly of the average of the two periods over a specific country for each climate model • Result: 10 times 2D fields over a small domain • Estimated datasize after post-processing: 1 Mb Data reduction...
- 7. GEF – a few years ago
- 8. About the EUDAT Generic Executive Framework Generic tool to encapsulate calculation using docker technology Generic -> convenient for all communities (Climate, Earth Science or Litterature) Community admin can create specific services
- 9. Generic Executive Framework (GEF) **Figure from GEF github
- 11. GEF – EGI interoperability European Grid Infrastructure (EGI) Federation of Cloud e-infrastructure World-wide but mainly in Europe Generate large computing resources on EGI Use GEF docker rule engine on EGI => How to generate resources ? => How to make EGI computing resources and GEF interoperable?
- 12. Generate EGI resources & set-up GEF on EGI docker image - GEF service repository x509 credential user user private key Contextualisation (.yml) jOCCI API - java Create proxy Instantiate VM with EGI appDB Describe VM attribute using VM id Return VM id (URL) Public IP VM computing state Import JSON file Input from EGI appDB • endpoint • resource template • operating system Some other inputs • … 3) Start the GEF with EGI VM docker endpoint + client certificate path 1) Create client/server TLS certificate 2) Bind EGI’s docker daemon to the VM’s IP & port 2376
- 13. ENES usecase - Icclim Icclim stands for: Index Calculation CLIMate Open Source python library developed by CERFACS for Climate Community Perform calculation on a netCDF file (average e.g) Return the result on a new netCDF file
- 14. Encapsulate the icclim calculation ENES_UseCase_icclim.py Average the netCDF downloaded inside the container and return a new netCDF file Data input Format file: netCDF Variable: tas Calculation: mean Data output Format: netCDF Variable: tas
- 15. ENES usecase Big Picture before demo
- 16. Time to relax GEF – live demo – video
- 17. GEF – EGI perfomance quantification
- 18. Measurement Methodology ENES usecase workflow is split between two main phases: Downloading the data Computing performed on data Quantification of the performance: Measuring the time of each phase
- 19. GEF – EGI performance quantification Comparaison between EGI – CERFACS – DOMESTIC network on a 3.41GB dataset EGI downloading speed faster from 7 to 10 times Computing process time about 2 to 3 times faster Weakness of this quantification: only 1 sample Network comparaison Computing performance comparaison
- 20. GEF – EGI performance quantification Experiment: Run ENES usecase on data ranging from 1GB to 200GB (multiple files) Fit the result with a linear curve Realize projection on potential larger data up to 2TBs Analyze of the downloading & computing time
- 21. GEF – EGI performance quantification Downloading time Figure: Downloading time vs Data Volume Downloading perfectly linear Linear fit curve
- 22. GEF – EGI performance quantification Computing time Figure: Computing time vs Data Volume Linear approximation Linear fit curve
- 23. GEF – EGI performance quantification Downloading + Computing time Figure: Workflow time vs Data Volume Linear fit curve (of course)
- 24. GEF – EGI performance: Projection up to 2TBs Download Time about 42 hours Computing Time: 20 hours Total time: 62hours
- 25. GEF – EGI performance conclusion Projection on 2TB: Perfomance is linear – acceptable Parallelisation should reduced both downloading & computing It means create VM machine – Cloud orchestration Could be perform with the jOCCI Icclim has option to realise fragmentation -> Find the best value related to a given data volume
- 26. GEF – EGI performance conclusion Area where using multiple VM on EGI could reduce the Total time
- 27. GEF – EGI performance conclusion It would imply instauration of some ”convention” Allowing user to perform calculation up to a certain amount of data Special authorization if calculation
- 28. Get closer to the data Downloading file is the most costly phase (~66%) Considerably reduce this phase by getting closer to the data Implies to have computing nodes and data nodes working together This is the ESGF - ENES – GEF – EGI interoperability
- 29. Enlarge the interoperability Climate for Impact (C4I) description European portal for climate research Earth System Grid Federation (ESGF) American server cwt allow to perform calculation close to the data Generic Executive Framework (GEF) European service developed by EUDAT European Grid Infrastructure (EGI) European e-infrastructure
- 30. Climate for Impact Portal ESGF – IS-ENES – EGI – GEF “Big Picture”
- 31. How C4I looks like:
- 32. Conclusion GEF – EGI interoperability: Perform data reduction Offer faster downloading and computing performance New solution to explore: Cloud orchestration Quantify how faster the downloading and computing using multiple VM & docker swarm Icclim optimisation ESGF – ENES – GEF – EGI in development Very likely to strengthen our big data approach Should be close to be operational by this year