ICHEC - Observation systems, technologies and big data
0 likes1,080 views
ICHEC is a national technology center established in 2005 to work on high performance computing, big data, and data analytics. It has expertise in processing large remote sensing datasets using workflows on HPC systems. ICHEC proposes establishing an "exploitation portal" to process remote sensing data as it arrives in Ireland using automated workflows. This would allow for producing climate and environmental products for Ireland as well as combining remote sensing data with other datasets and models through data assimilation techniques. ICHEC argues it could provide a platform for curating useful remote sensing datasets and products for Ireland over the long term.
ICHEC - Observation systems, technologies and big data
- 1. Observation Systems, Technologies and Big Data Alastair McKinstry EPA Climate Workshop, 19 September 2013
- 2. ICHEC overview National Technology Centre Established in 2005 Hosted by NUI Galway Mandate: HPC & Big Data/Data Analytics Industry engagement Platform Science & Technology 25 staff in Dublin & Galway Mix of software developers, domain specialists 4 in Climate/Environmental area EPA Climate Workshop, 19 September 2013 2
- 3. Old vs New: a x1000 step change Old: • Everybody downloads the data • e.g. data on 50km grid. Few MB/day. • 1-3 bands. New: • Move the work to the data – 100+ GB/day – 20-60m resolution, 12-15 bands EPA Climate workshop, 19 September 2013 3
- 4. Big Data: Networking • ICHEC and HEAnet have 10gb links – Not affordable on commercial rates – Used in CMIP5 data project with eINIS – Point-to-Point with European partners • Move one copy to Ireland, process it at an “Exploitation portal” – Share workflows. – Processing triggered on data arrival EPA Climate workshop, 19 September 2013 4
- 5. Big Data: Compute • Workflows are no longer a “hobby” task – Not on a simple PC at 20-50m, but … • GPGPUs/ Intel MIC Accelerators: – 80 Tflop/s of capability on upcoming ICHEC system – C.f. 40 Tflop/s needed to process EUMETSAT data • Shared workflows: atmospheric correction, QA • ICHEC has portal experience: BDI, Bioportal, • Automated: repeatability. SFI Review – Royal Irish Academy, Dublin – 21st October 2010 5
- 6. Curation: an unsolved problem • What to keep? • Useful to Ireland: – Products, raw data not archived at primary sites – Archiving “just Ireland” gives valuable time series ICHEC could provide a platform for this: – Funding needed from Beneficiaries or agencies. – Lack of sustainability a problem (C4I, CMIP5) – Curation needs human work: data scientists. SFI Review – Royal Irish Academy, Dublin – 21st October 2010 6
- 7. Processing in Ireland ? • Some products may not be produced upstream – E.g. Algal blooms for North Atlantic • Need rapid processing of raw data • Critical for aquaculture • Time critical. – May pave way for ground station for later satellites EPA Climate workshop, 19 September 2013 7
- 8. Data Fusion • Combining Remote Sensing data with other datasets: – Ground truthing – Precipitation, soil moisture (SMOS), runoff, river gauges, … • Needs consistent data, interoperability: – Technical limitations – Orgs. To make data available to each other: collaborations EPA Climate workshop, 19 September 2013 8
- 9. Combining with models • Experience with weather and climate • Coupling models and data assimilation key science skills at ICHEC • “Virtual Ireland” : assimilating observations and model data for • Pollution control: e.g. ICOS • Flooding, hydrology • Policy analysis EPA Climate workshop, 19 September 2013 9
- 10. Other datasets • Not just Remote Sensing: – Make other datasets available: same grids, etc. • Model data, observations, – Somewhere for users to upload data: • Indexed, Archived, remapped to new formats • Data scientists who understand metadata and the science behind the data EPA Climate workshop, 19 September 2013 10
- 11. Citizen Science • Data to the citizen: – A portal for making datasets available: – Making WxS layers available for GIS, Google Earth, … – Enable “mashups”, analysis apps. • From the citizen: – Smart apps for uploading observations, measurements EPA Climate workshop, 19 September 2013 11
- 12. Global opportunities • Commercial spinoffs: tech. startups looking for testbeds of global opportunities – Promote tech. sector in Ireland, not just exploitation of data in Ireland e.g. showcase big databases, fast networks EPA Climate workshop, 19 September 2013 12
- 13. New value in old data • The big investment has been made – Ireland’s contribution to ESA, – “Random” Datasets in public sector, academia • Applications in: – Agriculture – Policy and planning – Tourism EPA Climate workshop, 19 September 2013 13