Open-source Scientific Computing and Data Analytics using HDF
Download as PPTX, PDF1 like454 views
Kitware uses HDF as a widely adopted data format for scientific computing and visualization across several domains. HDF supports climate modeling, geospatial data, medical imaging, and more. Kitware is looking to improve HDF support for streaming big data, cloud computing, and web applications to enable more advanced analytics and sharing of scientific data. Future work may include pure JavaScript implementations of HDF tools and optimizing performance for cloud storage.
Open-source Scientific Computing and Data Analytics using HDF
- 1. Aashish Chaudhary aashish.chaudhary@kitware.com Technical Leader with Patrick O’Leary, Dr. Rama Nemani (NASA), Chris Harris, Chris Kotfila, Doruk Aztek, Andrew Michaelis (NASA) Open-source Scientific Computing and Data Analytics using HDF July 24th 2017 ESIP Summer
- 2. What We Do at Kitware? Open Source and Open Data is strongly encouraged and practiced at Kitware
- 3. It started with VTK
- 4. Parallel Processing and Rendering - Paraview
- 5. Computer Vision Images, Video, Point Clouds Recognition by Function Content- based Retrieval Event & Activity Recognition Anomaly Detection 3D Extraction and Compression Detection & Tracking
- 6. Medical Computing Quantitative imaging Electronic health records Vascular analysis Surgical guidance And simulation Digital pathology Orthopedic analysis Longitudinal and population shape analysis Interactive medical applications and visualizations
- 8. HDF at Kitware Climate Community High Performance Computing Extensible Data Model and Format - Developed to exchange scientific data between HPC codes and tools - Heavy data is stored using HDF5 Network Common Data Form (NetCDF) - Most projects use NetCDF4 Medical Community Vision Community Leading-edge algorithms for registering and segmenting multidimensional data
- 9. ACME The Accelerated Climate Modeling for Energy (ACME) project is sponsored by the Earth System Modeling (ESM) program (Biological and Environmental Research) with eight national laboratories and six partner institutions to develop and apply the most complete, leading-edge climate and Earth system models to challenging and demanding climate-change research imperatives. Most commonly used data format - NetCDF4 Data streaming using OpenDAP Python Interface for most of the tools
- 10. OpenNEX NEX is a platform for scientific collaboration, knowledge sharing and research for the Earth science community Global Daily Downscaled Projections (NEX- GDDP, NetCDF4) MODIS-Land and Atmosphere (HDF)
- 16. Possible Improvements Streaming and Big Data analytics - Any useful ingestion of HDF data into cluster requires ETL pipeline - For some tools, computation cannot move close to the data, streaming support is necessary in such cases - Optimal read/write on cloud storage Web-Support - More tools and projects are moving to support web-enabled data analysis and visualization - Pure JS implementation if possible
- 17. Summary ● HDF is widely data format for scientific computing, climate/geospatial visualization, and in other domains at Kitware ● Recently we have started using HDF for information visualization ● We are looking forward to HDF usage on cloud and web-environment ● Kitware is always looking for strong open source collaborations and is committed to push open-source scientific computing to its next level
- 18. Information Aashish Chaudhary: aashish.chaudhary@kitware.com LinkedIn: www.linkedin.com/in/aachaudhary Kitware: http://www.kitware.com NASA-NEX: https://nex.nasa.gov/nex Kitware-AIST: https://github.com/OpenGeoscience/nex HPC Cloud : http://www.kitware.com/publications/item/view/1784 HPCloud Github: https://github.com/Kitware/HPCCloud