iRODS: Interoperability in Data Management
Download as PPTX, PDF2 likes1,329 views
iRODS is an open source data management software developed by DICE at UNC and UCSD as a follow-on to SRB. It provides a customizable, policy-driven framework for implementing data grids and managing data across heterogeneous storage resources. Key features include modularity, extensibility through microservices and rules, and interoperability with systems like HDF5, NetCDF, and storage systems through integration extensions. RENCI provides support and commercial offerings around iRODS through their E-iRODS distribution.
iRODS: Interoperability in Data Management
- 1. iRODS: Interoperability in Data Management Leesa Brieger, RENCI-UNC Mike Wan, DICE-UCSD
- 2. integrated Rule-Oriented Data System (iRODS) • Developed by the Data Intensive Cyber Environments (DICE) group, UNC and UCSD • Follow-on to SRB, the Storage Resource Broker from SDSC – decade-long development experience, community-driven • Modular, extensible, customizable • Open source (BSD license) • Supported by the Renaissance Computing Institute (RENCI), UNC – a research unit of UNC Chapel Hill – state-supported – governed by the Triangle universities (UNC, NCSU, Duke) HDF, HDF-EOS Workshop XV, April 1719, 2012 2
- 3. iRODS I. Data grid middleware II. Data management infrastructure III. Framework for implementing policy-driven data management The extensibility and modularity of iRODS make it a customizable and resource-agnostic infrastructure. HDF, HDF-EOS Workshop XV, April 17-19, 2012 3
- 4. iRODS as Data Grid iRODS View of Distributed Data User Client User sees a single collection My Data: disk, filesystem, WOS storage unit... My Data: tape, database, filesystem ... Partner’s Data remote disk, tape, filesystem... •iRODS installs over heterogeneous data resources • Users share & manage distributed data as a single collection • iCAT metadata catalogue: DB that manages the logical-tophysical mappings (data objects, users, resources) HDF, HDF-EOS Workshop XV, April 1719, 2012 4
- 5. Data Life Cycle Usage evolves across stages of the data life cycle; management policy evolves along with it. Creation Active Use Publication & Sharing Local Policy Reference Collection Service/Use Distribution Discovery and Re-purposing Archival Collection/ Deletion Retention/ Preservation iRODS modularity and extensibility allows support for changing s ds management requirements over the data life cycle. HDF, HDF-EOS Workshop XV, April 1719, 2012 5
- 6. iRODS Design Goals • Data grid abstraction for data, users, resources • Abstract out the data management – Separate data administration from storage administration • drivers allow iRODS to talk local storage protocol • rule engine runs services and data operations – Policy-based data management • Data management: specialized modules of microservices (C code) and rules for running data-side services • Policy-based: event-triggered rule execution – Policy follows data around the grid • collection management independent of remote storage HDF, HDF-EOS Workshop XV, April 17-19, locations 2012 6
- 7. Interoperability • Federation – Data grids with independent administration can federate and crosscommunicate • Clients – User-supplied or specialty client interfaces – Many specialized views of the collections • iRODS core extensions for resource agnosticism/fitting in with existing infrastructure – – – – network transport (RBUDP) authentication mechanisms (Kerberos, Shibboleth, GSI, etc) external databases (DataBase Resources - DBRs) storage drivers (HPSS, WOS, EC2, etc) HDF, HDF-EOS Workshop XV, April 17-19, 2012 7
- 8. Interoperability Through Microservices iRODS provides a structure for implementing custom services – Rules and microservice modules – Can be user-defined – Data-side services: format conversion, extraction, visualization, accounting &reporting, … – Archival: replication, curation procedures, long-term archival procedures – Access: access control policy – Discoverability: metadata organization and management – Symbolic links: integrate data from other collections into iRODS repository • microservice drivers – Universal mass storage driver – plug in new protocols HDF, HDF-EOS Workshop XV, April 1719, 2012 8
- 9. Interoperability Through Integration with Existing Infrastructure • Data management integrated with storage management: OSG, DDN • Data management integrated with standard interfaces and services: – – – – Fedora (librarians) DataVerse (social scientists) HDF5 (cosmologists) NetCDF (NASA climate scientists, NSF earth scientists - hydrologists) HDF, HDF-EOS Workshop XV, April 1719, 2012 9
- 10. Integration with HDF5 Mike Wan and Peter Cao, 2008 Interactive access to HDF5 files on a remote iRODS server – browsing of metadata and data sharing with services • Clients access to data (subsets) and metadata in HDF5 files stored remotely; transfers only of requested data and metadata, not of full files • iRODS microservices and APIs created to support HDF5 functionality on HDF5 objects • islice – extracts a slice from a FLASH (cosmology) file stored on a remote iRODS server • Remote viewing of HDF5 iRODS data • HDFView HDF, HDF-EOS Workshop XV, April 1719, 2012 – iRODS HDF5 Java objects were added to the HDF-Java products 10
- 11. Integration with NetCDF Mike Wan, 2012 • Add NETCDF functionalities to iRODS: – wrap NETCDF APIs into iRODS APIs and micro-services • New iRODS APIs to wrap basic NETCDF APIs (libnetcdf) and a higherlevel libcf subsetting function – Basic: nc_create, nc_open, nc_close – Inquiry functions: nc_inq_varid, nc_inq_dimid, nc_inq_dim, nc_inq_var – Subsetting functions: nc_get_vars_text, nc_get_vars_string, nc_get_vars_int, nc_get_vars_float, nc_get_vars_double, … – Higher-level subsetting function of libcf for CF data: nccf_get_vara • New NETCDF-based iRODS micro-services – Allow NETCDF workflows to be performed data-side on the iRODS servers HDF, HDF-EOS Workshop XV, April 17-19, – One for each of the new APIs, for server-side operations 11 2012 – 5 micro-services for accessing data elements in the new data structures
- 12. iRODS for Interoperability – NASA (NCCS) Separating metadata from the data object (from NetCDF files into the iCAT) Using an iRODS FUSE client to expose data to the ESG Data Node In support of discovery, long term curation, and reuse/repurposing of the data HDF, HDF-EOS Workshop XV, April 1719, 2012 12
- 13. E-iRODS from RENCI – the RedHat Model • Initial release based on iRODS 3.0 – Tracks community code, with a delay – Download beta release binaries at http://e-irods.com • Hardened binary release of iRODS – Passes continuous integration with back-ported bug fixes from community trunk – Packaging and signing: initially RPM and DEB • Certification • Documentation • Subscription Support Contracts – leesa@renci.org for information HDF, HDF-EOS Workshop XV, April 17-19, 2012 13
Editor's Notes
- #7: This is the first mention of microservices… defined in the next slide.