Building a Tiered Digital Storage Environment on User-Defined Metadata to Enable eResearch
1 like206 views
The document outlines the development and functionalities of iRODS, a next-generation hierarchical storage management and data management system designed for the needs of e-research and big data. It emphasizes the importance of user-defined metadata in automating data movement and organization for analysis, while ensuring data provenance, traceability, and reproducibility. iRODS enables seamless data management through automation and customizable capabilities, addressing the challenges posed by the increasing volume and complexity of sensor and big data.
Building a Tiered Digital Storage Environment on User-Defined Metadata to Enable eResearch
- 1. Building a Tiered Digital Storage Environment Based on User-Defined Metadata to Enable eResearch David Fellinger Data Management Technologist iRODS Consortium October 24, 2019
- 2. iRODS: Data Management at Scale 2 Modern RAM (1965) 1K non-volatile Magnetic Core 1µs access Historical Reference https://en.wikipedia.org/wiki/Magnetic-core_memory
- 3. iRODS: Data Management at Scale 3 Modern Hard Disk Storage (1956) 5MB Random Access @3ms Historical Reference https://en.wikipedia.org/wiki/IBM_305_RAMAC#/media/File:IBM_350_RAMAC.jpg
- 4. iRODS: Data Management at Scale 4 Historical Reference https://www.computerhistory.org/revolution/early-computer-companies/5/100/1491 Modern Digital Storage Tape (1951) 1.4M per 1500 feet READ at 100 ips
- 5. Hierarchical Storage Management (HSM) • Early HPC users saw an immediate need for HSM – Technology dating to the mid 1960s enabled HPC users to apportion storage based on access time and cost. – CSIRO developed the Drum and Display (DAD) operating system as one of the first HSMs in the 1960s. • IBM and others began HSM development including Workstation Data Save (WDS) for AIX in the early 1980s. • All HSM software was initially based on moving data based on attributes; – File creation time and date – File name – Name extension – Access controls • All HSMs were focused on moving data between vertical tiers and not on horizontal data distribution tiers. iRODS: Data Management at Scale 5
- 6. iRODS is the Next Generation HSM and Data Manager • The Integrated Rule-Oriented Data System (iRODS) has been designed by the iRODS Consortium with 4 key functionalities; iRODS: Data Management at Scale 6 iRODS is: • Open Source • Distributed • Data Centric • Metadata Driven
- 7. Metadata Driven • iRODS moves data based User Defined Metadata. – Latitude, longitude, altitude – Anomalies in genomic sequences – Data collection points – Instrumentation details enabling the data collection – Specific relevance in a research area • The use of Rich Metadata enables: – Discovery – Data grouping based on content to enable analysis – Data movement to analytic platforms • iRODS is Data Centric and Metadata can be extracted from file headers or actual file content. – Metadata extraction is based on set rules to produce a collection – Data can be apportioned instantly based on metadata – Metadata can include citation instances and can change dynamically iRODS: Data Management at Scale 7
- 8. The Rise of Sensor Data and “Big Data” • It can be argued that sensor data has changed the paradigm of HPC. – Huge amounts of data must be collected – The data has the characteristics of, Volume, Variety, Velocity and Value – The data must be organized for analysis – In many instances the data must be moved to a file system close to the analytic element – An analytic process must be started only when the full data set is available • All steps must guarantee provenance of the collected data to assure Veracity. • Full automation includes moving the results to a data distribution file system. iRODS: Data Management at Scale 8 Arcot Rajasekar DataNet Federation Consortium
- 9. iRODS Automating, Gathering, and Organizing Data iRODS: Data Management at Scale 9
- 10. Migration to HPC and Policy Driven Analysis iRODS: Data Management at Scale 10 • iRODS ties to the machine scheduler to move data at the correct time for analysis. • Data with similar metadata charactistics are selected for the analysis process. • Data are moved to a parallel file system for processing. • iRODS moves the data but is not in the process path. • When the process is concluded with notification from the scheduler iRODS can purge the scratch file system to a data distribution file system. • iRODS can provide notifications that the reduced or analyzed is available.
- 11. Automated Management Through Synchronization to the Cloud iRODS: Data Management at Scale 11 • iRODS can migrate data to any filesystem maintaining location data in the catalog. • Data can be synchronized to the cloud or a federated partner. • Notifications can be provided at each step of the process and an audit report can be generated at any time.
- 12. Packaged Capabilities Allow Tracking and Managing Data Through Publication iRODS: Data Management at Scale 12 • iRODS provides eight packaged capabilities which can be configured and deployed to serve the needs of the data center. • Organizations can seamlessly address their immediate needs. • Additional capabilities can be added or reconfiguration can occur as the need arises. • A plugin architecture allows customization to address any data migration need.
- 13. Automation to Enable the Establishment of an Archive iRODS: Data Management at Scale 13 • iRODS provides eight packaged capabilities which can be configured and deployed to ser
- 14. Secure Federation Enables Geographically Protected Data Archives iRODS: Data Management at Scale 14
- 15. Deployment: CyVerse iRODS: Data Management at Scale 15 Diagram available from: https://www.cyverse.org/about accessed 25 September 2019
- 16. Deployment: EUDAT CDI iRODS: Data Management at Scale 16 Diagram available from: https://eudat.eu/eudat-cdi Accessed 26 September 2019
- 17. Conclusion • eResearch has evolved to accommodate sensor and other types of “big data”. • The use of user defined and extracted metadata improves the disposition of data at every level. • iRODS can enable complete workflow control, data lifecycle management, and present discoverable data sets with assured traceability and reproducibility. iRODS: Data Management at Scale 17
- 18. The iRODS Consortium (iRODS.org) The iRODS Consortium • Leads software development and support of iRODS • Hosts iRODS Events • Tiered membership model iRODS: Data Management at Scale 18
- 19. Questions? iRODS: Data Management at Scale Thank you! David Fellinger davef@renci.org iRODS.org 19