Principles for proper data management and reuse--An RDA view
Download as PPTX, PDF2 likes599 views
The document discusses principles for proper data management and reuse from the perspective of the Research Data Alliance (RDA). It notes that RDA has over 2000 members with diverse opinions. There is an ongoing discussion around trends in data practices and principles that most members agree with, such as data needing to be findable, accessible, combinable and interpretable by others. The document outlines some results from RDA working groups, including a common data model using persistent identifiers, a data type registry, a generic application programming interface for persistent identifier records, and a set of best practice policies for typical data management and data processing tasks.
Principles for proper data management and reuse--An RDA view
- 1. Principles for proper Data Management and Re-Use ā an RDA view Peter Wittenburg Max Planck Society
- 2. 2 ļ§ does RDA have one view ā yes & no ļ§ RDA is basically a bottom-up organization driven by the many ācreativeā minds who want to change data practices ļ§ RDA has now about 2000 members ā so we have 2000 opinions? ļ§ we have an intensive discussion process since 2012 (ICRI Conference Copenhagen) and we can see that there are a number of trends and principles all or most seem to agree with ļ§ still RDA is a very young initiative and needs much attention and grease Clarification
- 3. 3 Why is this all relevant? ļ§ Naoyuki Tsunematsu (JST ): ā¢ Data exchange (and thus the need for proper data management) difficult to convey in Japanese Science ā¢ parallel trends observed for Japanese Science ā¢ not so often included in collaborations anymore ā¢ not so often represented in the top papers ā¢ enormous decrease in international ranking ā¢ serious worries about counterproductive encapsulation ā¢ this concern seems to be relevant for all of us
- 4. 4 Trends I ā Volume, Complexity from simple structures ... ... towards complex relationships
- 5. 5 Trends II - Anonymity direct exchange between known colleagues Domain of Repositories
- 6. 6 Trends III ā Re-Usage Domain of trusted Repositories ā¢ Data will be re-used in different contexts ā¢ Data needs to be findable, accessible, combinable and interpretable for others
- 7. 7 Data Practices I ā Survey ļ§ ~120 Interviews/Interactions ļ§ 2 Workshops with Leading Scientists (EU, US) ļ§ too much manual or via ad hoc scripts ļ§ too much in Legacy formats (no PID & MD) ļ§ there are lighthouse projects etc. but ... ļ§ DM and DP not efficient and too expensive (Biologist for 75% of his time data manager) ļ§ federating data incl. logical information much too expensive ļ§ hardly usage of automated workflows and lack of reproducibility
- 8. 8 Data Practices I ā Survey ļ§ ~120 Interviews/Interactions ļ§ 2 Workshops with Leading Scientists (EU, US) ļ§ too much manual or via ad hoc scripts ļ§ too much in Legacy formats (no PID & MD) ļ§ there are lighthouse projects etc. but ... ļ§ DM and DP not efficient and too expensive (Biologist for 75% of his time data manager) ļ§ federating data incl. logical information much too expensive ļ§ hardly usage of automated workflows and lack of reproducibility
- 9. 9 12 21 26 95 95 96 97 266 676 DIF DwC DC EML FGDC Open GIS ISO My Lab none Metadata standards Data Practices III - Metadata slide von Bill Michener, DataONE
- 10. 10 ļ§ lack of proper documentation, schemas, semantics, relations, etc. ļ§ directory structures, spreadsheets etc. are ad hoc creations and knowledge fades away ļ§ etc. Data Practices II ā Data Entropy
- 11. 11 Community Center Common Data Center Changes needed ā EUDAT and others many excellent projects are working on changes: ESFRI projects, DataNet projects, e- Infrastructures, national projects RDA needs to build on experiences and expertise
- 12. 12 RDA widely agreed I ā time to change ļ± management of data objects is widely type and discipline independent ļ± still every project defines its own strategies leading to huge stack of software that will not be maintainable
- 13. 13 RDA widely agreed II ātime to change what Value Added Services Data Sources Persistent Identifiers Persistent Reference Analysis Citation Apps Custom Clients Plug-Ins Resolution System Typing PID Local Storage Cloud Computed Data Sets RDBMS Files Digital Objects PID record attributes bit sequence (instance) metadata attributes points to instances describes properties describes properties & context point to each other
- 14. 14 RDA Results I: common data model ā¢ PIDs at the beginning of trust chain ā¢ have a worldwide, independent and robust PID system worldwide (DONA Handles ā DOIs are Handles)! ā¢ metadata are essential in anonymous data world taken from RDA WG Data Foundation & Terminology
- 15. 15 ļ§ result: a registry for data types ļ§ you get an unknown file, pull it on DTR and content is being visualized ļ§ extended MIME Type concept ļ§ no free lunch: someone needs to register and define type ļ§ code available begin 2015 ļ§ PIT Demo already working with DTR RDA Results II: Data Type Registry Federated Set of Type Registries Visualization Data Processing10100 11010 101ā¦. Data Set Dissemination 10100 11010 101ā¦. 10100 11010 101ā¦. Terms:ā¦ Rights Agree Visualization Processing Interpretation 3 Domain of Services 2 1 Human or Machine Consumers 4 ā¢ NIST is already working with communities on fargoing ideas
- 16. 16 ļ§ result: a generic API and a set of basic attributes ļ§ a PID Record is like a Passport (Number, Photo, Exp-Date, etc.) ļ§ if all PID Service-Provider agree on one API and talk the same language (registered terms) SW development will become easy ļ§ Test-Installation in operation together with DTR RDA Results III: PID Information Types LOC location, path CKSM checksum CKSM_T checksum type RoR owning repository MD path to MD
- 17. 17 ļ§ due to unforeseen circumstances need until P5 ļ§ Practical Policies = executable Workflow Statements ļ§ result at P5: a set of Best Practice PPs for a number of typical DM/DP tasks (Integrity Check, Replication, etc.) ļ§ currently a large collection of PPs, currently being evaluated ļ§ you could add your policies RDA Results IV: Practical Policies replication policy X replication policy Y integrity policy A integrity policy B integrity policy C md extraction policy l md extraction policy k etc. Policy Inventory Repository selection implementation execution data manager
- 18. 18 ļ§ need to place many RDA WGs & IGs on a common landscape since finally everything needs to fit together -> Data Fabric RDA ongoing: Data Fabric
- 19. 19 1973 Changes take long ... 1990 1993 TCP/IP Specification 1977 TCP/IP Stress-test WWW-Mosaic available worldwide adoption ļ§ many different suggestion & protocols ļ§ first no advantage for TCP/IP ļ§ at the beginning discussion about different email systems ļ§ at the beginning no interest from researchers and also industry (toi of some freaks) ļ§ required some top-down decisions to enforce unification 20 years!
- 20. 20 RDA is about global bridge building 20 RDA is about building the social and technical bridges that enable global open sharing of data. Researchers, scientists, data practitioners from around the world are invited to work together to achieve the vision Funders: NSF, EC, AU Gov, Japan, Brazil, DE?, UK?, ZA?, FI?, etc.
- 21. 21 Thanks for your attention. http://www.rd-alliance.org http://europe.rd-alliance.org
- 22. 22 ļ§ see Science 2.0 Initiative of EC ļ§ nr. of researchers increases enormously ļ§ there is a pressure in the direction of Grand Challenges and those topics relevant for societies ļ§ research is increasingly often data intensive ļ§ border-crossing research is a fact (countries, disciplines) ļ§ faster cycles (hypothesis ā analysis ā publications ā reviews) Trends IV: research is changing
- 23. 23 bottom-up process top-down process uptake to come RDA is about global bridge building
- 24. 24 EUDAT Services 24 EUDAT Box dropbox-like service easy sharing local synching Semantic Anno checking , referencing and annotating Dynamic Data immediate handling Generic Workflow automating data processing B2DROP B2NOTE
Editor's Notes
- #10: Suzie Scientists want to be able to use other scientistsā datasets, they are willin to share their own data and they feel it is appropriate to create new datasets from shared data.