d:swarm - A Library Data Management Platform Based on a Linked Open Data Approach
Download as PPTX, PDF6 likes5,710 views
The document discusses the development of d:swarm, a library data management platform based on linked open data, aimed at addressing data heterogeneity and integration challenges in libraries. It emphasizes the importance of removing data silos, adopting open formats, and ensuring data quality through collaboration between librarians and domain experts. The document also outlines the functionalities of the d:swarm tool, including ETL processes and the use of property graphs for data management.
d:swarm - A Library Data Management Platform Based on a Linked Open Data Approach
- 1. A Library Data Management Platform Based on Linked Open Data SLUB Dresden slub-dresden.de CC BY-SA 4.0 Avantgarde Labs Robert Glaß 25 November, 2014 Jens Mittelbach | Robert Glaß
- 2. A Library Data Management Platform Based on Linked Open Data Back in Those Days The Age of Discovery Library Data Management Qualify, Link and Free Your Data: D:SWARM Live Demo SLUB Dresden slub-dresden.de CC BY-SA 4.0 Avantgarde Labs Robert Glaß D:SWARM 25 November 2014 | Page 2 Dr. Jens Mittelbach
- 3. Data Heterogeneity Multiple individual data silos • ILS, document repositories, databases, … Data saved in heterogeneous formats • MAB, MARC21, … Each data silo gets processed individually • Multiple admin interfaces • Multiple search interfaces • Data unrelated to one another Comprehensive view of resources almost impossible (for users and librarians) SLUB Dresden slub-dresden.de CC BY-SA 4.0 Avantgarde Labs Robert Glaß Back in Those Days … 09 December 2014 | Page 3 Dr. Jens Mittelbach
- 4. Data Normalization More comprehensive view of resources for users, but no real discovery/exploration Data gets normalized into one storage but not integrated Data available in record-oriented structures • External data (e.g. GND) has to be squeezed in the record • Metadata records are independent of each other • No explicit semantic quality of data SLUB Dresden slub-dresden.de CC BY-SA 4.0 Avantgarde Labs Robert Glaß The Age of “Discovery” 09 December 2014 | Page 4 Dr. Jens Mittelbach
- 5. Library Data Management What Libraries Actually Need Get rid of data silos • Open formats for exchange Lossless data integration instead of reductive normalization Data integration with entity level granularity • Get rid of pre-compiled data records Focus on linking entities/objects: • Graph structures creating the knowledge graph Stick to quality policy of libraries • Versioning and provenance of data SLUB Dresden slub-dresden.de CC BY-SA 4.0 Avantgarde Labs Robert Glaß 09 December 2014 | Page 5 Dr. Jens Mittelbach Library Data
- 6. Library Data Management What Should Library Data Actually Look Like? SLUB Dresden slub-dresden.de CC BY-SA 4.0 Avantgarde Labs Robert Glaß 09 December 2014 | Page 6 Dr. Jens Mittelbach
- 7. Library Data Management Whose Job Is Library Data Integration? Data integration should be done by domain experts • Librarians, not IT staff (IT always understaffed) • Programming skills should not be a requirement • Good user experience is a prerequisite for adoption Example driven modelling approach Value created in the community should be reusable SLUB Dresden slub-dresden.de CC BY-SA 4.0 Avantgarde Labs Robert Glaß 09 December 2014 | Page 7 Dr. Jens Mittelbach
- 8. Library Data Management What Tools Do We Need? Our Approach: An Open Source Data Management Platform SLUB Dresden slub-dresden.de CC BY-SA 4.0 Avantgarde Labs Robert Glaß 09 December 2014 | Page 8 Dr. Jens Mittelbach
- 9. Library Data Management How Can Data Integration Be Done? SLUB Dresden slub-dresden.de CC BY-SA 4.0 Avantgarde Labs Robert Glaß 09 December 2014 | Page 9 Dr. Jens Mittelbach
- 10. Qualify, Link and Free Your Data: D:SWARM Who’s behind this Project? Collaborative development team of SLUB Dresden and Avantgarde Labs GmbH Started work in June 2013 Funded from the European Regional Development Fund (ERDF) SLUB Dresden slub-dresden.de CC BY-SA 4.0 Avantgarde Labs Robert Glaß 09 December 2014 | Page 10 Dr. Jens Mittelbach
- 11. Qualify, Link and Free Your Data: D:SWARM Our Challenge: Existing Data Formats: MAB, MARC • „selection of keywords“ • Relevant MAB fields are 902x, 907x, 912x, 917x, 922x. • These fields have subfields a, b, c, … coded with further information (type of keyword, person, time, place, concept...) • From field 902x to field 922x we have to check • If in subfield "a" there is one of these strings (800|801|820|830|845|850|860|870|880)? • If so, is there one of these strings (c|g|k|p|s|t|z) in subfield "b“? • If so, the value in subfield "c“ qualifies as a keyword • Keyword needs to be trimmed (which is the easiest part) SLUB Dresden slub-dresden.de CC BY-SA 4.0 Avantgarde Labs Robert Glaß 09 December 2014 | Page 11 Dr. Jens Mittelbach
- 12. Qualify, Link and Free Your Data: D:SWARM Our Challenge: Existing Tools: Talend SLUB Dresden slub-dresden.de CC BY-SA 4.0 Avantgarde Labs Robert Glaß 09 December 2014 | Page 12 Dr. Jens Mittelbach
- 13. Qualify, Link and Free Your Data: D:SWARM Our Challenge: Existing Tools: Open Refine SLUB Dresden slub-dresden.de CC BY-SA 4.0 Avantgarde Labs Robert Glaß 09 December 2014 | Page 13 Dr. Jens Mittelbach
- 14. Qualify, Link and Free Your Data: D:SWARM What Is D:SWARM? Graphical web based ETL modelling tool that serves to: • import data from heterogeneous sources with different formats • map input to output schemata and design transformation workflows • load transformed data into property graph database With additional functionalities: • Exporting of data models as RDF • Sharing mappings and transformation workflows SLUB Dresden slub-dresden.de CC BY-SA 4.0 Avantgarde Labs Robert Glaß 09 December 2014 | Page 14 Dr. Jens Mittelbach
- 15. Qualify, Link and Free Your Data: D:SWARM How Does D:SWARM Work? Modelling GUI and job repository Execution environment • Operational data from heterogeneous data sources (ILS, OAI-PMH, CSV …) get processed according to the transformation logics defined in modelling GUI Admin centre • Scheduling & execution planning • Monitoring of system (data ingest, processing, errors) SLUB Dresden slub-dresden.de CC BY-SA 4.0 Avantgarde Labs Robert Glaß 09 December 2014 | Page 15 Dr. Jens Mittelbach
- 16. Qualify, Link and Free Your Data: D:SWARM Why a Property Graph? Node (S) – Edge (P) – Node (O) Extension of RDF data model - each element can be endowed with additional information (key : value) • Version number • Provenance information • Type information SLUB Dresden slub-dresden.de CC BY-SA 4.0 Avantgarde Labs Robert Glaß 09 December 2014 | Page 16 Dr. Jens Mittelbach
- 17. Qualify, Link and Free Your Data: D:SWARM Intermediate Results as of November 2014 Modelling GUI in 2nd version • Available file importer: XML, CSV, MABXML • Simple schema editor & graphic schema mapper • Transformation workflow designer & filter (Metafacture) Execution of mappings and transformations in modelling GUI Persistence in graph database (Neo4J) Exporter: Turtle, N-Quads, N3, … Publication under Open Source licence (Apache 2): https://github.com/dswarm SLUB Dresden slub-dresden.de CC BY-SA 4.0 Avantgarde Labs Robert Glaß 9 December 2014 | Page 17 Dr. Jens Mittelbach
- 18. Qualify, Link and Free Your Data: D:SWARM Live Demo SLUB Dresden slub-dresden.de CC BY-SA 4.0 Avantgarde Labs Robert Glaß 09 December 2014 | Page 18 Dr. Jens Mittelbach http://demo.dswarm.org
- 19. Qualify, Link and Free Your Data: D:SWARM Our Next Steps Provision of URI templates for resource matching and linking Scalable execution engine for production mode Extension of transformation function set Extension of importers Implementation of an administration centre Deduplication and FRBRization Integration of SLUBsemantics Enrichtment Service Implementation of sharing features SLUB Dresden slub-dresden.de CC BY-SA 4.0 Avantgarde Labs Robert Glaß 09 December 2014 | Page 19 Dr. Jens Mittelbach
- 20. Qualify, Link and Free Your Data: D:SWARM Your Next Steps Follow us on twitter.com/dswarm or www.dswarm.org or github.com/dswarm Try it out and get in contact with us • http://demo.dswarm.org • https://github.com/dswarm/dswarm-documentation/wiki • team@dswarm.org Help us prioritize our backlog • https://jira.slub-dresden.de/ Fork us on github.com/dswarm SLUB Dresden slub-dresden.de CC BY-SA 4.0 Avantgarde Labs Robert Glaß 09 December 2014 | Page 20 Dr. Jens Mittelbach
Editor's Notes
- #3: Where we come from, what we have, what we want to have How we can achieve it with D:SWARM Live presentation of D:SWARM
- #5: Data normalization poses quite a number of problems to librarians, admins and users normalisation: at the cost of information richness Deduplication questionable Reliable enrichment only for parts of data Linkage (especially with external resources) is technically not possible (at this stage) Metadata records are independent of each other (sit side by side in xml silos)
- #6: Data integration instead of mere normalization: Harvesting (external) data from a variety of sources and integrate all available information into a knowledge structures Versioning and provenance: this is related to what Markus Krötzsch said in his talk about statements according to the Wikidata data model
- #8: Example driven modelling approach: users can directly observe the concrete results of their abstract modelling work
- #12: Robert
- #18: http://hub.culturegraph.org/resources/static/mab2.pdf