A Decade of Evaluating Europeana: Constructs, Contexts, Methods & Criteria
- 1. A Decade of Evaluating Europeana CONSTRUCTS, CONTEXTS, METHODS & CRITERIA Vivien Petras & Juliane Stiller, Thessaloniki, 19. September 2017, TPDL 2017 uropeana-5 by Europeana EU CC BY-SA 2.0
- 2. Agenda 1. Europeana 2. Approach 3. Results 2
- 3. Launch of the portal A Decade of Europeana 3 2011 2014 20172008
- 4. Europeana Ecosystem Visualisierung von Elco van Staveneren unter
- 5. Approach 5
- 6. Saracevic‘s Framework 6 Elements Description Construct What is evaluated? Describes the aspect, which is the focus of the evaluation, for example the metadata or the search functionality Context Which perspective is used for the evaluation? user- centered perspective, the interface perspective and the system-centered perspective? Criteria Which objective is evaluated? Measures How are the criteria evaluated? Methodology Which approach, instrument or tool is used for data collection and analysis for the evaluation? Saracevic, T. (2000). Digital library evaluation: Toward an evolution of concepts. Library Trends, 49(2), 350
- 7. Data and Information Extraction Existing lists of evaluations, Web of Science and Google Scholar 38 evaluations with Europeana as the object, 3 evaluations using Europeana data, and 14 meta-studies, which named Europeana as a use case Extracted information from 41 publications As close to source as possible Grouped in categories designed by Saracevic, e.g. Context Developed categories for other elements 7
- 8. Results 8
- 9. Contructs: What was Evaluated? 17 9 7 5 3 0 2 4 6 8 10 12 14 16 18 Europeana DL External Service Europeana component Europeana in comparison Algorithm 9
- 11. Contructs & Contexts 11 15 4 1 4 0 1 8 7 5 3 6 2 1 0 0 0 5 10 15 20 25 Europeana DL External Service Europeana component Europeana in comparison Algorithm User-centered System-centered Interface *studies can appear in more than one context
- 12. Criteria & Measures Often studies define their own criteria Usability and the effectiveness of the interaction design, user behavior and algorithm performance were used as objectives more often than others 12
- 13. Methods Method Description # Criteria-based Certain criteria were determined to assess a service or algorithm 16 Gold standard-based Use of a manually created gold standard to assess performance 9 Logfile analysis Uses an automatically created logfile of user interactions 8 Usability study Several methods to assess usability of a service, e.g. user studies, interviews, surveys 7 Impact study Expert assessment of the overall value of a service 2 13
- 14. Conclusions 14
- 15. Conclusions Studies with a focus on system perspective prevail Take on the institutional and societal perspective Studies on overall impact of Europeana are still rare: What are success criteria of CH systems? Europeana Impact Framework tries to find answers to this question: https://pro.europeana.eu/what-we-do/impact 15
- 16. Conclusions Learn from an Evaluation Archive Track improvements over time Standardize evaluations Document the changes 16
- 17. Questions? 17 QUESTIONS „The Question“ by Derrick Story (CC-BY-NC-ND 2.0)
- 18. References Saracevic, T. (2000). Digital library evaluation: Toward an evolution of concepts. Library Trends, 49(2), 350. www.europeana.eu 18
Editor's Notes
- #4: Animieren ...