Assessing a human mediated current awareness service
Download as PPTX, PDF0 likes659 views
1) The average editing time for creating a report was 15.5 minutes, with a minimum of 2.5 minutes and maximum of 53 minutes. 2) There was no correlation found between the issue size, editing time, and the success of a report, which is assumed to be mainly driven by topic and age. 3) Editors worked comfortably with the document presorting, with on average 77-82% of selected documents coming from the top N presorted documents. The relative search length, how deep editors inspected the document list, was on average 8% of the list, suggesting editors selected from the upper part of the presorted list.
![1. Introduction
• Current awareness in digital libraries
– To inform users / subscribers about new / relevant
acquisitions in their libraries [1].
• Current awareness services allow subscribers to keep up to
date with new additions in a certain area of research.
• Selection of relevant documents can be done (semi-
)automatically or manually.
• For this work we focus on the intellectual editing process
• Aim of this work:
How do editors work when creating a subject
specific report in Digital Libraries (DL)?
Slide 4 / 31](https://image.slidesharecdn.com/isifinal-150521101602-lva1-app6891/85/Assessing-a-human-mediated-current-awareness-service-4-320.jpg)
Assessing a human mediated current awareness service
- 1. Assessing a human mediated current awareness service International Symposium of Information Science (ISI 2015) Zadar, 2015-05-20 Zeljko Carevic1, Thomas Krichel2 and Philipp Mayr1 1firstname.lastname@gesis.org 2lastname@openlib.org
- 2. Outline 1. Introduction 2. RePEc and NEP 3. Results 3.1 Editing time 3.2 Indicators for report success 3.3 Editing effort 4. Conclusion and Outlook Slide 2 / 31
- 3. Motivation • Thomas Krichel, the founder of RePEc, visited GESIS – Cologne in Oct. 2014 • Sharing his Russian souvenir • ~100 GB of XML log files Slide 3 / 31
- 4. 1. Introduction • Current awareness in digital libraries – To inform users / subscribers about new / relevant acquisitions in their libraries [1]. • Current awareness services allow subscribers to keep up to date with new additions in a certain area of research. • Selection of relevant documents can be done (semi- )automatically or manually. • For this work we focus on the intellectual editing process • Aim of this work: How do editors work when creating a subject specific report in Digital Libraries (DL)? Slide 4 / 31
- 5. 2. Use case: RePEc • RePEc (Research Papers in Economics) is a DL for working papers in economics research. • Covers metadata for working papers and journal articles. • Usually document metadata contains links to full texts Slide 5 / 31
- 6. 2. RePEc statistics 0 200 400 600 800 1000 1200 1400 1600 1800 1996 1998 2000 2002 2004 2006 2008 2010 2012 2014 2016 Numberofdocuments Year Contr. Archives Documents Full text Documents Regist. Authors Abstract views (April 2015) ~1,700 1.77 mio 1.63 mio ~45,000 >2 mio Slide 6 / 31
- 7. 2. Current awareness service NEP • NEP (New Economics Papers) is a current awareness service for new additions in RePEc. • NEP covers subject specific reports from over 90 specific fields. – Business, Economic and Financial History – Public Economics – Social Norms and Social Capital • Issues are sent to subscribers via E-Mail, RSS and Twitter • Reports to new additions are generated by subject specific editors. • Relevant document selection is done manually by the editor! Slide 7 / 31
- 8. Nep-acc Nep-afr Nep-all • Contains all new RePEc docs • Created roughly on weekly base • Contains avg. 488 doc Selects Nep-upt Nep-ure Selects Selects Selects Sends issue Sends issue Sends issue Sends issue Manual selection of relevant documents is a time consuming task. Slide 8 / 31
- 9. ERNAD • ERNAD (Editing Reports on New Academic Documents) is a purposed built system • Re-rank nep-all for each editor based on the specific report topic • Looking at past issues of a report to produce a ranked nep-all • If presorting works well editors select highly ranked documents from nep-all Slide 9 / 31
- 10. ERNAD example for Nep-Africa (NEP-AFR) 1. Tax compliance.. 2. Mental accounting.. … 212. Ethnic ..in Africa 317. Sino-African relations: Nep-all unsorted Nep-all presorted Slide 10 / 31 1. Ethnic ..in Africa 2. Sino-African relations: … 50. Tax compliance.. 51. Mental accounting..
- 11. Editing stages Slide 11 / 31
- 12. Research questions • RQ 1: How long is the editing duration? • RQ 2: What influences the success of a report? – Editing duration – Issue size • RQ 3: How much effort is invested for selecting and sorting papers per issue? – Precision @ N – Relative search length Slide 12 / 31
- 13. RQ 1: Editing time How much time do editors invest to create a report? Slide 13 / 31
- 14. Pre-selection • Editing an issue can be interrupted • This would distort the results • Exclude interrupted issues by separating the edit duration in 3-minute chunks Slide 14 / 31
- 15. Pre-selection 0 1000 2000 3000 4000 5000 6000 7000 8000 9000 3 6 9 12 15 18 21 24 27 30 33 36 39 42 45 48 51 54 57 60 63 66 69 72 75 78 81 84 87 90 >90 Numberofissues 3-minute chunks Limit edit time < 90 min Slide 15 / 31
- 16. 0 10 20 30 40 50 60 nep-ets nep-gro nep-opm nep-pke nep-cba nep-hea nep-rm g nep-geo nep-hap nep-tid nep-dem nep-soc nep-cse nep-net nep-ifn nep-lab nep-ltv nep-for nep-law nep-m ig nep-cdm nep-m on nep-exp nep-neu nep-ino nep-m st nep-ore nep-fm k nep-ara nep-m kt Averageeditingtimeinminutes Report Avg. editing time RQ 1: Editing time Avg. 15.5 minutes. (sd = 10.1) Min. 2.5 minutes NEP- RES (Resource economics) Max. 53 minutes NEP-ETS (Economic time series) Slide 16 / 31
- 17. Summarize RQ 1 • Average editing time is comparable low with 15.5 minutes • Huge scattering between the reports: –Min. 2.5 minutes –Max. 53 minutes Slide 17 / 31
- 18. RQ 2: Influences to successful reports • Popularity of a report can be measured by the number of subscribers. • Huge scattering between number of subscribers per report – Max. 6859 NEP-HIS Business, Economic and Financial History – Min. 75 NEP-CIS Confederation of Independent States • Factors influencing reports success for example: topic, age of a report.. • Does the issue size or the editing time influence the report success? Slide 18 / 31
- 19. Editing time 0 1000 2000 3000 4000 5000 6000 7000 0 10 20 30 40 50 60 Numberofsubscribers Average editing time Avg. edit time Avg. number of subscribers Education 2198 sub. (avg. 836) Project, Program and Portfolio Management 43,5 min (avg. 15.5) Slide 19 / 31
- 20. Issue size 0 1000 2000 3000 4000 5000 6000 7000 0 10 20 30 40 50 60 Numberofsubscribers Average issue size Avg. issue size Avg. number of subscribers Sports issue size 2.5 (avg. 12.4) Demographic Economic issue size 21 (avg. 12.4) Slide 20 / 31
- 21. Summarize RQ 2 • There is no correlation between: – Issue size and number of subscribers – Editing time and number of subscribers • We assume that the success of a report is mainly driven by topic and age. Slide 21 / 31
- 22. RQ 3: Effort in selecting and sorting How much effort is invested in selecting and sorting relevant documents from nep-all? Two measures are used: Precision @N Relative search length Slide 22 / 31
- 23. Precision @ N • How many of the top n documents from pre-sorted nep-all are selected for the issue? • N set to: 5, 10, 15, 20 • We only consider issues where issue size > N • A document is relevant if its index position in nep-all is < N. Slide 23 / 31
- 24. Example: P@ 5 • M={(D1, 4), (D2, 1), (D3, 7), (D4, 3), (D5, 9)} • P@5 for issue I in report J = ⅗ • Editors vary between using pre-sorted and un-sorted nep-all. Therefore: – Only consider issues with pre-sort usage > 50 Slide 24 / 31
- 25. Results for P@N Avg. P@5 (82 rep) Avg. P@10 (64 rep) Avg. P@15(50rep) Avg. P@20 (31 rep) 0.77 0.80 0.80 0.82 • Max. found for nep-env (Environmental Economics) with P@5 = 0.99 • Min. found for nep-cba (Central Bank) with P@5 = 0.35 Slide 25 / 31
- 26. Summarize P@N • Editors work comfortably with the presorting in nep-all. • The number of papers per issue has no significant influence for the precision. Slide 26 / 31
- 27. Relative Search Length • We know how many of the top N document from nep-all selected. • To what depth do editors inspect nep-all? • Ratio between the highest index position (hin) of the last relevant document in nep- all and the length of nep-all Slide 27 / 31
- 28. Example RSL • Editor is given a nep-all containing 300 documents. • M={(D1, 4), (D2, 10), (D3, 7)} • RSL = 10/300 • We assume that the editor has inspected nep-all to document 10. Slide 28 / 31
- 29. Relative Search Length 0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 nep-mac nep-dem nep-cwa nep-eur nep-iuenep-cbe nep-afrnep-mic nep-bec nep-intnep-knm nep-com nep-regnep-ifnnep-cdm nep-tidnep-effnep-inonep-upt nep-edu nep-fornep-neu nep-cisnep-ltvnep-net nep-dev nep-ppm nep-spo AverageRSLperReport Report Avg. RSL NEP-MAC (Macroeconomics) RSL = 0.35 NEP-SPO (Sports and Economics) RSL = 0.01 Avg. RSL = 0.08 Slide 29 / 31
- 30. Summarize RSL • The relative search length is comparable low with 0.08 • Editors select papers from the very upper part of nep-all. Slide 30 / 31
- 31. Conclusion • Focused on observable system features – Editing time – Influences on report success – Effort in creating an issue • Summarize: The system supports the editor well in creating an issue • A complete view requires a more user-centred observation. • Future work: – Why and under what conditions is a document relevant? • NEP provides many opportunities for further research on data that is relatively easily available. Slide 31 / 31