Stephanie Faulkner - Using Metrics Beyond Citations to Demonstrate the Impact of SciELO Books

Editor's Notes

• #3: Our Metrics Manifesto Need to use different metrics and common sense Decisions should be based on both quantitative and qualitative input Should always use at least two metrics (more than one way to ‘excellence’) The methodologies should be open, transparent, valid and replicable
• #4: Modern research metrics give a more comprehensive and holistic view of impact. These new metrics are more timely than citation metrics and can keep pace with new formats faster than the entrenched, legacy practices. They are now available to provide fresh insights into how research is interacted with, shared, commented on, promoted, and more. So we define altmetrics as ALL METRICS. And PlumX is the only metrics provider that includes usage.
• #5: We developed 5 categories of metrics of how to represent the data exhaust that we are collecting. Usage, Captures, mentions, Social Media and Citations. I will give you a brief overview of each category.
• #6: Usage is very important – knowing how many times a particular article or artifact is downloaded, viewed or played (if it’s a video). One of the usage metrics we have for books is how many libraries hold a book.
• #7: Captures goes beyond viewing the artifact. Someone is saving it or bookmarking it for later or adding it to Mendeley of CiteULike. This indicates that the person wants to come back to this piece of research. Several studies have shown that captures are really good indicator of future citations. People save and store work that they will come back to when they are ready to write their paper or create their research output.
• #8: Mentions show engagement – someone is commenting, blogging or reviewing research or even referencing it in a Wikipedia article. It’s where we say the stories are hidden.
• #9: Many people think of altmetrics synonymously with social media. We think social media is a great way of measuring how work is promoted – by the researcher or the publisher.
• #10: Citations are a part of the overall impact assessment and are important to consider alongside the other metric types we capture. In addition to traditional citation indexes, PlumX metrics also include clinical, patent and policy citations which are a nice measure of impact in society.
• #11: The Plum Print changes dynamically based on each artifact’s metrics. It is like a fingerprint – unique to the specific research output. You can see the color-coding and easily identify which of the 5 categories of metrics has the most or least impact. We’re working towards getting the Plum Print on the Scielo Livros platform.
• #12: We track 67 different types of output in our database of over 70 million artifacts and 9.5 billion interactions.
• #13: Currently we provide metrics from over 50 different types of sources, and these are just a few examples of what we are measuring. Some metric sources update in near-real time, others daily or weekly.
• #14: How do we track the different types of output being generated? We can use an array of unique identifiers from ISBN, to ORCID ID, Slideshare Slideshow ID, of course a DOI. Even a Repository handle URI can be used.
• #15: Our technology starts with that single identifier and goes and finds additional identifiers related to the research. It could be an ISBN, publisher URL or DOI. In this example, we started with a single repository URL. From that one URL, we found multiple URLs and other identifiers. In the end we are tracking metrics on 3 identifiers and over 20 URLs. We take all of those identifiers and look for them in our over 50 metrics sources.
• #16: We know that measuring the impact of books is challenging as books and book chapters are underserved by traditional citation analysis. PlumX includes inventive new measures for books, including how many libraries hold a book, ebook downloads, reviews from Amazon and GoodReads, Wikipedia references, and more.
• #17: In 2016 a study was conducted by several bibliometricians who analyzed metrics on over 70,000 books. They compared traditional and non-traditional metrics (using PlumX metrics) and grouped books into subject areas. The authors acknowledge other research on books that states in general, books are not cited as often as articles and in in many disciplines books become obsolete much faster than articles. Abrizah, A., & Thelwall, M. (2014). Can the impact of non-Western academic books be measured? An investigation of Google Books and Google Scholar for Malaysia. Journal of the Association for Information Science and Technology, 65(12), 2498-2508. doi:10.1002/asi.23145
• #18: In the study, it was interesting to see which subject areas performed better in specific metric categories that other subject areas. This image illustrates which book subject areas tend to receive the most social media attention. Notice the subject areas with much less social media impact, like Education, Engineering, Ecology and Religion.
• #19: The analysis also revealed which book subject areas had the most views and downloads (usage). Notice how the subject areas with lesser social media impact, have better usage. I would imagine with a stronger social media presence, these same areas would have even higher usage.
• #20: Finally, a look at subject areas with the most book reviews. Medicine and Business, which had high social media impact, are not even listed here.
• #21: Let’s take a look at some book examples on the SciELO platform, starting with this publication with a good number of reviews. From any PlumX artifact page (an artifact is what we call any piece of research output), we not only display the number of reviews,
• #22: we link to the reviews and ratings on the platform (Goodreads or Amazon). This allows you to see exactly what people are writing.
• #23: In this example, you can see this book has 9 Wikipedia references. PlumX currently covers 4 Wikipedia editions – English, Spanish, Portuguese and Swedish. Each language has unique content – the references are not just translations. We link to the full Wikipedia articles where you can see the reference to the artifact. More languages are coming soon. It’s interesting to see the different types of Wikipedia articles that references this single book.
• #24: When I look at usage for this particular publisher, this book has high usage compared to other books on the SciELO platform. You can see that for all of our metric categories, we link back to as many sources as possible. Some of our sources (like EBSCO for example), only provide us with a count. This book has high abstract views on the EBSCOHost platform.
• #25: We have loaded all of the SciELO book identifiers into a PlumX Dashboard which you will have access to. A PlumX Dashboard allows you view metrics in a variety of ways and uncover insights about the books. The SciELO books dashboard is organized by publisher. Click on the publisher name to see the list of books and associated metrics. Throughout the dashboard you always have the option to export the metrics and perform your own analyses or use our built in analytic reports. A PlumX Dashboard gives you lots of ways to drill into or find information with several ways to filter – by specific metric type, by artifact type or by publication year range. Let’s take a closer look…
• #26: Each column can quickly be sorted to see which books have the highest impact . Clicking on the “plus” symbol below the metric category displays the metrics in more detail. After you sort the column, click on the title of any book to view all of the metrics.
• #27: View the source for each metric by clicking on the “down arrow” to expand. We display news and blog mentions, original tweets, Wikipedia references and other details if we have them.
• #28: When you click on the Analytics button from the PlumX Dashboard, you will find several prepared reports. The first report - artifacts by publication year -- shows you the last 10 years of metrics in each of our 5 categories so you can identify trends over time. The blue line represents the number of books in the year. The colored bar graphs represent each of our metric categories. Usage is in a separate graph from citations, captures, social media and mentions because usage tends to be a much larger number compared to the other metric categories. Mouse over a particular year to see the detailed metrics category counts. Look at the high social media metrics in 2009. Click on any bar graph to go to the filter view and sort by social media to see why this category is so high in this year.
• #29: You can see that same book from the previous example published in 2009 by Editora FIOCRUZ had a huge amount of social media compared to all the other books by this publisher in the same year. Expand the social media column, or click on the title to see the social media metric breakdown.
• #30: Another report in the analytics area is the Sunburst. The sunburst provides a nice visualization to find the book with the most overall impact by a particular publisher. Work from the inside ring out to the third ring and click on the largest section to see the book details. You can click on the title in the pop-up window to go to the artifact page.
• #31: And don’t forget that you can always export all of the detailed metrics as a spreadsheet by clicking the Export Data button on the dashboard. This will allow you to perform your own analyses, selecting the metrics that you choose to highlight.
• #32: To summarize: we track 67 types of research output from over 50 sources, grouping them into five main metric categories. We track not only published versions of research, but preprint, A&I and open access and currently have metrics for over 70 million artifacts.
• #33: We look forward to your feedback and questions