The Right Metrics for Generation Open [Open Access Week 2014]

Editor's Notes

• #2: Welcome! My name is Stacy Konkiel, and I’m the director of marketing & research at Impactstory, a non-profit devoted to altmetrics. Before we get started, please “mute” your computer microphones for the presentation portion of this chat. After I conclude, you can unmute ‘em so we can talk freely. You’re here today because you know that the traditional means of sharing your impact–citation counts for journal articles–aren’t meeting your needs. What you need is a way to understand how your outputs, beyond the journal article, are being shared, saved, discussed, and more on the social web. Altmetrics can give you that information. Over the next half-hour, I’m going to share with you the key ways that you can get credit for practicing Open Science. I’ll describe what altmetrics are and the types of altmetrics you can expect to receive as someone who practices Open Science. We’ll also cover real life examples of scientists who used altmetrics to get grants and tenure--and how you can do the same. So, let’s get started!
• #3: This is Holly Bik. Holly is a researcher in metagenomics & environmental sequencing She’s a research fellow--the equivalent of an Assistant Professor--at the University of Birmingham. In the next few years, as she makes progress towards tenure, Holly will need to publish, get grants, and win awards. Those are some of the ways that she can benchmark her success, and prove it to others. Oftentimes, as she applies for those ever-important grants and awards, she’ll need to document how her research is making an impact on other scholars and the public.
• #4: Used to be she’d come up with a research idea, conduct her research with a handful of colleagues that she already knows, publish in a journal with a high impact factor, and wait for a year or two for citations to accumulate (to understand the reception to her work, and also get tenure). Dr. Bik would want to come across as productive as she possible can, since the more publications she has, the more favorably she is sometimes looked upon. publish as much as possible on a single study--a practice that some call “salami slicing”--in order to fatten up the publications section of her CV. Then she’d start all over again with a new or related idea. This system worked alright for many decades, but it had several disadvantages: it takes months or years for Holly’s paper to be published, delaying the dispersal of important findings it takes months or years for Holly’s paper to be cited after publication, making it hard for her to share with others the importance of her paper--something that can be damaging when she’s got to document that paper’s importance when applying for grants or tenure All of the other research products she created in the course of the study that led to this single paper--her data, the software she used to analyze the data, the slides and posters she presented at conferences when sharing her preliminary results, and so on--never see the light of day, meaning that other researchers cannot build upon her data, reuse her software, and so on, leading to a lot of duplicated efforts and waste. This system incentivizes “playing it safe”--publishing primarily in journals with high impact factors, and publishing only positive results that are “sexy,” so to speak. Incremental advances or work that’s proven negative--and can be used by other scholars to help rule inefficient methods out--also never see the light of day.
• #5: Luckily, things have changed a LOT in the last ten years. Nowadays, she’ll come up with an idea, start a literature review and save a heap of citations to Mendeley, find an author on Academia.edu, email them and end up becoming collaborators, Then she’ll start her study and begin to collect genomic data from her samples, post that data to a repository like Figshare or GenBank, and use research software and scripts that she created (and has shared on a website like GitHub or SourceForge) to analyze the data, She’ll share her preliminary results at a conference, and the talk will be recorded and posted to Youtube, talked about on Twitter by other scientists, as it’s happening, and she’ll post the slides to slideshare, after the meeting, so researchers who didnt’ attend have a chance to see her results. Once she gets feedback enough to write a preprint on the study, she posts it to the ArXiv.org repository to get even more feedback, often submitted via Twitter and blogs by other bioinformaticians. They help her hash out some problems with the study and shape it up so that it’s good enough to submit a formal journal article, which along with her preprint, accumulates citations. It’s also bookmarked on Mendeley, shared and discussed on Google+, and even makes it into a Wikipedia article related to the study subject. Holly then blogs about her study to help communicate it to the public and other researchers. All this time, while she’s gathering feedback, presenting, etc--she’s getting altmetrics for her work. And she’s seeing that her work is being used and discussed not only among other bioinformaticians, but also among the public and policymakers--two audiences that she couldn’t have known about if she were only tracking citations to her work. She then includes that information in her grant progress reports, so her funder--the National Science Foundation--can see the impacts she’s having and--hopefully--give her more money to continue her work in the coming years.
• #6: Here is what Holly’s story has to do with you. Holly is you. Early career researchers are changing the way science is done, making Open Science the “New Normal.” And the systems that are in place to recognize and reward researchers, awarding them jobs, giving them tenure, promoting them, and securing grants so they can conduct their research, are starting to change, too.
• #7: In the US, the National Science Foundation recently changed their award guidelines to recognize not just important publications, but also important products--making it possible for Holly to showcase research software she’s created that’s been very useful to others in her community, and so on. And the Wellcome Trust in the UK is using altmetrics to understand the reach and impacts of the research they fund. Also in the UK, faculty, funders and universities are exploring the role of metrics, including altmetrics, in the Research Excellence Framework exercise, which helps determine how funding is allocated to universities at a national level
• #8: We’re also beginning to hear from faculty, who are using altmetrics to document their impacts for the purposes of jobs, tenure, and promotion. We’ll get into specific examples later on in the presentation, but I want to make it clear, as we begin: universities like University of North Carolina and Indiana University are increasingly writing guidelines for evaluating so-called “digital scholarship”--that is, web native research products like datasets, software, and so on--into tenure & promotion guidelines. And researchers are beginning to use altmetrics to show just how widely their digital scholarship is being used, by whom, and how. So. It’s clear that altmetrics are an important part of documenting scholarly impact for cutting edge researchers like you, and that universities and funders are taking these supplementary metrics seriously. Let’s get down to brass tacks and talk about the basics of altmetrics and how they can be used.
• #9: Altmetrics measure the attention surrounding the work you’ve made available online, for all sorts of scholarly products beyond the journal article.
• #10: That’s because the science communication lifecycle has many components. It starts with an idea--and data. Then you goes on through the thinking and analysis phase. Eventually, stories are told in order to communicate your findings to others--those stories are often in the form of journal articles, but can also be in the form of blogposts, videos, and so on. And then conversation surrounding your work happens--sometimes in the literature, where other researchers might discuss your articles or data, citing your past work; sometimes it happens out in the open, on blogs and Twitter, or on journal club websites like Pubpeer or post-publication peer review sites like Publons. Eventually, those conversations may lead to future research. Point is, a lot is happening with your work.
• #11: So, how can you capture the impacts your work is having?
• #12: It used to be simplistic. We used to chart the path of scholarly communication using citations to journal articles. They served as an important breadcrumb, helping us to credit work that came before our own. And they were also an important indicitaor to us as authors, helping us to know if our articles were considered useful, important, or interesting to our colleagues.
• #13: In the past, when we were only measuring citations, We could easily manage our scholarly identities. It was like sipping from a small data stream. We’d put citations onto our CVs and into our annual reviews and were done with it. Impact backed up with data. Job done.
• #14: But now, administrators and funders often want to know: how’s your work making an impact in the quote-unquote real world? What effect is it having not only on other scholars, but also practictioners, the public, and policy makers. And its great that others are citing your work, but are they reading it? Do they consider it worthy of saving and revisiting, recommending to others, discussing in their spare time? As we showed from our example with Holly, there’s a lot of engagement that it’s now possible to measure, and you can measure it for research outputs other than journal articles.
• #15: Now it’s like drinking from a firehose--so much information we sometimes can’t make sense of it all!
• #16: So, How can we measure the impact of web native scholarship?
• #17: That’s where altmetrics come in. Altmetrics can help you measure and understand their impact for all their diverse research outputs, among many different stakeholder audiences. The term altmetrics refers specifically to social media metrics and other things that can be measured online, like pageviews and downloads, related to scholarship.
• #18: Altmetrics aren’t only for papers, but also all the other outputs that are created along every step of the research lifecycle.
• #19: And we can use altmetrics to understand the many varied flavors of impact of our work, among different stakeholder audiences. For example,
• #20: this 2012 study, “Altmetrics in the Wild”, introduced several example “flavors” by looking at correlations between different types of altmetrics for articles published in PLOS journals. They saw correlations between: How often something was Read & and then later Cited--you can see here in this heatmap that there are moderate correlations between how often something is viewed and downloaded and then later cited in the literature. Another flavor is the “popular hit,” which was discovered by looking at correlations between Facebook and Twitter shares And finally, there’s the Expert pick, seen when we examine the moderate correlations between post-publication expert peer review on Faculty of 1000 prime and later citations. Other researchers are using altmetrics to find other types of flavors. Lutz Bornmann recently found that research articles tagged on Faculty of 1000 as being “Good for teaching” are often more widely shared on non-scholarly social networks--a useful indicator of popularity among non-scientists. One can imagine finding other flavors for non-article outputs like datasets and software. This is an area that hasn’t been as widely studied yet, but we’re starting to explore. For eaxmple, what if your dataset is being used in the classroom to teach concepts to undergraduates?you’d see your dataset included in syllabi and shared in lesson plans? That’s a great contribution you’re making to training future scientists. And what if your software is being installed and reused by hundreds of other scientists in your domain, since it’s so cleverly created and does so much? Those installation statistics, and mentions of your software in journal articles, are two useful indicators of the scholarly impact your software is having. Now, the goal is not to compare flavors: one flavor is not objectively better than another. However, recognizing different types of contributions might help us appreciate scholarly products for the particular needs they meet.
• #21: And there are a lot of needs to be met. These impact categories were created by the Science Foundation of Ireland to help researchers and the public understand the many ways in which funded research can have an impact. Economic or commercial A new business sector or activity has been created. A business or sector has adopted a new or significantly improved technology or process Societal The awareness, attitudes, education and understanding of the public have been enhanced by engaging them with research of social or cultural significance. The work of an NGO has been improved or enhanced by the research. Public policy or services Policy debate has been stimulated or informed by research evidence. Policy decisions or changes to legislation, regulations or guidelines have been informed by research evidence. and so on. So, what types of metrics can you expect to find for your work, based on what type of work you’re sharing?
• #23: This is not comprehensive--it’s primer. Let’s look at metrics for papers first.
• #26: For other scholarly products, the type of metrics you receive are often related to the platform where it’s being shared. So, for software that’s often shared on sites like GitHub, here’s what you can expect
• #30: So, how do you go about finding these metrics?
• #31: You can use altmetrics aggregators to discover where your research is being talked about, cited, shared, and bookmarked online. There are three main independent aggregators out there today.
• #32: Altmetric.com is a london-based startup that is primarily used by publishers and institutions to track the impact of publications. But they also track anything else with a permanent ID like a DOI, such as data and figures shared on Figshare. Their strengths are that they do publications *really* well, and they’re also great at mining mainstream media mentions and mentions in public policy documents. So, they’re great for showing the impact of your publications. Their weaknesses are that they are focused mainly on publications, and they also don’t aggregate metrics for your work into a single profile--at least not using their browser bookmarklet, which is free for authors. If you want to track multiple publications, you have to use their bookmarklet to sign up for email updates for each paper.
• #33: PlumX is a US-based company that was recently acquired by EBSCO. They track metrics for a variety of research outputs, and are oriented towards meeting the needs of funders and institutions. Their strengths are that they aggregate the most diverse data sources for all types of research outputs, making them the altmetrics provider with the most complete coverage. A downside to PlumX is that it’s not possible for individuals to sign up for profiles at this time--they only sell institutional subscriptions.
• #34: I work for Impactstory.org. We’re a US and Canada-based non-profit startup, and we also track metrics for all your research outputs--Articles Datasets Figures Posters Slides Software & more. We offer researcher-based profiles, built specifically to meet the needs of scientists. We’re committed to open data, open source software, and operate our non-profit on the premise of radical transparency. Individual subscriptions are the only subscriptions you can get, which is a downside for some, who’d like to use our platform to track impacts for their institutions. But it’s also our biggest strength--with the laser-sharp focus on researchers, we’re able to meet the needs of individual scientists really well. That said, let’s finish this webinar by looking at individuals who have used altmetrics data from Impactstory and other platforms to advance their career.
• #36: Steven roberts studies the effects of environmental change on shellfish at a university in the united states Steven used altmetrics in his tenure dossier at the University of Washington. here’s how he did it Included this table in his Education and Outreach section of his dossier Also included metrics in the CV itself, to showcase the impacts of his scholarly work. for example the slideshare views of his slides for this invited presentation And the article views and supplemental data views for this paper
• #37: Ahmed’s a biologist who has made a significant contribution to the understanding of the origin of photosynthesis in eukaryotes. Ahmed used altmetrics in his tenure dossier at the American University in Cairo in 2013. Here’s how he did it: in the narrative section of his dossier, he included a section on the quote-unquote total impact of his work by showing a screencap of his most impactful publications from his Impactstory profile, alongside badges that showed the relative attention his work has received (highly discussed by scholars, highly viewed by the public, and so on). He included this alongside He also described a software product he created, JAligner, that is used often by others in his field. And in his description, he included a link to all the citations that JAligner has received on Google Scholar--over 160 citations! Ahmed actually asked a member of his review committee what they thought about the inclusion of altmetrics, and they said they were impressed!
• #38: Holly has developed a bioinformatics data visualization tool called Phinch, which was funded by the Alfred P Sloan foundation. When reporting back to Sloan on the success of her project, she included Figshare views for related posters and talks, Github statistics for the Phinch software, and other altmetrics related to the varied outputs that the project created over the last few years. Holly’s hopeful that these metrics, in addition to the traditional metrics she’s reported to Sloan, will make a great case for renewal funding, so they can continue their work on Phinch.
• #39: Funders and an increasing number of universities are accepting altmetrics as supplementary measures of impact nowadays. Whether or not you use them is up to you--if you think your situation warrants a more traditionally-minded approach, by all means, be traditional. But in situations where you need to make your diverse impacts clear, and when you need to make a case for non-traditional impacts, altmetrics can help.
• #40: Thanks very much! I’ll make a short plug here for using Impactstory when aggregating altmetrics. You can sign up for a free, 30-day trial at Impactstory to track the impacts of your work. And if you’re interested in subscribing, we’re offering a “2 years for the price of 1” deal. Just use coupon code “OAW_241” when signing up! Now, questions? Thoughts? Let’s talk about how altmetrics might be useful for you.