Rubrics for DMPs

Editor's Notes

• #3: The Data Management plan As a Research Tool (DART) project was comprised of academic librarians from five universities in the United States. They received funding from the Institute of Museum and Library Services. They developed a tool to assess National Science Foundation data management plans. This facilitates administrators in supporting researchers in creating a good data management plan as well as using the content of plans to inform the research data support provided to researchers.
• #4: As an instrument created by researchers themselves, data management plans are a rich source of information about an institution’s researchers and their research data management (RDM) knowledge, capabilities, and needs. The DART project recognized that by careful evaluation of data management plans, they could determine information about four areas of research data management. 1 – They could determine how much the researchers who were writing the plans knew about their funder’s expectations. 2 – They could determine the extent to which their researchers and institutions had the capacity to meet funder data expectations. 3 – They could evaluate different research data management practices in their institution and compare and contrast between the groups in their study and also between different fields of research. 4 – They could determine where the gaps in data management knowledge were and then use this information to improve data management training. In order to do this, they needed a standardised way of evaluating data management plans.
• #5: A rubric is a descriptive scoring scheme. “A rubric is simply a scoring tool that identifies the various criteria relevant to an assignment or learning outcome, and then explicitly states the possible levels of achievement along a continuum.” http://teachingcommons.depaul.edu/Feedback_Grading/rubrics.html An analytic rubric provides detailed information about sub-components of the work through independent evaluation of various criteria. For our purposed, it was pretty clear that an analytic rubric would be necessary to provide sufficient detail from a Data Management Plan review. A shortcoming of using an analytic rubric is that there is a trade-off between rater consistency and level of diagnostic assessment. -> More granular assessment (one with more levels) leads to more variability in ratings (more room for error). This is a blank example of an analytic rubric.
• #6: An example of the rubric – NOT a complete final version (and isn’t complete for Section 1 due to space constraints). This is missing the example text for each assessment criterion A rubric is basically a descriptive scoring scheme. “A rubric is simply a scoring tool that identifies the various criteria relevant to an assignment or learning outcome, and then explicitly states the possible levels of achievement along a continuum.” http://teachingcommons.depaul.edu/Feedback_Grading/rubrics.html A holistic rubric generally provides an overall score for the thing being evaluated, while an analytics rubric provides more detailed information about sub-components of the work through independent evaluation of various criteria. For our purposed, it was pretty clear that an analytic rubric would be necessary to provide sufficient detail from a DMP review. When Amanda presented this concept, Mary realised that rubrics for evaluating data management plans would be useful in the UK context where institutions might be looking at plans for many different funders. It would allow us to become more consistent in our feedback. From this the UK DMP Rubrics project was born.
• #7: This slide is pretty self-explanatory. Amanda Whitmire of Stanford University Library gave an overview of the United States project at International Digital Curation Conference 2016 Major point is we’re still in the middle of this (but reasonably well on-target). Each group was given a skeleton structure for a rubric, similar to the blank outline in slide 4.
• #8: Rubric outline document was prepared in Google sheets. Supporting information was in Google docs. Things to consider included: 1 - whether we should ask funders to ‘approve’ developed rubrics, or just give them sight of the rubrics. We didn’t want to lose the community ownership of them, but we did want some level of buy-in from the funders. 2 – how we would go about keeping the rubrics up to date 3 – who would take responsibility for each rubric
• #9: 1, 2 and 3 are simplified versions of the instructions given to each working group. 4 and 5 have evolved. This process has been iterative. For some rubrics, there have been multiple revisions and comments, for others, there has been relatively little revision from the initial draft. 6 and 7 are mostly being done by Mary, but some other participants who have funder contacts are also helping.
• #10: As an update to this, we’ve had funder comments back from BBSRC. These will be incorporated in the next week or so and the revised rubric will be available to the community by Christmas (I hope).
• #11: The third point here is important. Some funders ask for a lot of info, but don’t provide much space in JeS. We hope to be able to demonstrate this, using the rubric to demonstrate what a comprehensive answer would look like. Alternatively, if would be interesting if they could provide what they consider to be a good example of a DMP that meets their needs, and for us to evaluate it using the rubric.
• #12: This is what I would like notes and idea on. There might be lots of ideas I haven’t thought of so far. The second point here is basically what the DART project did. However, instead of pulling 500 DMPs from their records and doing it as a study, we could do it as we go along and amass the data over time. Other institutions (eg LSHTM) do store a DMP for each project, so could do it as a study.