Taking the Time: Using a Rubric to Analyze How Students Find, Document, and Integrate Sources

Editor's Notes

• #2: Introduce presentation topic Today we will talk about a rubric we designed to assess student skill levels in senior seminar capstone classes. Explain how it has given us a greater understanding within an organization regarding student skills. Positioned us to have more in-depth discussions with faculty regarding improving student skills. Before moving into how we used the rubric and our findings, we would like to talk about the development process. Development Literature review to locate the use of rubrics in library instruction Slides Show rubric site Screenshot Show bibliography Many rubrics available with various measurements. We already had an idea of the elements we wanted to address but looked at how other rubrics measured similar skills. Crafted new rubric using out initial ideas and insights garnered from the rebrics we evaluated. Present Rubric Describe What does it measure Utilize authoritative works within the discipline to fulfill research needs Citing sources in a bibliography using consistent formatting Citing sources in the text of the paper using in-text citations or footnotes in a consistent format Attributing sources within the text of the paper (Note: this was removed from the scoring sheet due to inconsistencies in interpretation) Integrating sources into the paper Why did we choose these five categories Care most about quality of sources and how much they were taking our advice How well were they leveraging sources to enhance their own authority How well were they using the heart of the work How well were they paraphrasing and quoting their sources Validating our rubric Evaluated a sample set of papers using rubric to ensure agreement on application Discussed disagreements and inconsistencies How we applied it Gathered 60 papers from senior seminars across different disciplines including two senior seminars where we were heavily embedded. Evaluated 66 percent. 3 point scale for each section Weak, acceptable, proficient Dropped “not acceptable” Most important aspect of our rubric and method is choosing three citations and investigating those to see how the student used them in their paper. Method Tried to choose citations used more than once Chose ones that were accessible through our resources Brisk ILL department Locate cited sections using parenthetical citations or footnotes Sometimes problematic due to incorrect/missing pages and/or author/in-text citation disagreement Findings We were surprised by the results. What we thought was a problem was not a problem. Use of authoritative works Most cited works were high quality We gained insight into other more pressing issues Problems with citation Missing page numbers in in-text citations/footnotes Inconsistent formatting Evaluation of source integration disclosed Instances of plagiarism Problems with incorrect paraphrasing and/or quoting Use of sections of a work that were not important to the author’s argument (i.e. filler) A lack of depth in student use of source material Heavy reliance on quotes and frequently only reference sources once in a paper. Areas where the author’s ideas were taken out of context and misquoted or misrepresented Differences and similarities across disciplines Scoring consistent across disciplines, reinforcing areas of success and concern In particular we were interested to know whether the papers that received mentoring had a higher quality of resources than in other senor seminars. Slightly better quality of resource selection and documentation for classes that received mentoring Students in all disciplines struggle with integrating sources into the paper. Conclusions and follow-up Use the scoring sheet developed for the analysis of papers to provide constructive, formal feedback on at least one draft of a paper submitted by student participating in library research mentor programs. For departments represented in the study, share results and discuss how this might change library research instruction efforts. Encourage faculty to designate a preferred citation style and upload resources created by the library to Moodle to support the preferred style. Analyze student papers in ENG 110 and upper-level courses within the major to identify when students become better at selecting quality sources. How we work with the senior seminar classes Each year there are roughly x senior seminars, we work with Y of them and in 2 senior seminars we have a very formal mentor program. Class visit versus mentoring Did a survey - Survey results showed that participants in the mentor program found it very helpful, but students in other classes were resistant to being required to meet with a library mentor.