Phase II Overview of Information Literacy Assessment Project
- 1. A Standardized Approach to Assessment – Phase 2New Improvements Based on Student , Consultant & Library Faculty FeedbackShannon M. StaleyEducation & Web Services Librarian
- 2. In 2008 / 2009, We Began By:Developing and testing an assessment tool that could:Facilitate the development of pre and post achievement testsLink survey questions to learning outcomes and ACRL standardsDeliver real time reports and statistical analysis
- 3. In 2010, We Asked the Question:How can we improve the tool and the assessment process?Usability testing sessions with librarians who have not used the toolMore cognitive interviews with students to improve question developmentImproved statistical analysis Focus group with librarians who have used the tool
- 5. Usability Sessions: Changes you asked for Comment: I didn’t understand where I was when I entered the tool.
- 6. Usability Sessions: Changes you asked for Remedy: A clearly defined menu as the starting screen
- 7. Usability Sessions: Changes you asked for Comment: Make it easier to toggle from my personal account to all questions and possibly select and copy some for my account.
- 8. Usability Sessions: Changes you asked for Remedy: “View Other Librarians’ Questions” was changed to “View All Questions.”
- 9. It’s easier to add and remove questions.
- 10. The “My Questions” view and “All Questions” view are different colors for more distinction.
- 11. We have indicated which questions have been standardized.Usability Sessions: Changes you asked for Comment: I didn’t understand when the survey was actually created – also the link to the survey was on a site somewhere.
- 12. Usability Sessions: Changes you asked for Remedy: A message indicates the survey was created and the URL to the survey is provided so that it can be posted anywhere.
- 13. 2. More Cognitive Interviews
- 14. More Cognitive Interviews:Cognitive interviewing is a survey development technique that involves one-on-one interviews with a sample of the target population to assess the quality of survey questions.
- 15. During the cognitive interview process, the interviewer uses techniques to elicit feedback on survey questions.
- 16. We asked interviewees to read questions aloud and tell us what they thought the questions meant. Also, whether there was any terminology that was confusing.More Cognitive Interviews:Original pilot study involved only 5 one-on-one interviews with undergraduate students
- 17. Wanted to aim for 10 interviews prior to Fall 2010 when we planned to use the tool again in the classroomCognitive Interviews:Changes we madeQuestion:What are a few of the core databases for Education students? Changes: We changed “core databases” to “most important databases” after we learned that some interviewees were confused by the original term.
- 18. We changed “Education students” to “students in the College of Education” to be clearer in our meaning.Cognitive Interviews:Changes we madeQuestion:When you start a search, why is it best not to limit results to full text only? Changes: We added “articles” after “full text” in the question after we learned that some interviewees weren’t clear on the meaning of the phrase “full text.”
- 19. Cognitive Interviews:Changes we madeQuestion:In PsycInfo, how do I save a search result for the current search session? Changes: We made explicit that PsycInfo was a database, since many of the interviewees did not know what the term meant.Cognitive Interviews:Changes we madeQuestion:How can you tell you are reading a popular magazine? Changes: Interviewees thought “popular” meant desirable or heavily used material rather than non-scholarly so we changed the question to:
- 20. How can you tell you are reading a magazine instead of a scholarly source of information?3. Improved Statistical Analysis
- 21. Improved Statistical Analysis:Built in TTEST ReportsWe built in TTEST Reports through a link to an excel spreadsheetImproved Statistical Analysis:Built in TTEST Reports Snapshot of the an excel spreadsheet report with TTEST tabulationImproved Statistical Analysis:Advanced Correlation TemplatesKendall’s Bivariate Correlation Analysis
- 22. Asteriks by values in the matrix show statistically significant relationships between background factors.
- 23. This data set shows that students who received prior library instruction had higher pre-scores than those who had not. Also those who received prior library instruction spent more time using library resources than those who had not.4. Focus Group
- 24. Focus Group:Points for DiscussionAre we testing teaching effectiveness or student learning?
- 25. Do numbers offer the big picture of teaching effectiveness?
- 26. Need more research to develop a cut score or acceptable percentage of correct responses.
- 27. Need to gather support from other SJSU librarians and get campus endorsement
- 28. The data is helpful to librarians and faculty – it gives us some ballpark idea of whether students are learning or notWeb Site Tool Kit:URL:http://www.sjsu.edu/faculty/sstaley/sila/Provides complete overview of the entire assessment project from pilot study to present
- 29. Complete data collected from usability testing sessions
- 30. Complete data collected from focus groupQuestions?