How to Improve Your Library Instruction: Assessment in Five Minutes (February 2017)

Editor's Notes

• #2: All images are creative commons, cited in the bottom right corner.
• #3: Choose your questions and model based on the outcomes you need to meet. Everyone has been watching the new Information Literacy framework develop. I agree with Megan Oakleaf that the framework doesn’t overthrow the importance of learning outcomes, though it might shift the way you were writing them if you were already doing them. We will incorporate the framework’s tenets today and talk about how you can assess the threshold concepts that they’re comprised of.
• #4: Example: terminology—article, journal, database This type of assessment is not an end in itself—it supports change and best practice in teaching.
• #5: Did they take away what you thought they were going to take away? If not, why? Threshold concepts: “core tenets in a particular discipline that are transformative, irreversible, integrative, bounded, and potentially troublesome” MEYER and LAND They tend to be highly constructive
• #6: I don’t care what scenario you did it in. An objective states what we aim to do. Outcome describes what the student will be able to do at the end.
• #7: In many cases, these words are used interchangeably. Some people are moving in favor of Outcomes over objectives, because the term objective has become muddled over the years, so that now, objectives are sometimes written as teacher-centric instead of user-centric. That means that the language is geared toward what I will do (as the teacher), rather than what you will be able to do (at the end of the class session).
• #8: Our first step in good, quick assessment is to write learning outcomes. The assessment questions which you choose will, in general relate specifically to those learning outcomes. They will also provide a roadmap for you, as the instructor, in terms of what you must cover. Whatever assessment method you choose later will tie back to these. A good learning outcome will state what the student will be able to do, and WHY. It will use Bloom’s taxonomy verbs of various cognitive levels. http://valenciacc.edu/facultydevelopment/tla/Candidate/documents/ExamplesandDefinitionsofLOs.pdf
• #10: First step: consider the top three or four class priorities. Write them down. If you are going to do some basic things then you can have more, like four or five. If you want to start mixing in larger concepts—things that are likely to trip them up and require some deep thought to understand--then you may find that you can’t fit in as many—but also, you may find that the way you teach is transformed, and that those more specific pieces aren’t necessary. (I’m still in my own liminal space with transforming to the thresholds, and a lot of others are, so if you feel nervous, you’re not alone.) Think more broadly than “they will know which ten clicks will lead them to a perfectly refined search. Rather, help them understand information types and why they care about them in the first place. Don’t be afraid to focus on “Big ideas and enduring understandings” Do they understand why they would use a scholarly article search versus a Google search? Do they understand why the scholarly conversation is critical? Can they identify a scholarly article and its parts? Learning outcomes should stem from a needs assessment. The needs assessment will determine the gap between the students’ existing condition and a desired condition—that condition is usually an increase in knowledge, skills, or attitudes. Remember that these are just a pithy starting place for you. One outcome may take almost the full hour to unpack.
• #12: Future tense—timeframe Identifies learner (some people say “students”, some like we)
• #13: Benjamin Bloom, an American educational psychologist, helped to write these in 1956. Domains of learning. Cognitive domain. I want to go through these in depth a little, because they will directly inform your assessment questions. So, as you choose verbs, consider them not just as objectives, but as assessment questions for your post test.
• #14: “Knowledge is the act of remembering previously learned material. It’s the lowest level of learning outcomes in the cognitive domain. “
• #15: “Comprehension is the ability to grasp the meaning of material. Students can demonstration comprehension through interpretation or summarization of material. These learning outcomes are a step beyond just remembering. They represent the lowest level of understanding.”
• #16: “Application refers to the ability to use learned material in new and concrete situations.” We do this a lot in one shots.
• #17: “Analysis is the ability to break down material so that its organizational structure can be understood. Differentiation, comparing and contrasting, identification. At this point you may notice that some of these verbs are on more than one list.
• #18: “Synthesis is the ability to put parts together to form a new whole.” It can be hard to get to this in an hour. Our students often do this at the course level—they have to write a paper, a speech, a bibliography, a literature review. Synthesis is the ultimate goal, in many cases.
• #19: “Evaluation is the ability to judge the value of material in a particular scenario. The judgments are to be based on definite criteria. Learning outcomes in this area are highest in the cognitive hierarchy because they contain elements of all the other categories, plus value judgments.
• #20: This should include context as often as possible without being too wordy or jargon-filled. So, for example, in this last I’ve said “advanced techniques” rather than saying boolean logic, subject header searches. We’ll get to those things, but not in our outcomes. You don’t have to have the context, but I like it. If you don’t include it in your written outcomes, still think about it and be able to articulate the context to the students.
• #21: Stem and outcome Demonstrable/assessable Bloom’s cognitive domain—high level No jargon Often INCREMENTAL—step-based. They fit together to make a whole. They also, if you show them, provide a sort of roadmap for the students. Giving them a meaningful outline of the topics you’re going to cover is shown to help students engage with the class more thoroughly. Let me again address the threshold concept piece of this. I hope it’s clear from each of these how they can be jumping off points for in-depth and transformative conversations, rather than point and click sessions. You can use them as a basis for teaching them the WHYs and WHENs of applying these skills, rather than just the HOWs.
• #22: Learner not identified—teacher focus Not assessable Poor verb No verb Jargon, too specific HUGE in scope
• #24: Again, I don’t care about the scenario in which you did this. IF you did it at some point, it counts—with other librarians, in a class, whatever. You maybe did it verbally, or with a poll, or with a quiz, or with an essay, or even with analysis of papers.
• #25: Just read through, no elaboration We’re going to talk about each and which might be best for you, given your setting.
• #26: Polls are good for feedback throughout the session--this kind of assessment is called formative assessment OR just at the end, for summative assessment. Formative assessment is anything you use it in the moment to immediately determine where the students are in their understanding. This type of assessment helps you as the instructor to identify material which needs to be clarified or covered in a different way—or who might need more time to think about the concepts you’re exploring. I’m going to talk about the technical way to do these, but I also often just ask these kinds of questions verbally and skip over the techy part, especially if they are more diagnostic in nature—ie what skills do you already have. Polls are great if you have information that builds and is cumulative. So, you can get them to answer a question or two before you move on to the next topic. They’re also a fun active learning exercise and they give you immediate feedback. The questions have to be simpler, with no short answer or essay, but you can have more of them. You can set them up to be embedded on a site/LibGuide (if your students have computers), or use free tools to set them up as text messages. If you have a clicker system, you can use them as well. Not too expensive, but they do cost $$$. (1000 to 2000... Other?) You can take those polls and embed them into PowerPoints, and then the answers will show up there (PollEverywhere). They can also just raise their hands, of course! You can have students work together to submit a group answer, or you can give each individual a clicker. Drawback—can be distracting, can have technical problems, can take up a lot of class time.
• #27: Regardless of what method you choose to deliver the content (phone, embedded poll, clicker) you can have different question types. Multiple choice, yes/no. These probably seem basic, and that’s because they are, but they are often the basis for everything you’re trying to share with your class. If they have a shaky foundation, it means you can’t build a house on it and your efforts will be a failure, so these aren’t always exciting, but they’re critical. Substantive or friendly—can also help you to set the stage for what you’ll cover in the beginning. I know these aren’t hifalutin, but it’s hard to build on a shaky foundation. These can be your foundation questions. These can be great, because they are anonymous in the moment. So, if someone feels shy about raising a hand to say s/he is confused, this can help them to do it without drawing attention to themselves.
• #31: questionnaires tend to be my weapon on choice, especially when I was first starting with quick assessment and trying to determine where my basic disconnects with the students were. You can position these however you like in your class time. I used to deliver them about ten minutes from the end of class, give the students five minutes, and then use the last five minutes to go over the answers quickly and take one last stab at fixing any confusion issues that arise. Lately, I’ve been moving to a different model where I have them answer the questions right when they come in, then we discuss what they wrote as a group. I have found that this approach has a very different feel. It’s a great way both for you to get a sense of where the students are from the start, and to help them apply your teaching to something they have written in front of them. You can also do a pre and post questionnaire. With both quizzes and polls, you can easily convert your learning outcomes to question, if they’re carefully drafted. So, “You will know how to construct a search statement becomes “construct a search statement.” Not more than five “meat” questions, maybe a couple of close out questions that are a bit more free-form.
• #32: This is a new one I’ve been working on and I want to tell you a bit about it. I tweak it every time, and if you adopt it you probably will too. The intent of this first question is to learn what they’re anxious about and what I should focus on. The second one is to hear their thoughts on why the library is important. Often they come up with abstractions like “my teacher says I have to” and that can actually be a fabulous jumping off point for a discussion about source credibility. The third one tends to go much the same—they say things like “I click the peer-reviewed button.” When covering these, I’m always careful not to shut them down when their answers are off target—rather, I talk about how we are now operating in a different framework—the research framework—and that that is different from the open web. The fourth one has a specific function, and that is to ask them to apply what we are about to talk about to something already in their minds. This kind of anchoring technique can help them to relate what you’re saying to their reality,a nd therefore, find more meaning in it.
• #33: Pre and post, or just post. Pre and post can gauge learning, post can gauge what they’re walking out with. If you can get teacher cooperation, you can also do them again as a follow up, to gauge retention on various time scales. Some of these are open ended, some multiple choice. It’s a good idea to have a variety. Not too many easy ones, or you may not learn anything useful. You can also format these as quick assignments. Great for classes with one set task—evaluate and cite websites, for example. You can also treat these like scavenger hunts, if you have enough time, and ask the students to get up and locate things.
• #35: You don’t need to be perfect your first time. If your questions aren’t perfect the first time or your instruction isnt perfect, that’s okay You will realize even as you get more accomplished, that you sometimes don’t say things perfectly! You can correct that on the spot. You can use answers as an opportunity to congratulate them for their awesomeness.
• #36: What is a minute paper? You have them write or type for just one minute—only two real questions (and maybe some basic information). On paper or online is your choice. It’s easier to tabulate things online though. Drawbacks: It’s hard to evaluate these in the moment, but you can. It’s also very hard to tie these in with your objectives—however, they are useful. I like to do a combination of hand-raising polling, with a post quiz that has a minute paper attached.
• #37: Gives them space to clarify and explain, if they want to. You can also invite them to leave contact info.
• #40: With polls and quizzes, if they’re brief, it’s easy to tabulate and review responses immediately. I like to do this. Skim over the answers and address problems right there. “It seems like I was unclear on Point X. Can someone describe the answer? HAVE YOUR ELEVATOR SPEECHES READY. This is the most important step. If you don’t integrate the information, it’s not useful to anyone. We’re not perfect—this kind of thing can be jarring at first. Change can relate to the way the questions are worded, ordered, or what is included, and also in the ways you teach.
• #41: Create a warm and welcoming environment by walking around in the beginning of class and during. Insure the students that the process is difficult and messy, and the assessments are intended to insure that they have what they need and have a chance to ask questions while you’re there to help. Let them know the things you learn from their struggles help you to improve the classes for future students. Appeal to altruism=awesome.
• #43: Most of these can be used for both.
• #45: Absorbed Zoomerang Basic: free 10 questions per survey 100 responses per survey No fancy things like skip logic Select: $26 per month ($312 per year) Unlimited questions Unlimited responses Custom urls Skip logic etc.
• #46: Can handle short answer, you can embed images, and it does quizzes and surveys in addition to polls. If you’re asking what the difference is between a quiz and a survey in terms of this software, if you structure your setup as a quiz, you can set it to know the correct answers in advance. Free account: Used to be limited to 100 responses per month, but they took that away. So, the free version should be everything you want unless you want custom urls.
• #47: This is usable with any computer, tablet, or phone—even old ones, and it can handle short comment answers. You can embed in PowerPoint. Free version 25 responses per poll or 40 responses per poll if you are in K through 12 or higher education. They don’t love public libraries, apparently. The pricing beyond that point gets kind of complicated based on your situation, but it’s $14 per month and up.
• #48: Free, but somewhat clunky to use.
• #51: List of verbs: http://www.acu.edu/academics/adamscenter/course_design/syllabus/verbs.html