Toolkit 4 evaluating students learning
Download as PPTX, PDF2 likes561 views
The document discusses different types of questions that instructors can ask to evaluate the effectiveness and impact of teaching projects, including questions about whether learning outcomes were achieved, how student learning can be better understood, and how students are experiencing the course. It provides examples of specific questions and suggests collecting a variety of evidence, such as student work and surveys, to help answer evaluation questions. The goal of evaluation is to determine what is working and how teaching can be improved.
![Inquiry into Student Learning
Questions
• What thinking processes do
students engage in when
learning about X?
• What are students’ beliefs about
how they best learn Discipline X?
• How do practices of self-
reflection change their learning
about X?
• What do students find most
difficult about learning in Course
X?
• How do students prior
understanding of Discipline X
affect their ability to acquire
new understanding in that field?
• How does their attitude towards
Y affect their ability to do work
on issue X?
“Here the effort is aimed not so
much at proving (or disproving) the
effectiveness of a particular
approach or intervention but at
describing what it [learning] looks
like, what its constituent features
might be.” – Pat Hutchings](https://image.slidesharecdn.com/toolkit4-evaluatingstudentslearning-150526173929-lva1-app6892/85/Toolkit-4-evaluating-students-learning-7-320.jpg)
Toolkit 4 evaluating students learning
- 1. Evaluating Your Students’ Learning & Whether Your Project Is Succeeding Innovators’ Toolkit 4 Julie Sievers, Center for Teaching Excellence
- 2. Why Evaluate Your Project? To see if it was worth your time and effort To decide whether your colleagues / department / field should adopt similar strategies in other courses. To enable you to base your arguments about these issues on evidence, not just anecdote and impression. Scholarship of Teaching & Learning: To enable you to present at a conference, or publish in a journal
- 3. A Taxonomy of Questions 1. Assessing Effectiveness of Course, Project, Method • Did I accomplish my goals? • Does this strategy work? • Does it work better than other strategies? “What Works?” Q – in Pat Hutchings taxonomy of questions, Opening Lines (2000) Assessment Q: – Similar to program assessment for SACS • “Are you accomplishing what you set out to do?” • “How do you know?”
- 4. Effectiveness Questions Success or failure question: Did students meet the SLOs? Yes or no. • Variation: Did students’ meet this or that SLO? • Collect evidence similar to that used for program assessment. Comparative question: Did the students meet the SLOs BETTER in this version of the class than in another? • Requires comparison data • May compare to your previous version, or a standard version you are also offering. • May compare to a colleague’s course. Improvement question: Did the students IMPROVE in their ability to meet an SLO over the course of the semester? • Requires baseline data – where were students on this SLO in week 1? Where are they in week 15? • Proves success even if SLO is still not met Yardstick: learning outcomes
- 5. More effectiveness questions Questions about non-SLO goals For example: Did this project . . . • Increase student engagement? • Increase student motivation? • Facilitate deeper learning? • Improve classroom climate? • Create a classroom that supports productive failure Yardstick: Your goals (need to be articulated, but does not have to be an official SLO)
- 6. A Taxonomy of Questions 2. Inquiry Into Student Learning Goal: To better understand • learning processes, • classroom social dynamics, • student errors or misconceptions, • the roles played by various factors in student learning, etc. • Descriptive / documentarian • Less oriented around practical questions of a method, more about fundamental learning issues and processes at work in the course / experience. “What Is (Happening)?” Q – in Pat Hutchings taxonomy of questions, Opening Lines (2000) SoTL Q • Common in Scholarship of Teaching & Learning inquiry • Less similar to assessment questions.
- 7. Inquiry into Student Learning Questions • What thinking processes do students engage in when learning about X? • What are students’ beliefs about how they best learn Discipline X? • How do practices of self- reflection change their learning about X? • What do students find most difficult about learning in Course X? • How do students prior understanding of Discipline X affect their ability to acquire new understanding in that field? • How does their attitude towards Y affect their ability to do work on issue X? “Here the effort is aimed not so much at proving (or disproving) the effectiveness of a particular approach or intervention but at describing what it [learning] looks like, what its constituent features might be.” – Pat Hutchings
- 8. A Taxonomy of Questions 3. How Are Things Going? Goal: To better understand • How students are experiencing the course • How students are doing – before they submit a major assignment • Student preconceptions, misconceptsion • Student study and work habits / processes May be: • Anonymous • Ungraded • Diagnostic Designed to enable you to intervene quickly before semester / unit / project – even class! -- is over
- 9. How Are Things Going Questions Simple / quick tools: • Misconception / Preconception check • minute paper • Muddiest point (minute) paper • Classroom opinion polls • Course-related self-confidence surveys • Productive study-time logs • Annotated portfolios • Documented problem solutions • mid-semester evaluation (These can also be evidence towards answering your other questions) See Angelo & Cross, Classroom Assessment Techniques
- 10. What’s Your Problem, Anyway? “One telling measure of how differently teaching is regarded from traditional scholarship or research within the academy is what a difference it makes to have a “problem” in one versus the other. In scholarship and research, having a “problem” is at the heart of the investigation process; it is the compound of the generative questions around which all creative and productive activity revolves. But in one’s teaching, a “problem” is something you don’t want to have, and if you have one, you probably want to fix it. Asking a colleague about a problem in his or her research is an invitation; asking about a problem in one’s teaching would probably seem like an accusation. Changing the status of the problem in teaching from terminal remediation to ongoing investigation is precisely what the movement for a scholarship of teaching is all about.” Randy Bass, “The Scholarship of Teaching: What’s the Problem?” Inventio: Creative Thinking about Learning and Teaching. (1999)
- 11. Activity: Draft Your Question(s) By Yourself -- 10 minutes 1. Brainstorm a list of questions you wish to answer at end of project 2. Narrow down list to the top 1 or 2 3. Revise the questions to 1. make them more specific 2. Make them measurable With your peer -- 5 min each 1. Explain your top 1 or 2 questions. 2. Peer’s job: Help peer revise questions. Pairs: • Alex & Yuliya • Gary & Jason • Richard & Mary • Kate & Jimmy • Chris & Rachael
- 12. Answering Your Question with Evidence Evidence . . . AKA • Data • Artifacts • Student work Some things to consider: • Direct vs. indirect evidence • Qualitative, quantitative, or both? • Embedded (“normal educational practice”) vs. add-on • Product vs. process
- 13. Two Examples From the “How to Start” worksheet in the Vanderbilt SoTL Guide For literature class, in which a “difficulty log” has been introduced, to be completed as students read a text. • Analyze the logs, looking for themes or patterns in responses. From the logs, document specific types of difficulty. From calculus class, in which students have been asked to document their problem solving steps as a way of helping them develop metacognitive skills. • Looking at scores on test prior to and after the new activity.
- 14. Activity: What’s Your Evidence? By Yourself -- 5 min Brainstorm the types of evidence you will need to collect to answer your question? Evidence should fit: • Your questions • Your discipline • Your course • Your timeline With your peer -- 5 min each 1. Explain your evidence choices. 2. Peer’s job: Help peer refine plan. Pairs: • Alex & Yuliya • Gary & Jason • Richard & Mary • Kate & Jimmy • Chris & Rachael
- 15. Blog Post 5 Post to your blog your question and the evidence you plan to collect to answer that question. Discuss your choices. Due: Fri, May 29