Designing Learning Analytics for Humans with Humans

Editor's Notes

• #5: (6 advisers interviewed – led to the students) Schematics – phases for LAK. How sampled people.
• #6: Learning analytics (LA) is a technology for enabling better decision-making by teachers, students, and other educational stakeholders by providing them with timely and actionable information about learning-in-process on an ongoing basis. To be effective LA tools must thus not only be technically robust but also designed to support use by real people.
• #7: Learning analytics (LA) is a technology for enabling better decision-making by teachers, students, and other educational stakeholders by providing them with timely and actionable information about learning-in-process on an ongoing basis. To be effective LA tools must thus not only be technically robust but also designed to support use by real people. One powerful strategy for achieving this goal is to involve those who will (hopefully!) use the learning analytics in their design.
• #8: Schwendimann, B. A., Rodriguez-Triana, M. J., Vozniuk, A., Prieto, L. P., Boroujeni, M. S., Holzer, A., ... & Dillenbourg, P. (2016). Perceiving learning at a glance: A systematic literature review of learning dashboard research. IEEE Transactions on Learning Technologies, 10(1), 30-41. Bodily, R., & Verbert, K. (2017). Review of research on student-facing learning analytics dashboards and educational recommender systems. IEEE Transactions on Learning Technologies, 10(4), 405-418. Maybe they were involved a bit, but bringing them into a lab doesn’t cut it
• #9: Schwendimann, B. A., Rodriguez-Triana, M. J., Vozniuk, A., Prieto, L. P., Boroujeni, M. S., Holzer, A., ... & Dillenbourg, P. (2016). Perceiving learning at a glance: A systematic literature review of learning dashboard research. IEEE Transactions on Learning Technologies, 10(1), 30-41. Bodily, R., & Verbert, K. (2017). Review of research on student-facing learning analytics dashboards and educational recommender systems. IEEE Transactions on Learning Technologies, 10(4), 405-418.
• #10: de Quincey, E., Briggs, C., Kyriacou, T., & Waller, R. (2019, March). Student Centred Design of a Learning Analytics System. In Proceedings of the 9th International Conference on Learning Analytics & Knowledge (pp. 353-362) Cuban, L. (2001). Oversold and underused: Computers in the classroom. Cambridge, MA: Harvard University Press
• #11: de Quincey, E., Briggs, C., Kyriacou, T., & Waller, R. (2019, March). Student Centred Design of a Learning Analytics System. In Proceedings of the 9th International Conference on Learning Analytics & Knowledge (pp. 353-362) Cuban, L. (2001). Oversold and underused: Computers in the classroom. Cambridge, MA: Harvard University Press
• #12: Schwendimann, B. A., Rodriguez-Triana, M. J., Vozniuk, A., Prieto, L. P., Boroujeni, M. S., Holzer, A., ... & Dillenbourg, P. (2016). Perceiving learning at a glance: A systematic literature review of learning dashboard research. IEEE Transactions on Learning Technologies, 10(1), 30-41. Bodily, R., & Verbert, K. (2017). Review of research on student-facing learning analytics dashboards and educational recommender systems. IEEE Transactions on Learning Technologies, 10(4), 405-418.
• #13: Schwendimann, B. A., Rodriguez-Triana, M. J., Vozniuk, A., Prieto, L. P., Boroujeni, M. S., Holzer, A., ... & Dillenbourg, P. (2016). Perceiving learning at a glance: A systematic literature review of learning dashboard research. IEEE Transactions on Learning Technologies, 10(1), 30-41. Bodily, R., & Verbert, K. (2017). Review of research on student-facing learning analytics dashboards and educational recommender systems. IEEE Transactions on Learning Technologies, 10(4), 405-418.
• #14: Gathering information from intended users Observing teaching and learning practices Directly engaging them in participatory designIn this webinar, I'll present a diverse set of examples of the ways that NYU's Learning Analytics Research Network (NYU-LEARN) is including educators and students in the process of building and implementing learning analytics. involve students in the creation and revision of learning analytics solutions for their own use I research on you, ask you things, create with you (bring you here, I go to there)
• #19: Key theme of how to deal with the lack of awareness and ideas faculty may have about how to do this and what is possible Not just going to them and saying “what do you want form the data” We didn’t go with a blank slate We talked to them The conversation was questions do you have (not what data do you want to see)
• #20: (5)  The dashboards used in this study were developed by NYU IT based on the consultations with each instructor about the kinds of student activity and performance information they would like to see in the dashboards.   (6) The dashboard for each instructor consists of three to four distinct views. For example, student resource activity view was developed to help instructors to identify who were not engaging with the resource or which course materials were not frequently accessed.   This view displayed which course resource was accessed by each student, and visualized its number of times as size and its duration as color.
• #21: (7) Another view displayed which quiz item was difficult by the class, and visualized its average score and the number of students who completed this item.
• #22: Very different than usability testing
• #23: Tension -> opportunity. Create opportunities for tension, creates opportunities for tension. Generative. Rather than a “pat” process. Shout out – designers are aware of this, but institutions aren’t -> create space in your “production schedule” for this. Create space with the users for this. People never use your tool the way you expect. Creating the dashboard alone isn’t enough -> other artifacts (website, community)….
• #24: Semi-structured retrospective interviews where instructors walked us through their process of dashboard use
• #26: (9) Then, we tried to align the empirical findings into the hypothetical model. 20 themes emerged.  However, there was no theme conceptualized in checking impact. Rather, seven themes emerged related to the other important aspects of instructors use of the analytics.  These themes were synthesized into the final model we conceptualized as the process of instructor analytics use.
• #27: (1) The model we conceptualized consists of two-part structure with multiple phases. First, sense-making. Second, pedagogical response.
• #32: Gathering information from intended users Observing teaching and learning practices Directly engaging them in participatory designIn this webinar, I'll present a diverse set of examples of the ways that NYU's Learning Analytics Research Network (NYU-LEARN) is including educators and students in the process of building and implementing learning analytics. involve students in the creation and revision of learning analytics solutions for their own use I research on you, ask you things, create with you (bring you here, I go to there)
• #34: The project emerged as a collaboration between NYU IT + LEARN. IT had data which had been wrangled for previous project and saw opportunity. VALUE OF STUDENT_FACING DATA LEARN saw opportunity to approach process differently from faculty dashboard. Because when designing for students issues of power are more “dangerous” (with tools designed “on” students and not “for” students, Lab and IT discussed using participatory design as a process.) IT was keen, and *importantly* willing to slow down their delivery timelines to acommodate
• #35: There are different levels of participation in design. As discussed before, the lowest one is asking stakeholders, the highest one having stakeholders involved in every phase of the process. A way to conceptualise this is, to quote Hamilton, is to ask ourselves who is in the “room where it happens” and who isn’t. Who are we inviting to be part of the process? How much information do they have when they are in the room? Do they know all information of the project or are elements hidden to them, either because they could be complicated institutionally, or because there are barriers of knowledge (differences between experts and amateurs). At what times do they access the information and can participate? Is it constant or sporadic? Are the researchers and designers interpreting the stakeholder’s desires or are stakeholders designing themselves? In general it has been the case that students are excluded from these processes. We wanted to flip that in this project, inviting them and other stakeholder to multiple phases of the project.
• #36: PHASES OF PROJECT: On a grand level, we would have three phases; A needs analysis, where we would assemble stakeholders and interview stakeholders to understand the problems and opportunities in this space better, A process of co-design, where students, researchers and designers would participate in workshops to work together to come up with solutions to the challenges, and Iterative phases of prototyping, testing, and repeating to improve the prototypes. FROM -> WITH -> FROM / WITH STEPS We formed a steering commitee with different stakeholders. We had: Representatives from IT Academics The director of educational technology from one of the schools at NYU The head of advising staff at that school. PHASES OF PROJECT: With the help of the committee we outlined stakeholders who we would interview: faculty (3), advisors (6) and students (15). Advisor interviews surfaced that there were groups of students who were particularly challenged, and we discussed making them the “extreme users” who we would design for. Extreme users is a technique used often in design; you find someone who could use the tool, but who may have special challenges or needs, or use it very differently from other users. The assumption is that designing for this extreme needs allows for designs that meet the needs of average users better, and also meets the needs of those users who are not average. in a university this is specially important if we want to have a focus on equity. We identified extreme users as ‘first gen” students, who have special challenges in the school to college transition.
• #37: CALL FOR WORKSHOP We made an open call to students, with advertisement in housing and the library. We also specially emailed a list of 100 first generation students, from which many of the participants in the workshop came. The workshop format was chosen because of the difficulty of having a large number of students involved with the project for long periods of time. However, this created challenges of its own, and planning of the workshop was done to adress those challenges.
• #38: DESIGN SESSIONS: Our objectives in these sessions were: To gain insight into what types of tools students think they would use. Collaborate with students in designing these tools. Also, we gained insights into some of the challenges and fears of students when using tools that use their data. This was not unexpected but had not been incorporated into our design.
• #39: FOR THE SESSIONS, we had key ingredients we believe were central to the success; Leave the problem space wide and open. Sometimes when designing we limit ourselves to a clear problem, and that is, partly, the suggestion in some methodologies of design thinking. However, because students are not designers, there is an iterative process between finding solutions and gaining more insights about the problems, which lead to more nuanced or interesting solutions. Moreover, as the team explored the space, new issues and problems arose. For example, the design which the team finally developed as a promising solution, the “Hive”, emerged from the question of dealing with competing deadlines in multiple courses, which wasn’t even in the forefront of the discussion of the challenges to design for in the first of the design sessions. The openness allows for increased depth and exploration.
• #40: Power dynamics can be a challenge and an opportunity in workshops like these. Maybe part of the reason why these ones generated insights was because facilitators were PhD students, who students could relate to. Similarly in the faculty project there was certain rapport in the interviews because the interviewer was a fellow academic. Signaling that the space is safe by encouraging dissent, showing that you (the facilitator) does not have all the answers, performing low hierarchy, using humor, among other techniques, is important to allow them to be creative and sincere. More tangible than "I will create something for myself" → I will learning something today: DESIGN.
• #41: 2. Look and feed tensions. In human centered design we sometimes address the user directly and ask them for their need. But sometimes the need is unclear, and they may not even know they could use a solution until they see it. Often, the “real” or surprising needs and solutions come when tensions arise. Don’t sweep them under the rug, and be attentive to them. In our sessions, one of the students, for example, asked the facilitators whether the objective was to make students more academically successful, or happy. This led to a deep conversation about student wellbeing and the role of the university in student’s lives, which is represented in designs for an emotional tracker as part of the proposed solutions.
• #42: Phases of the workshop (4 min): Ice breakers or fire starters: given the little time for interaction between participants and the need for them to become comfortable with each other (because of the relevance of psychological safety to lead to creative insights), ice breakers are an overlooked but important part of the process. The ones chosen in this workshop were aimed at inciting group work and also making participants physically connect and be vulnerable with each other. Design Thinking Mini Lecture: In the workshop, participants learned about design thinking methodology before applying it. Empathy Mapping: Traditional Human-centered design technique. Personas were based on research discovery interviews, presented as an open-ended challenge to student participants in the form of quotes extracted from the interviews. Participants asked to fill in gaps with knowledge about themselves and peers. Phases of the workshop (4 min): Data Mapping: To get participants familiarized with the kinds of data available, chat about what data could be useful for the challenge. Ideation: First stab at coming up with ideas for a solution. Short list of ideas, selecting the ones that students thought were most promising, Prototyping, making paper graphic representations of the ideas, Then discussing the prototypes and making new versions of them.
• #43: THREE TOOLS On the workshops, there were three tools which, we believe, helped generate a lot of designs in a relatively short time. Empathy Mapping: It is used often in human centered design. The more time is spent on this part of the process, the more participants are getting into the mental space of our user. If we had had the time, we would have had the students actually participate in the interviews, and maybe co-research, talking to other students to understand their challenges. Because of time constraints, we designed a shortcut, which both brought into the room the insights of the research team, and took advantage of the fact that the participant students were not very different from the interviewed students themselves; they could flesh out with the knowledge they had from themselves and their peers, and use this experience to theorize what kinds of solutions could be appropriate. 
• #44: PHASES OF PROJECT: On a grand level, we would have three phases; A needs analysis, where we would assemble stakeholders and interview stakeholders to understand the problems and opportunities in this space better, A process of co-design, where students, researchers and designers would participate in workshops to work together to come up with solutions to the challenges, and Iterative phases of prototyping, testing, and repeating to improve the prototypes. FROM -> WITH -> FROM / WITH STEPS We formed a steering commitee with different stakeholders. We had: Representatives from IT Academics The director of educational technology from one of the schools at NYU The head of advising staff at that school. PHASES OF PROJECT: With the help of the committee we outlined stakeholders who we would interview: faculty (3), advisors (6) and students (15). Advisor interviews surfaced that there were groups of students who were particularly challenged, and we discussed making them the “extreme users” who we would design for. Extreme users is a technique used often in design; you find someone who could use the tool, but who may have special challenges or needs, or use it very differently from other users. The assumption is that designing for this extreme needs allows for designs that meet the needs of average users better, and also meets the needs of those users who are not average. in a university this is specially important if we want to have a focus on equity. We identified extreme users as ‘first gen” students, who have special challenges in the school to college transition.
• #45: PHASES OF PROJECT: On a grand level, we would have three phases; A needs analysis, where we would assemble stakeholders and interview stakeholders to understand the problems and opportunities in this space better, A process of co-design, where students, researchers and designers would participate in workshops to work together to come up with solutions to the challenges, and Iterative phases of prototyping, testing, and repeating to improve the prototypes. FROM -> WITH -> FROM / WITH STEPS We formed a steering commitee with different stakeholders. We had: Representatives from IT Academics The director of educational technology from one of the schools at NYU The head of advising staff at that school. PHASES OF PROJECT: With the help of the committee we outlined stakeholders who we would interview: faculty (3), advisors (6) and students (15). Advisor interviews surfaced that there were groups of students who were particularly challenged, and we discussed making them the “extreme users” who we would design for. Extreme users is a technique used often in design; you find someone who could use the tool, but who may have special challenges or needs, or use it very differently from other users. The assumption is that designing for this extreme needs allows for designs that meet the needs of average users better, and also meets the needs of those users who are not average. in a university this is specially important if we want to have a focus on equity. We identified extreme users as ‘first gen” students, who have special challenges in the school to college transition.
• #47: THREE TOOLS Design cards: To make sure that the conversation and solutions stayed closed to the existing data, as well as to upskill students to think about data like learning scientists, a set of cards were created. (go to next slide)
• #48: Blue cards - actions / verbs; What data may do which aids learning and insight Green - types of information and inferences you can make from them; what data we have. Orange - Feelings; There to stimulate lateral thinking (looking at the combinations from a different perspective). Students would randomly join a blue and a green card, and if blocked could add an orange card. Each “set” was a “solution”, a description of what a solution could do, which inspired students to think of a “form” (how would the solution look like, what does it do to achieve that result, and what are the implications). Randomized cards can be a great way to both bring information into a discussion, and create random serendipitous combinations that can lead participants to discovery and insight.
• #49: THREE TOOLS On the workshops, there were three tools which, we believe, helped generate a lot of designs in a relatively short time. Live prototyping. Instead of students simply sharing ideas, both them and designers developed images of prototypes. The images of designers would then be shared back with participants to immediately member check and improve. This led to multiple iterations upon ideas in a single session.
• #50: This is a student’s first rendering of the idea of the “Hive”; a space where they could see, organized, multiple courses and resources.
• #51: Facilitators and designers then played with the idea alongside students, incorporating ideas such as “rate of completion” of different courses, or forms of comparing which courses had the most activity (which could signal the need for attention).
• #52: Students played with the idea further, and after the workshops, designers began playing with the concept. This is the first digital prototype; the bottom is a timeline with assignments, sized by “grade impact”
• #53: THIS IS A SECOND ITERATION, which adds other tools that students designed, and adds for example, an emotional tracker and notifications with information from different courses.
• #54: FROM THE WORKSHOP Three insights emerged: It seems that students lean more towards solutions with a strong social component. This is not surprising if we think for a second; they interact with social media daily. Moreover, both on the interviews and in the workshops, they described experiences where some peer (a mentor, another student) provided information which helped them reframe situations and face some of their academic and life challenges at the university better. However, many of the student-facing solutions that the field has produced don’t have a strong social component, which hints to an opportunity for the field to use analytics and social media combined. While many of the solutions that the field has produced are centered on academic needs, students saw their academic and life needs as intertwined. This is consistent on what we know about student experience, and the fact that it is not rare that what looks like an academic challenge is actually a life challenge in a student’s life (adjusting to college, depression, loneliness, etc.). As we promote holistic approaches for learning, so we should perhaps with analytics-powered solutions. Consistent with the two previous points, in student’s solutions there is data and elements of dashboards, but they were often only part of the solution. Data was there to help, but it was not the only way to get help nor often the most important. They leaned toward systems which would do multiple things for them, as opposed to the dashboards that are often proposed as solutions.
• #55: Slowness – space and time and partner Important – value of directions we never would have gone otherwise
• #56: Very different than usability testing