Using games and mixed realityfor understanding and communicating complex ideas: technical approaches and challenges

Editor's Notes

• #43: Let me tell you a story, far in the distant past…
• #44: …2007, 10 years ago. I was using Second Life to teach computer programming, along with one of my PhD students, Micaela Esteves. We were getting nice results, using it as a virtual toy programming platform, simulating shops for customer-tracking backoffice code, etc. So, naturally, we wanted to try it in more ambitious settings. What if a full class used virtual worlds for a semester, twice a week, every week?
• #45: The problem was that my PhD student lived xxxx km away, where she was a teaching assistant, and couldn’t be physically present twice a week. That’s OK, I figured, why don’t you conduct the classes entirely online in Second Life? Why travel at all? Where’s a spoiler: it worked, we learned a lot about how to do it, what happened, and you can actually find more details in a BJET paper published some years ago.
• #46: But there was a problem implementing it that we did not anticipate… …bureaucracy. “You need to keep track of attendance”, they said.
• #47: So… I was Andabata Mandelbrot. Who was everyone else? Avatar names wouldn’t match student names… But virtual environments are more fluid. When someone enters this room, or leaves it, we notice it. As in a physical classroom. But in a virtual world… people may “pop in” or “pop up” for lots of different reasons. Perhaps their network failed. Perhaps they weren’t interested and left. Perhaps they arrived late… but there is not virtual door opening. There is no gush of air, no sound of footsteps leaving, someone simply appears or disappears… When did the student arrive? When did the student leave? What was missed? Was it a moment, or a large piece of the lesson?
• #48: We did it by hand, as best as we could. Meanwhile, I supervised a Master student that came up with was a Moodle module to keep track of a minute-by-minute attendance of a class, and associate each student with its virtual world avatar. This generated an attendance chart for each student, showing when was the student present in the class, and allowing the teacher to decide if it could count as attending or not. It also allowed the teacher to realize what a student might have missed.
• #49: At the same time, we were feeling the chores of managing all the tasks in virtual world classes: Has the student progressed? What was delivered by each student? How do I deliver an object or component to each student? How do I track to whom I’ve already delivered components? Where, exactly, is a student working, within the virtual space? How do I provide contextual feedback? Do I have to give instructions on how to reach the location of an issue, instead of commenting there? Why can’t I annotate in 3D as in a shared Google Doc?
• #50: Altice Labs (at the time, called Portugal Telecom Innovation), asked us to help them use 3D virtual worlds in professional training. They were having much the same concerns, including aspects such as: How do we ensure that only the enrolled students participate? How can a training manager check what trainers are doing, i.e., how can we manage training sessions? Can we reuse and assign 3D objects to various training sessions, as we do with documents, presentations, videos or images? Who sets up a virtual space? Can it be automated? Can the notification/logging in/logging out workflow be automated, too?
• #51: This made us realize the importance of integration of virtual worlds and games with learning management systems. If games and virtual worlds – and indeed AR and Mixed Reality – are not embedded in the daily tasks of students, teachers, and managers, it’s hard to envision how they can become widespread. I believe they will, eventually, but their adoption rate, their dissemination will be slower – much slower than we desire.
• #52: Another happy circumstance occurred, at around the same time. I was approached by a Sports Sciences PhD student (who I ended up co-supervising) who had a radical new idea for training handball coaches on team tactics: use a virtual world. This way, you could teach the “perfect”, conceptualized idea of a tactic. No more depending on videotaping that lacked the proper perspective, or on endless rehearsing a tactic so that no player is early or late on the spot, looking the wrong way, gesturing unnecessarily, etc. But how could that Sports Sciences student define the tactics without the help of programmers?
• #53: We came up with a “magnet board” application which allowed the Sports Sciences student to drag markers on a timeline, and thus define the tactics. Then the resulting file was read by a replay program that would control 14 Second Life bots and replay the tactic upon request, with 3D avatars. The coaches during training sessions would collectively watch the tactic, in 3D, change the perspective at will, and the trainer could pause it, resume it, etc. They could even position themselves amidst the team, in order to better argue on a problem or new idea.
• #54: We were, however, concerned about the lock-in of our system with Second Life or OpenSimulator. The tactics were defined outside these systems, but the replay software was tied to its programming interfaces, the very definition of motion and events depended on Second Life / OpenSimulator’s concepts of motion, such as “activating a gesture”, moving “forward” for a few meters, etc. Knowledge on handball tactics will probably last more than the few years a given technological platform lasts! What about for research? If we encode tactics as we study teams in the physical world, e.g., the tactics of teams in a World Championship, do we tie them to the specific replay requirements of a game platform that will inevitably become outdated?
• #55: Shouldn’t we be able to pick a tactic from 2010, then replayed in Second Life, and be able to witness it in mixed reality today, displayed before our eyes, overlaid on a physical handball court? Sure, we’d like to use better-rendered players, but that’s the least of our concerns. Even if we had to make do with default avatars, even if we had to make do with 3D Monopoly game pieces, wouldn’t it be better than to simply lose our handball tactics data, lose our 3D educational content, lose our research records, only because technology evolved?
• #56: We’ve started working on this problem, and realized that games and virtual worlds – and indeed AR and mixed reality – should witness the emergence of a new type of data: virtual choreographies. That is, we should be able to let teachers and students define what happens in a game platform – which we already do, by playing the game – but reuse those events, those behaviors, for analysis, for discussion and review of ideas, for providing new students with our collection of prepared learning situations. And not just loose all of it when we move from a old technological platform for a new one. We can imagine a library of virtual choreographies – an “app store” of educational situations – that are shared among students, teachers, and across different games, AR & Mixed Reality platforms, that survive the obsolescence of their original rendering technologies. Knowledge must survive its technological platforms. Knowledge must be as eternal as its truthfulness. Virtual choreographies are our proposed solution for this.