Gamification for Crowdsourced Data Collection in Mobile Usability Field Studies

Editor's Notes

• #2: Work by Silvia Malatini, Ph.D. on usability for mobile device interfaces and conversational interfaces (Alexa, Siri).
• #3: In this work, focus on mobile application usability. Large body of previous works, existing literature, focused on usability for mobile devices/smartphones. Many previous usability studies have been done on older devices: smaller screens (hundreds of pixels), different form factors, different UI paradigms. How can guidelines and studies be kept up to date, easily and cheaply?
• #4: One aspect of the problem: growing screen size of smartphones. Usage patterns and usability problems change with screen size.
• #5: Well known image of reachability: with right hand usage (thumb), screen in lower-center is much more reachable than rest of screen. Many guidelines and UI patterns around this concept. Growing screen sizes mean that reachability areas have and will change.
• #6: Another issue: most literature focused on single hand (right) thumb usage. But usage of smartphones is diverse: different postures, depending on kind of work and application.
• #7: Not a new idea: crowdsourcing data collection through a mobile application on Android (Henze). Uncontrolled field trial using a “game” app as a vehicle. No data on promotion, how app is distributed, 100,000,000 taps “is it much/little?”, is this approach applicable generally and without much expense? No posture data, No orientation data, Relatively old, old device form factor.
• #8: Basic crowdsourcing setup (no groundbreaking idea here): Client application, collects data about touch usage by user. Collected data is sent to server for aggregation. Server produces raw data for analysis, heatmaps, and other graphs.
• #9: Once you get an application installed, make use of it: extensible application, supports a variety of games, which can be freely customized. First deployment/experiment: “Circle” game. Tapping on randomly-appearing targets. When hit, circle disappears and a new one appears. 30 seconds of time to play. Metrics collected: Inter-tap delay: measures touch performance by user, Precision (distance from target center): not always good metric (targets on edges, approximation by user), but widely used in literature.
• #10: First novel addition: 6 different postures considered. Selected at start of game.
• #11: Second addition. Gamification elements: Achievements, medals awarded when a set of levels is passed, Online leaderboard, with nicknames (if set) Voluntary sharing of achievements No user data is collected/stored. Investigating anonymous gamification elements (no online communication, no online profile, no registration) for future work. All games contribute to the overall leaderboard/score/achievement system.
• #12: 2 months of usage, Google Play. More than 50%: right-hand usage. Stylus and left-hand is negligible. Game design did not encourage variation of postures.
• #13: Sample. Left-hand samples omitted because of the small amount of data collected. Confirms what is generally known: right-hand usage has a “reachable” zone in the lower-right, where performance is higher than in rest of screen. Index finger usage has less delay on average than thumb-usage.
• #14: Performance declines at screen edges, users are more careful. No significant difference between left/right hand and index/thumb.
• #15: Precision grows at screen edges (but also higher delays, according to previous slide). Index is generally more precise than thumb. Left hand more precise on average: most users are right-handed and slower/more precise with left.
• #16: As screen size grows average target diameters increase (target are proportionate to screen). Precision decreases linearly with screen size: makes sense, average distance is around 1/5 of target diameter. Delay decreases with target size, stabilizes with diameter ≥ 10mm. Many guidelines argue that tap targets should never be smaller than 9mm.
• #17: Traditional usability studies/field trials require effort and are expensive to perform. Rapidly evolving landscape of hardware and software require updating of existing guidelines. Reusable application: Can be adapted rapidly to different games, different usability metrics, different interactions, etc. Benefits from existing user-base, once published, Can be used to collect data over time and document changes in usability issues depending on HW/SW evolution, Once developed, requires little to no effort to develop new games, which can be submitted to the existing user-base. Results are valid and generally match results from previous research. Postures and their effects are recognizable, Performance decreases as screen size increases, Performance decreases as target size decreases, Better performance at center of screen. Confirm: crowdsourcing works. Crowdsourcing is a precious resource, that can be enhanced by the use of gamification elements (high/rapid adoption, usage, engagement, keeping interest of users over time). Limitations: Reliability of data must be checked (postures), Gamification, does it work? Does it have an impact? Difficult to know. Usability studies in a game scenario, subject to “training effect”.