Gestural Interaction, Is it Really Natural?
Download as PPTX, PDF0 likes123 views
The document summarizes Jean Vanderdonckt's upcoming lecture on gestural interaction. It will cover the psychological, hardware, software, usage, social and user experience dimensions of gestural interaction. On the psychological dimension, it discusses definitions of gestures and theories of gesture types. On the hardware dimension, it outlines paradigms of contact-based and contact-less gesture interaction. On the software dimension, it provides an overview of gesture recognition algorithms such as Rubine, Siger, LVS and nearest neighbor classification.
![• McNeil’s interpretation of Kendon’s continuum (1994)
Psychological dimension: What is a gesture?
Gesticulation
Speech-framed gestures
Pantomimes
Emblems
Signs
“Adam Kendon once
distinguished gestures
of different kinds along
a continuum that I
named “Kendon's
Continuum”, in his
honor.” [McNeill, 1992]
Source: D. McNeill. Hand and Mind: What Gesture Reveals about Thought. University Chicago Press, 1992
Kendon, A., Do gestures communicate? A review. Research on Language and Social Interaction 27, 1994, 175-200
Mandatory presence
of speech
Optional presence
of speech
Mandatory presence of
speech frames
Optional absence
of speech
Mandatory absence
of speech](https://image.slidesharecdn.com/vanderdonckt-francquichair-20feb2020-210501183614/85/Gestural-Interaction-Is-it-Really-Natural-13-320.jpg)
![• McNeil’s interpretation of Kendon’s continuum (1994)
Psychological dimension: What is a gesture?
Gesticulation
Speech-framed gestures
Pantomimes
Emblems
Signs
Source: D. McNeill. Hand and Mind: What Gesture Reveals about Thought. University Chicago Press, 1992
Kendon, A., Do gestures communicate? A review. Research on Language and Social Interaction 27, 1994, 175-200
Mandatory presence
of speech
Optional presence
of speech
Mandatory presence of
speech frames
Optional absence
of speech
Mandatory absence
of speech
“As one moves along Kendon’s
Continuum, two kinds of
reciprocal changes occur. First,
the degree to which speech is an
obligatory accompaniment of
gesture decreases from
gesticulation to signs. Second, the
degree to which a gesture shows
the properties of a language
increases.”
[McNeill, 1992]](https://image.slidesharecdn.com/vanderdonckt-francquichair-20feb2020-210501183614/85/Gestural-Interaction-Is-it-Really-Natural-14-320.jpg)
![36
Nearest-Neighbor-Classification (NNC)
• Pre-processing steps to ensure invariance
• Re-sampling
• Points with same space between: isometricity
• Points with same timestamp between: isochronicity
• Same amount of points: isoparameterization
• Re-Scaling
• Normalisation of the bounding box into [0..1]x[0..1] square
• Rotation to reference angle
• Rotate to 0°
• Re-rotating and distance computation
• Distance computed between candidate gesture and
reference gestures (1-NN)
Software dimension: Which algorithm?](https://image.slidesharecdn.com/vanderdonckt-francquichair-20feb2020-210501183614/85/Gestural-Interaction-Is-it-Really-Natural-36-320.jpg)
![37
Nearest-Neighbor-Classification (NNC)
• Two families of approaches
• “Between points” distance
• $-Family recognizers: $1, $3, $N, $P, $P+,
$V, $Q,…
• Variants and optimizations: ProTractor,
Protactor3D,…
• “Vector between points” distance
• PennyPincher, JackKnife,…
[Vatavu R.-D. et al, ICMI ’12]
[Taranta E.M. et al, C&G ’16]
Software dimension: Which algorithm?](https://image.slidesharecdn.com/vanderdonckt-francquichair-20feb2020-210501183614/85/Gestural-Interaction-Is-it-Really-Natural-37-320.jpg)
Gestural Interaction, Is it Really Natural?
- 1. Francqui Chair 2020, Inaugural Lesson: Gestural Interaction, Is it Really Natural? Jean Vanderdonckt, UCLouvain Vrije Universiteit Brussel, February 20, 2020, 4 pm-6 pm Location: Room I.0.02, Pleinlaan 2, B-1050 Brussels Presented by Prof. Dr. Beat Signer
- 3. 3 Jean Vanderdonckt Université catholique de Louvain (UCLouvain) Louvain School of Management (LSM) Louvain Research Institute in Management and Organizations (LouRIM) Institute of Information and Communication Technologies, Electronics and Applied Mathematics (ICTEAM) Director of Louvain Interaction Lab Place des Doyens, 1 – B-1348 Louvain-la-Neuve, Belgium
- 4. Gestural Interaction (Francqui Chair, VUB, Brussels, February 20, 2020) 4 Some links • Web site https://wise.vub.ac.be/news/francqui-chair-2020-prof-jean- vanderdonckt • Join me on • SlideShare: https://www.slideshare.net/jeanvdd • LinkedIn: https://www.linkedin.com/in/jeanvdd/ • YouTube: https://www.youtube.com/user/jeanvdd • Amazon: https://www.amazon.com/Jean- Vanderdonckt/e/B01640UKYK
- 7. 7 • The intersection between the 5 human senses gives rise to many possible interaction modalities Psychological dimension: What is a gesture? Gesture
- 8. 8 • The intersection between the 5 human senses gives rise to many possible interaction modalities • Gesture communication emerges in young children even before development of language Psychological dimension: What is a gesture? Source: S.W. Goodwyn, L.P. Acredolo, C. Brown. (2000). Impact of Symbolic Gesturing on Early Language Development. Journal of Nonverbal Behavior 24, 81-103
- 9. 9 • Innate gestures (natural?) • Gestures that the user intuitively knows or that make sense, based on the person’s understanding of the world • Examples • Pointing to aim a target • Grabbing to pick an object (MS Kinect) • Pushing to select something • Learned gestures (less natural => memorability?) • Gestures the user needs to learn before • Examples • Waving to engage • Making a specific pose to cancel an action Psychological dimension: What is a gesture?
- 10. 10 • The intersection between the 5 human senses gives rise to many possible interaction modalities • Gesture communication emerges in young children even before development of language • Blind people gesture as they speak just as much as sighted individuals do, even when they know their listener is also blind. Psychological dimension: What is a gesture? Source: S.W. Goodwyn, L.P. Acredolo, C. Brown. (2000). Impact of Symbolic Gesturing on Early Language Development. Journal of Nonverbal Behavior 24, 81-103
- 11. 11 • The intersection between the 5 human senses gives rise to many possible interaction modalities • Gesture communication emerges in young children even before development of language • Blind people gesture as they speak just as much as sighted individuals do, even when they know their listener is also blind • People gesture without a visual model • Gestures therefore require neither a model nor an observant partner Psychological dimension: What is a gesture? Source: J. M. Iverson, S. Goldin-Meadow. (1998). Why people gesture when they speak. Nature, 396:228
- 12. 12 • Kendon’s classification of gestures (1972) Psychological dimension: What is a gesture? Gesticulation Spontaneous movements of the hands and arms that accompany speech Speech-framed gestures Gesticulation that is integrated into a spoken utterance, replacing a particular word Pantomimes Gestures that depict objects or actions, with or without an accompanying speech Emblems Familiar gestures accepted as a standard Signs Complete linguistic system
- 13. • McNeil’s interpretation of Kendon’s continuum (1994) Psychological dimension: What is a gesture? Gesticulation Speech-framed gestures Pantomimes Emblems Signs “Adam Kendon once distinguished gestures of different kinds along a continuum that I named “Kendon's Continuum”, in his honor.” [McNeill, 1992] Source: D. McNeill. Hand and Mind: What Gesture Reveals about Thought. University Chicago Press, 1992 Kendon, A., Do gestures communicate? A review. Research on Language and Social Interaction 27, 1994, 175-200 Mandatory presence of speech Optional presence of speech Mandatory presence of speech frames Optional absence of speech Mandatory absence of speech
- 14. • McNeil’s interpretation of Kendon’s continuum (1994) Psychological dimension: What is a gesture? Gesticulation Speech-framed gestures Pantomimes Emblems Signs Source: D. McNeill. Hand and Mind: What Gesture Reveals about Thought. University Chicago Press, 1992 Kendon, A., Do gestures communicate? A review. Research on Language and Social Interaction 27, 1994, 175-200 Mandatory presence of speech Optional presence of speech Mandatory presence of speech frames Optional absence of speech Mandatory absence of speech “As one moves along Kendon’s Continuum, two kinds of reciprocal changes occur. First, the degree to which speech is an obligatory accompaniment of gesture decreases from gesticulation to signs. Second, the degree to which a gesture shows the properties of a language increases.” [McNeill, 1992]
- 15. • McNeil’s interpretation of Kendon’s continuum (1994) Psychological dimension: What is a gesture? Gesticulation Speech-framed gestures Pantomimes Emblems Signs Source: D. McNeill. Hand and Mind: What Gesture Reveals about Thought. University Chicago Press, 1992. “Gestures enhance, complement, and sometimes even replace speech.”
- 16. • Gesture (…) are communicative movements of the hands and arms which express — just as language — speakers’ attitudes, ideas, feelings and intentions…” (Müller, 1998) Psychological dimension: What is a gesture?
- 17. • Saffer’s definition: “a gesture (…) is any physical movement that a digital system can sense and respond to without the aid of a traditional pointing devices, such as a mouse or stylus” Psychological dimension: What is a gesture? Source: Saffer, D., Designing Gestural Interfaces, O'Reilly Media, November 2008. Sensor Gesture recognizer Actuator Context of use = (User, Platform/device, Environment) Disturbances Feedback feeds drives operates on is sensed by produces
- 18. • Turk’s definition in Human-Computer Interaction (2002) • ”…expressive, meaningful body motions –i.e. physical movements of the fingers, hands, arms, head, face or body with the intent to convey information or interact with the environment.” Psychological dimension: What is a gesture? Sources: Turk, M. (2002). Gesture Recognition. In K. M. Stanney (Ed.), Handbook of Virtual Environments (pp. 223–237). London: Lawrence Erlbaum Associates, Publishers. Isabel Benavente Rodriguez, Nicolai Marquardt, Gesture Elicitation Study on How to Opt-in & Opt-out from Interactions with Public Displays, Proc. of ISS ‘17, pp. 32-41.
- 19. • Aigner et al.’s taxonomy of mid- air gestures • P= Pointing gestures (= deictic gestures) indicate people, objects, directions Psychological dimension: What is a gesture? Source: Aigner, R., Wigdor, D., Benko, H., Haller, M., Lindlbauer, D., Ion, A., Zhao, S., et al. (2012). Understanding Mid-Air Hand Gestures: A Study of Human Preferences in Usage of Gesture Types for HCI.
- 20. • Aigner et al.’s taxonomy of mid- air gestures • Semaphoric gestures are hand postures and movements conveying specific meanings • T= Static semaphorics are identified by a specific hand posture. Example: a flat palm facing from the actor means “stop”. • D= Dynamic semaphorics convey information through their temporal aspects. Example: a circular hand motion means “rotate” • S= Semaphoric strokes represent hand flicks are single, stroke-like movements. Example: a left flick of the hand means “dismiss this object” Psychological dimension: What is a gesture? Source: Aigner, R., Wigdor, D., Benko, H., Haller, M., Lindlbauer, D., Ion, A., Zhao, S., et al. (2012). Understanding Mid-Air Hand Gestures: A Study of Human Preferences in Usage of Gesture Types for HCI.
- 21. • Aigner et al.’s taxonomy of mid- air gestures • M= Manipulation gestures a guide movement in a short feedback loop. Thus, they feature a tight relationship between the movements of the actor and the movements of the object to be manipulated. The actor waits for the entity to “follow” before continuing Psychological dimension: What is a gesture? Source: https://www.microsoft.com/en-us/research/publication/understanding-mid-air-hand-gestures-a-study-of-human- preferences-in-usage-of-gesture-types-for-hci/
- 22. • A gesture is any particular type of body movement performed in 1D, 2D, or 3D. • e.g., Hand movement (supination, pronation, etc.) • e.g., Head movement (lips, eyes, face, etc.) • e.g., Full body movement (silhouette, posture, etc.) Psychological dimension: What is a gesture? Source: Kendon, A. Gesture: Visible Action as Utterance. Cambridge: Cambridge University Press, 2004.
- 23. • A gesture is any particular type of body movement performed in 1D, 2D, or 3D Psychological dimension: What is a gesture? Individual body part Combined body parts
- 25. 25 Hardware dimension: How to gesture? Paradigms of gesture interaction Contact-based interaction (surface limitation?)
- 26. 26 Hardware dimension: How to gesture? Paradigms of gesture interaction Contact-based interaction (surface limitation?) Contact-less interaction With wearable
- 27. 27 Hardware dimension: How to gesture? Paradigms of gesture interaction Contact-based interaction (surface limitation?) Contact-less interaction With wearable Without wearable Close Far
- 29. 29 Software dimension: Which algorithm? In Window mode In full screen mode Source: https://www.usenix.org/legacy/publications/library/proceedings/usenix03/tech/freenix03/full_papers/worth/worth_html/xstroke.html XStroke
- 30. 30 Software dimension: Which algorithm? Rubine (1991)
- 31. 31 Software dimension: Which algorithm? Siger (2005) Training phase Vector string: Stroke Vector direction Left L Right R Up U Down D Regular expression: (NE|E|SE)+(NW|N|NE)+(SW|W|NW)+(SE|S|SW)+. LU,U,U,U,U,U,RU,RU,RU,RU,RU,RU,R,R,R,R,R,R,R,R,RD,RD,RD, RD,D,D,D,D,LD,LD,LD,LD,L,L,LD,LD,LD,D,D,D,D,D,D,D,D,D,L
- 32. 32 Software dimension: Which algorithm? LVS (2007)
- 33. 33 Software dimension: Which algorithm? Source: Beat Signer, U. Kurmann, Moira C. Norrie: iGesture: A General Gesture Recognition Framework. ICDAR 2007: 954-958
- 34. 34 Software dimension: Which algorithm?
- 35. 35 Nearest-Neighbor-Classification (NNC) for 2D strokes Y 0 0.25 0.50 0.75 1 X 0 0.25 0.50 0.75 1 = candidate point x = reference point x x x x x x x x x x x x x x k nearest neighbors 1 nearest neighbor distance 0 1 x x x x x x x x x x x Reference gestures Training set candidate gesture p2 p3 p4 p1 k-NN k nearest neighbors 1-NN Single nearest neighbor applied to gesture recognition q3 q1 q2 q4 reference gesture Software dimension: Which algorithm?
- 36. 36 Nearest-Neighbor-Classification (NNC) • Pre-processing steps to ensure invariance • Re-sampling • Points with same space between: isometricity • Points with same timestamp between: isochronicity • Same amount of points: isoparameterization • Re-Scaling • Normalisation of the bounding box into [0..1]x[0..1] square • Rotation to reference angle • Rotate to 0° • Re-rotating and distance computation • Distance computed between candidate gesture and reference gestures (1-NN) Software dimension: Which algorithm?
- 37. 37 Nearest-Neighbor-Classification (NNC) • Two families of approaches • “Between points” distance • $-Family recognizers: $1, $3, $N, $P, $P+, $V, $Q,… • Variants and optimizations: ProTractor, Protactor3D,… • “Vector between points” distance • PennyPincher, JackKnife,… [Vatavu R.-D. et al, ICMI ’12] [Taranta E.M. et al, C&G ’16] Software dimension: Which algorithm?
- 38. 38 Nearest-Neighbor-Classification (NNC) • Two families of approaches • “Between points” distance • $-Family recognizers: $1, $3, $N, $P, $P+, $V, $Q,… • Variants and optimizations: ProTractor, Protactor3D,… • “Vector between points” distance • PennyPincher, JackKnife,… • A third new family of approaches • “Vector between vectors” distance: our approach Software dimension: Which algorithm?
- 39. 39 • Local Shape Distance between 2 triangles based on similarity (Roselli’s distance) 𝑎 𝑏 𝑢 𝑣 𝑎 + 𝑏 𝑢 + 𝑣 Paolo Roselli Università degli Studi di Roma, Italy Software dimension: Which algorithm? Source: Lorenzo Luzzi & Paolo Roselli, The shape of planar smooth gestures and the convergence of a gesture recognizer, Aequationes mathematicae volume 94, 219–233(2020).
- 40. 40 • Step 1. Vectorization for each pair of vectors between three consecutive points p1 p2 p3 p4 p5 p6 q1 q2 q3 q4 q5 q6 Training gesture Candidate gesture Software dimension: Which algorithm? Source: Jean Vanderdonckt, Paolo Roselli, Jorge Luis Pérez-Medina, !FTL, an Articulation-Invariant Stroke Gesture Recognizer with Controllable Position, Scale, and Rotation Invariances. ICMI 2018: 125-134
- 41. 41 • Step 1. Vectorization for each pair of vectors between three consecutive points p1 p2 p3 p4 p5 p6 q1 q2 q3 q4 q5 q6 p1 p2 p3 p4 p5 p6 q4 q5 q6 Training gesture Candidate gesture q1 q2 q3 Software dimension: Which algorithm? Source: Jean Vanderdonckt, Paolo Roselli, Jorge Luis Pérez-Medina, !FTL, an Articulation-Invariant Stroke Gesture Recognizer with Controllable Position, Scale, and Rotation Invariances. ICMI 2018: 125-134
- 42. 42 • Step 2. Mapping candidate’s triangles onto training gesture’s triangles p1 p2 p3 p4 p5 p6 q1 q2 q3 q4 q5 q6 p1 p2 p3 p4 p5 p6 q1 q2 q3 q4 q5 q6 Training gesture Candidate gesture p1 p2 p3 p2 p3 p4 p3 p4 p5 p4 p5 p6 Software dimension: Which algorithm? Source: Jean Vanderdonckt, Paolo Roselli, Jorge Luis Pérez-Medina, !FTL, an Articulation-Invariant Stroke Gesture Recognizer with Controllable Position, Scale, and Rotation Invariances. ICMI 2018: 125-134
- 43. 43 • Step 2. Mapping candidate’s triangles onto training gesture’s triangles p1 p2 p3 p4 p5 p6 q1 q2 q3 q4 q5 q6 p1 p2 p3 p4 p5 p6 q1 q2 q3 q4 q5 q6 Training gesture Candidate gesture p1 p2 p3 p2 p3 p4 p3 p4 p5 p4 p5 p6 Software dimension: Which algorithm? Source: Jean Vanderdonckt, Paolo Roselli, Jorge Luis Pérez-Medina, !FTL, an Articulation-Invariant Stroke Gesture Recognizer with Controllable Position, Scale, and Rotation Invariances. ICMI 2018: 125-134
- 44. 44 • Step 3. Computation of Local Shape Distance between pairs of triangles p1 p2 p3 p4 p5 p6 q1 q2 q3 q4 q5 q6 p1 p2 p3 p4 p5 p6 q1 q2 q3 q4 q5 q6 p1 p2 p3 p3 p4 p5 p2 p3 p4 p4 p5 p6 Training gesture Candidate gesture p1p2p3, q1q2q3 (N)LSD ( ) ( =0.02 p2p3p4, q2q3q4 ) =0.04 (p3p4p5, q3q4q5 ) =0.0001 ) p4p5p6, q3q4q5 ( =0.03 Software dimension: Which algorithm? Source: Jean Vanderdonckt, Paolo Roselli, Jorge Luis Pérez-Medina, !FTL, an Articulation-Invariant Stroke Gesture Recognizer with Controllable Position, Scale, and Rotation Invariances. ICMI 2018: 125-134
- 45. 45 • Step 4. Summing all individual figures into final one • Step 5. Iterate for every training gesture p1 p2 p3 p4 p5 p6 q1 q2 q3 q4 q5 q6 p1 p2 p3 p4 p5 p6 q1 q2 q3 q4 q5 q6 p1 p2 p3 p3 p4 p5 p2 p3 p4 p4 p5 p6 Training gesture Candidate gesture p1p2p3, q1q2q3 (N)LSD ( ) p2p3p4, q2q3q4 ) ( ) p4p5p6, q3q4q5 ( (p3p4p5, q3q4q5 ) =0.02 =0.04 =0.0001 =0.03 =0.02+0.04+0.0001+0.03=0.0901 (indicative figures) Software dimension: Which algorithm? Source: Jean Vanderdonckt, Paolo Roselli, Jorge Luis Pérez-Medina, !FTL, an Articulation-Invariant Stroke Gesture Recognizer with Controllable Position, Scale, and Rotation Invariances. ICMI 2018: 125-134
- 46. 3D Hand gesture recognition Software dimension: Which algorithm?
- 47. Full body gesture recognition Software dimension: Which algorithm? See video at https://youtu.be/RTEGMlDRDL0
- 49. • Many application domains Usage dimension: Which application domains?
- 50. • Smart Home: TV, fridge, coffee machine,… • Example: Samsung Smart TV Usage dimension: Which application domains?
- 51. • Ring device: gesture elicitation study Usage dimension: Which application domains? Source: Bogdan-Florin Gheran, Jean Vanderdonckt, Radu-Daniel Vatavu, Gestures for Smart Rings: Empirical Results, Insights, and Design Implications. Conference on Designing Interactive Systems 2018: 623-635
- 52. • Ring device at Home (Family management) Usage dimension: Which application domains?
- 53. • Ring device at Home (Family management) Usage dimension: Which application domains?
- 55. • History: chironomia Social dimension: critical factors Source: Gilbert Austin, Chironomia, or a Treatise on Rhetorical Delivery (1806). Ed. Mary Margaret Robb and Lester Thonssen. Carbondale, IL: Southern Illinois UP, 1966.
- 56. Social dimension: critical factors 56 Gestures in Movies • 12 Angry Men (dir.: S. Lumet) The defense and the prosecution have rested and the jury is filing into the jury room to decide if a young man is guilty or innocent of murdering his father. What begins as an open-and-shut case of murder soon becomes a detective story that presents a succession of clues creating doubt, and a mini-drama of each of the jurors' prejudices and preconceptions about the trial, the accused, and each other. Based on the play, all of the action takes place on the stage of the jury room.
- 57. Gesture in Movies Social dimension: critical factors
- 58. Range of motion • Range of motion • Relates the distance between the position of the human body producing the gesture and the location of the gesture • Possible values are: • C= Close intimate, I= Intimate, P= Personal, S=Social, U= Public, R= Remote Social dimension: critical factors
- 59. • Cultural influence and interpretation • The gesture ”The ring” has four major meanings: OK/Good, Orifice, Zero, Threat Social dimension: critical factors Source: Morris, Collett, Marsh, & O’Shaghnessy (1979) • “Very tasty”: how to gesture that?
- 60. • Cultural influence and interpretation • The gesture ”The ring” has four major meanings: OK/Good, Orifice, Zero, Threat Social dimension: critical factors Source: Morris, Collett, Marsh, & O’Shaghnessy (1979) • “Very tasty”: how to gesture that?
- 61. • Cultural influence and interpretation • The gesture ”The ring” has four major meanings: OK/Good, Orifice, Zero, Threat Social dimension: critical factors Source: Morris, Collett, Marsh, & O’Shaghnessy (1979) • “Very tasty”: how to gesture that? • “Select”: how to gesture that? Italy: pointing Sweden: open hand Turkey: two open hands
- 62. • Engagement by propagation: only by careful consideration of social gestures Social dimension: critical factors Source: Jean-Yves Lionel Lawson, Jean Vanderdonckt, Radu-Daniel Vatavu: Mass-Computer Interaction for Thousands of Users and Beyond. CHI Extended Abstracts 2018 www.skemmi.com See video at https://www.youtube.com/watch?v=IZaAl59AUk8
- 63. • Social acceptance or reluctance? Social dimension: critical factors What do you think of the Itchy Nose? See video at https://www.youtube.com/watch?v=IQ_LkPM_GHs
- 64. • Social acceptance or reluctance? Social dimension: critical factors 0.319 0.246 0.232 0.225 0.203 0.185 0.181 0.178 0.167 0.167 0.167 0.244 0.231 0.141 0.218 0.179 0.103 0.218 0.218 0.103 0.141 0. 0.509 0.236 0.273 0.291 0.182 0.4 0.109 0.182 0.236 0.145 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 Start Player Go to Previous Item Turn Alarm Off Increase Volume Turn Light On Decrease Volume Turn TV Off Turn TV On Dim Light Turn Alarm On Han C Global Male Both side Double tap 2 Center tap Left push Both sides Both hands tap Right push Left to right flick Right to left flick Continuous rubbing Top push Top to bottom flick Repe cent n Agreement rate Source: Jorge Luis Pérez-Medina, Santiago Villarreal, Jean Vanderdonckt: A Gesture Elicitation Study of Nose-Based Gestures. Sensors 20(24): 7118 (2020)
- 65. • Social acceptance or reluctance? Social dimension: critical factors
- 67. • Compatibility • Imposition: each OS imposes its own set of gestures • Some are natural to use • Some others are not natural at all and remain unused • Lack of acceptability: • Some gestures can be simple to be recognized by the system, yet hard to remember and reproduce for end users • Some gestures can be accepted by end users, but harder to be recognized • Gestures are often system-defined, sometimes designer-defined, rarely not user-defined • Need for Gesture Elicitation Study (GES) • Is a study for asking end-users to elicit their own gestures for a set of predefined functions through referents and reach a consensus User experience dimension: evaluation criteria
- 68. • Compatibility • Need for a natural conceptual model: Virtual Library User experience dimension: evaluation criteria Source: https://www.youtube.com/watch?v=ls5kj7oVwto
- 69. • Consistency • Imposition: each OS imposes its own set of gestures • A few gestures are common (hopefully, natural) • Most other gestures are inconsistent User experience dimension: evaluation criteria Source: Ryan Lee, www.gesturecons.com
- 70. • Consistency: standardization? • Not really, but ongoing User experience dimension: evaluation criteria
- 71. • Consistency: standardization? • Gesture example “Shake”: Wake up, Update, Reset, Next track, Shuffle, Unlock, Enter a comment (SnappView) User experience dimension: evaluation criteria
- 72. • Discoverability • GUI interaction is based on • action exploration: eg by menu • recognition (best) • Gestures are not easy to discover • Solutions appear: feedforward User experience dimension: evaluation criteria Sources: Donald A. Norman, J. Nielsen, Gestural interfaces: a step backward in usability, Interactions, V7N5, Sept. 2010 Olivier Bau, Wendy E. Mackay, OctoPocus: a dynamic guide for learning gesture-based command sets. UIST 2008: 37-46
- 73. • Control: explicit, mixed, not implicit • “When users think they did one thing but actually did something else, they lose their sense of controlling the system because they don't understand the connection between actions and results.” User experience dimension: evaluation criteria Sources: Donald A. Norman, J. Nielsen, Gestural interfaces: a step backward in usability, Interactions, V7N5, Sept. 2010. Bert Schiettecatte, Jean Vanderdonckt, AudioCubes: a distributed cube tangible interface based on interaction range for sound design. Tangible and Embedded Interaction 2008: 3-10 Commands AudioCube action(s) Grab and set DOF=0 (discrete) DOF=1 (linearly correlated) Move cube in 2D, almost 3D, 3D
- 74. • Control: explicit, mixed, not implicit • “When users think they did one thing but actually did something else, they lose their sense of controlling the system because they don't understand the connection between actions and results.” User experience dimension: evaluation criteria Sources: Donald A. Norman, J. Nielsen, Gestural interfaces: a step backward in usability, Interactions, V7N5, Sept. 2010. Bert Schiettecatte, Jean Vanderdonckt, AudioCubes: a distributed cube tangible interface based on interaction range for sound design. Tangible and Embedded Interaction 2008: 3-10 Commands AudioCube action(s) DOF=1 (linearly correlated) Move cube in 2D, almost 3D, 3D
- 75. • Control: explicit, mixed, not implicit • “When users think they did one thing but actually did something else, they lose their sense of controlling the system because they don't understand the connection between actions and results.” User experience dimension: evaluation criteria Sources: Donald A. Norman, J. Nielsen, Gestural interfaces: a step backward in usability, Interactions, V7N5, Sept. 2010. Bert Schiettecatte, Jean Vanderdonckt, AudioCubes: a distributed cube tangible interface based on interaction range for sound design. Tangible and Embedded Interaction 2008: 3-10 Commands AudioCube action(s) DOF=1 (rotationally correlated) Rotate in 2D, 3D DOF=2 (freeform) 2D, 3D gestures
- 76. • Physical demand depends on variables • Gesture form: specifies which form of gesture is elicited. Possible values are: • S= stroke when the gesture only consists of taps and flicks • T= static when the gesture is performed in only one location • M= static with motion (when the gesture is performed with a static pose while the rest is moving) • D= dynamic when the gesture does capture any change or motion User experience dimension: evaluation criteria
- 77. • Physical demand depends on variables • Laterality: characterizes how the two hands are employed to produce gestures, with two categories, as done in many studies. Possible values are: • D= dominant unimanual, N= non-dominant unimanual, S= symmetric bimanual, A= asymmetric bimanual User experience dimension: evaluation criteria Source: https://www.tandfonline.com/doi/abs/10.1080/00222895.1987.10735426 D (right handed) N (right handed) S (right handed) A (right handed)
- 78. • Agreement among end users • Agreement Rate = the number of pairs of participants in agreement with each other divided by the total number of pairs of participants that could be in agreement • Compute co-agreement for pairs, groups (eg male vs female), categories of referents (eg basic vs. advanced) User experience dimension: evaluation criteria agreement rate disagreement rate co-agreement rate Source: Radu-Daniel Vatavu, Jacob O. Wobbrock, Between-Subjects Elicitation Studies: Formalization and Tool Support. CHI 2016: 3390-3402.
- 79. • FUN! • In games, all gestures are permitted (body) • In professional contexts, a gesture could be considered as awkward, inappropriate User experience dimension: evaluation criteria Example: MiniEurope (Alterface)
- 80. Psychological dimension Hardware dimension Software dimension Usage dimension Social dimension User experience dimension Conclusion: is it really natural?
- 81. 81 • Gesture interaction is suitable for • Natural interactions: interact directly with objects in physical way • Less cumbersome or visible hardware • Flexibility in hardware • Fun • Gesture interaction is NOT suitable for • Heavy data input (use keyboards instead) • Absence of visual feedback (e.g., a system without a screen or targeting users with visual impairments) • Unmet physical demands (e.g., swipe to receive a phone call in winter) • Constrained contexts of use (e.g., privacy, embarrassment) • User and task • Platform/device • Environment Conclusion: is it really natural?
- 82. Thank you very much for your attention
Editor's Notes
- #13: Kendon [12] added that an important part of ‘kinetics’ research shows that gesture phrases can be organized in relation to speech phrases. We can parallel his arguments and reasonings to relatively coincide with pen gestures (as natural human gestures) and the instantiated sketch-objects (as dictionised speech contents). He also stated that there is a consistent patterning in how gesture phrases are formed in relation to the phrases of speech – just as, in a continuous discourse, speakers group tone units into higher order groupings resembling a hierarchy, so gesture phrases may be similarly organized. Gestures that are put together to form phrases of bodily actions have the characteristics that permit them to be ‘recognized’ as components of willing communicative action
- #16: Designed for speech-related gestures Not completely relevant for interaction design
- #19: Gesture-based interaction without the support of speech input Tailor-made for interaction design
- #20: Gesture-based interaction without the support of speech input Tailor-made for interaction design
- #21: Gesture-based interaction without the support of speech input Tailor-made for interaction design
- #22: Gesture-based interaction without the support of speech input Tailor-made for interaction design
- #56: Chironomia is the art of using gesticulations or hand gestures to good effect in traditional rhetoric or oratory. Effective use of the hands, with or without the use of the voice, is a practice of great antiquity, which was developed and systematized by the Greeks and the Romans. Various gestures had conventionalized meanings which were commonly understood, either within certain class or professional groups, or broadly among dramatic and oratorical audiences.