TapStrapGest: Elicitation and Recognition of Ring-based Multi-Finger Gestures
Download as PPTX, PDF0 likes4 views
Presentation of the ACM EICS '25 paper: https://dl.acm.org/doi/10.1145/3733047 We introduce TapStrapGest, a novel solution for customizable ring-based multi-finger gestures, encompassing the process from gesture elicitation to gesture recognition. Recognizing the growing demand for intuitive and customizable gesture interaction with fingers, TapStrapGest uses Tap Strap to enable users to perform simple and complex multi-finger gestures using smart rings. We conducted a gesture elicitation study, detailing the systematic process of soliciting and refining a custom set of user-defined ring-based finger gestures through participatory design and ergonomic considerations, including thinking time, goodness of fit, and memorization. Subsequently, we delve into the technical underpinnings of gesture recognition. We reduce the dimensionality of a dataset of 27 gesture classes from 21 to 15 by filtering, then from 15 to 5 by a Principal Component Analysis. We implement and compare four machine learning algorithms to show that a Quadratic Discriminant Analysis (precision=99.33%, recall=99.26%, and F1-score=99.26%) outperforms three other machine learning classifiers, i.e., a Linear Discriminant Analysis, a Support Vector Machines, and a Random Forest, as well as existing recognizers from the literature, to accurately recognize such gestures without the need to call for Deep Learning. Through a performance analysis, we demonstrate that TapStrapGest is a versatile and admissible solution for ring-based multi-finger gesture interaction, opening avenues for "eyes-free" or "screen-free" human-computer interaction in various domains.
![• What’s a ring-based multi-finger gestures?
TapStrapGest: Elicitation and Recognition of Ring-based Multi-Finger Gestures 4
• A “ring-based multi-finger gesture” is any movement of the device-wearing
finger(s) that causes a detectable change in the rings’ positions on a 2D
surface or in a 3D spatial system of reference centered on the device
• A Microgesture is defined as "a quick and subtle movement of the fingers".
[1][2]
[1] Way, D., & Paradiso, J. (2014). A Usability User Study Concerning Free-Hand Microgesture and Wrist-Worn Sensors. In BSN '14. IEEE Press, 138-
142.
[2] Caillet, A.C., Goguey, A., & Nigay, L. (2023). Glyph: a Microgesture Notation. In CHI '23. ACM, 269:1-269:28.3
𝜇](https://image.slidesharecdn.com/tapstrapgest-eics2025-presentation-250716070310-c0ed13fc/85/TapStrapGest-Elicitation-and-Recognition-of-Ring-based-Multi-Finger-Gestures-4-320.jpg)
![TapStrapGest: Elicitation and Recognition of Ring-based Multi-Finger Gestures
• TapStrap2 Device (2019)[3]
• Wearable device
• Rings equipped with accelerometers
• Input controller
• Bluetooth device
• Works on any surface or in mid-air
[3] TapStrap2, Tap https://www.tapwithus.com/product/tap-strap-2
Power
button
Haptic feedback unit
3D reference system
Optical
mouse
Multi-finger adjustable rings
Accelerometers
6
1
Introducing TAP Strap 2 (https://www.youtube.com/watch?v=Cr2grRn3ru4)](https://image.slidesharecdn.com/tapstrapgest-eics2025-presentation-250716070310-c0ed13fc/85/TapStrapGest-Elicitation-and-Recognition-of-Ring-based-Multi-Finger-Gestures-6-320.jpg)
![TapStrapGest: Elicitation and Recognition of Ring-based Multi-Finger Gestures 8
• Gesture Elicitation Study (GES)
Experiment:
• Gesture elicitation study based on original
methodology [4]
• Eighteen referents from UI "function“.
Participants:
• Thirty participants (14 females, 16 males).
• Participants’ average age is 33.4 (from 13 to 63
Years).
• Voluntary participants with different occupations.
• All participants were new to the device.
Apparatus:
• The experiment took place in a controlled environment.
• Showing referents to the participant.
• Measuring Thinking-Time for each gesture.
• Recording the gesture and collecting data.
• Ask participant to rate the rank, goodness-of-fit, complexity, memorability,
and fun.
[4] Vatavu, R. D. and Wobbrock, J. O. (2015). Formalizing Agreement Analysis for Elicitation Studies: New Measures, Significance Test, and Toolkit. In Proc. of CHI ’15. 1325–1334.](https://image.slidesharecdn.com/tapstrapgest-eics2025-presentation-250716070310-c0ed13fc/85/TapStrapGest-Elicitation-and-Recognition-of-Ring-based-Multi-Finger-Gestures-8-320.jpg)
![TapStrapGest: Elicitation and Recognition of Ring-based Multi-Finger Gestures 9
• Gesture Elicitation Study Results
Low
Moderate
High
Very
high
Referent (r)
Zoom in
Zoom out
Take a call
Hang up a call
Go to next item in a list
Play
Scroll up
Scroll down
Go to previous item in a list
Turn up the volume
Close window
Take a screencapture
Pause
Turn down the volume
Switch window
Open window
Lock screen
Unlock screen
Average
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8
Agreement rate AR(r) in [0..1] Thinking-TimeGoodness-of-Fit Consensus gesture
4.22 3.49 8.83 1.57 Pinch out
3.24 1.97 8.47 1.69 Pinch in
5.13 4.44 8.10 2.01 Vertical span
5.44 5.43 7.50 1.84 Horizontal span
7.28 5.43 8.10 1.72 Swipe left
7.84 10.40 6.67 1.76 Point index
5.77 5.76 8.43 1.56 Swipe up
3.94 2.74 8.40 2.18 Swipe down
8.33 8.75 7.07 2.14 Swipe right
5.25 5.38 8.07 1.61 Swipe up
6.48 5.66 6.63 2.09 Pinch in all fingers
10.11 10.62 7.43 1.76 Push forward
5.39 4.68 7.23 1.63 Raise fingers
5.43 6.89 8.53 1.41 Swipe down
14.29 14.24 6.57 2.53 Swipe left
14.40 11.45 6.20 2.01 Pinch out
8.34 7.49 6.57 2.19 Clockwise wrist rotat.
8.53 5.82 7.17 1.95 Counter-clockwise rot.
Agreement magnitude
Icon
7.18
8.02
7.48
2.06
b
Pinch in all fingers for
“Close window”
a
Index single tap in the air for
“Open window”
c
Clockwise wrist rotation for
“Lock screen”
d
Counter-clockwise wrist rotation for
“Unlock screen”
e
Push forward for “Capture
screen”
Pinch out two fingers for
“Zoom in”
k
Pinch in two fingers for
“Zoom out”
l
Swipe up for “Turn volume
up”
m
Swipe backward for “Turn
volume down”
n
Raise fingers for “Pause”
o
Swipe right for “Next
screen”
Swipe left for “Previous
screen”
f g
Create a vertical span for
“Take call”
h
Create a horizontal span
for “Hang call”
i j
Point index for “Play”
j](https://image.slidesharecdn.com/tapstrapgest-eics2025-presentation-250716070310-c0ed13fc/85/TapStrapGest-Elicitation-and-Recognition-of-Ring-based-Multi-Finger-Gestures-9-320.jpg)
![TapStrapGest: Elicitation and Recognition of Ring-based Multi-Finger Gestures 15
• Results & Performance Analysis
• Comparative testing of the recognizers
LDA QDA SVM RF
0.90
0.95
1.00
0.
9
9
11
0.
9
867
0.
9
933
0.
9
770
Recognizer [name]
Precision
[0..1]
LDA QDA SVM RF
0.90
0.95
1.00
0
.99
04
0
.99
15
0
.99
26
0
.97
52
Recall
[0..1]
Recognizer [name]
LDA QDA SVM RF
0.90
0.95
1.00
0.9
8
96
0.9
8
81
0.9
9
26
0.9
7
41
Recognizer [name]
F1-Score
[0..1]
**
*](https://image.slidesharecdn.com/tapstrapgest-eics2025-presentation-250716070310-c0ed13fc/85/TapStrapGest-Elicitation-and-Recognition-of-Ring-based-Multi-Finger-Gestures-15-320.jpg)
TapStrapGest: Elicitation and Recognition of Ring-based Multi-Finger Gestures
- 1. TapStrapGest: Elicitation and Recognition of Ring-based Multi-Finger Gestures GUILLEM CORNELLA, EUDALD SANGENIS, MEHDI OUSMER, JEAN VANDERDONCKT, SANTIAGO VILLARREAL- NARVAEZ, BRUNO DUMAS, ADRIEN CHAFFANGEON CAILLET 17th ACM SIGCHI Symposium on Engineering Interactive Computing Systems, EICS ‘25, 23-27 June 2025, Trier, Germany
- 2. Adrien Chaffageon Caillet Berlin University of Applied Sciences and Technology, Germany/ Univ. Grenoble Alpes, CNRS, Grenoble INP, LIG, France Guillem Cornella University of California at Irvine, USA Eudald Sangenis University of California at Irvine, USA Mehdi Ousmer LouRIM Université catholique de Louvain, Belgium Santiago Villarreal Narvaez University of Namur, Belgium Bruno Dumas University of Namur, Belgium Jean Vanderdonckt LouRIM Université catholique de Louvain, Belgium TapStrapGest: Elicitation and Recognition of Ring-based Multi-Finger Gestures 2
- 3. • How can we design effective ring-based multi-finger gesture interaction? • Key challenges: • Inconsistent definitions of multi-finger gestures across literature • Device variability in different contexts of use • Hardware and software limitations of existing devices • Inconsistent support across different gesture recognizers • Lack sufficient consideration of recognition effectiveness and efficiency • Insufficient guidance for researchers and developers TapStrapGest: Elicitation and Recognition of Ring-based Multi-Finger Gestures 3
- 4. • What’s a ring-based multi-finger gestures? TapStrapGest: Elicitation and Recognition of Ring-based Multi-Finger Gestures 4 • A “ring-based multi-finger gesture” is any movement of the device-wearing finger(s) that causes a detectable change in the rings’ positions on a 2D surface or in a 3D spatial system of reference centered on the device • A Microgesture is defined as "a quick and subtle movement of the fingers". [1][2] [1] Way, D., & Paradiso, J. (2014). A Usability User Study Concerning Free-Hand Microgesture and Wrist-Worn Sensors. In BSN '14. IEEE Press, 138- 142. [2] Caillet, A.C., Goguey, A., & Nigay, L. (2023). Glyph: a Microgesture Notation. In CHI '23. ACM, 269:1-269:28.3 𝜇
- 5. TapStrapGest: Elicitation and Recognition of Ring-based Multi-Finger Gestures 5 • Multi-finger Gesture Input Devices "Leap Motion @Game Jam" by qubodup is licensed under CC BY 2.0.
- 6. TapStrapGest: Elicitation and Recognition of Ring-based Multi-Finger Gestures • TapStrap2 Device (2019)[3] • Wearable device • Rings equipped with accelerometers • Input controller • Bluetooth device • Works on any surface or in mid-air [3] TapStrap2, Tap https://www.tapwithus.com/product/tap-strap-2 Power button Haptic feedback unit 3D reference system Optical mouse Multi-finger adjustable rings Accelerometers 6 1 Introducing TAP Strap 2 (https://www.youtube.com/watch?v=Cr2grRn3ru4)
- 7. • TapStrap2 Limitations • Physical Comfort Issues • Accessibility Barriers • Technical Limitations TapStrapGest: Elicitation and Recognition of Ring-based Multi-Finger Gestures 7 TapStrap2 System-defined Gesture vocabulary
- 8. TapStrapGest: Elicitation and Recognition of Ring-based Multi-Finger Gestures 8 • Gesture Elicitation Study (GES) Experiment: • Gesture elicitation study based on original methodology [4] • Eighteen referents from UI "function“. Participants: • Thirty participants (14 females, 16 males). • Participants’ average age is 33.4 (from 13 to 63 Years). • Voluntary participants with different occupations. • All participants were new to the device. Apparatus: • The experiment took place in a controlled environment. • Showing referents to the participant. • Measuring Thinking-Time for each gesture. • Recording the gesture and collecting data. • Ask participant to rate the rank, goodness-of-fit, complexity, memorability, and fun. [4] Vatavu, R. D. and Wobbrock, J. O. (2015). Formalizing Agreement Analysis for Elicitation Studies: New Measures, Significance Test, and Toolkit. In Proc. of CHI ’15. 1325–1334.
- 9. TapStrapGest: Elicitation and Recognition of Ring-based Multi-Finger Gestures 9 • Gesture Elicitation Study Results Low Moderate High Very high Referent (r) Zoom in Zoom out Take a call Hang up a call Go to next item in a list Play Scroll up Scroll down Go to previous item in a list Turn up the volume Close window Take a screencapture Pause Turn down the volume Switch window Open window Lock screen Unlock screen Average 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 Agreement rate AR(r) in [0..1] Thinking-TimeGoodness-of-Fit Consensus gesture 4.22 3.49 8.83 1.57 Pinch out 3.24 1.97 8.47 1.69 Pinch in 5.13 4.44 8.10 2.01 Vertical span 5.44 5.43 7.50 1.84 Horizontal span 7.28 5.43 8.10 1.72 Swipe left 7.84 10.40 6.67 1.76 Point index 5.77 5.76 8.43 1.56 Swipe up 3.94 2.74 8.40 2.18 Swipe down 8.33 8.75 7.07 2.14 Swipe right 5.25 5.38 8.07 1.61 Swipe up 6.48 5.66 6.63 2.09 Pinch in all fingers 10.11 10.62 7.43 1.76 Push forward 5.39 4.68 7.23 1.63 Raise fingers 5.43 6.89 8.53 1.41 Swipe down 14.29 14.24 6.57 2.53 Swipe left 14.40 11.45 6.20 2.01 Pinch out 8.34 7.49 6.57 2.19 Clockwise wrist rotat. 8.53 5.82 7.17 1.95 Counter-clockwise rot. Agreement magnitude Icon 7.18 8.02 7.48 2.06 b Pinch in all fingers for “Close window” a Index single tap in the air for “Open window” c Clockwise wrist rotation for “Lock screen” d Counter-clockwise wrist rotation for “Unlock screen” e Push forward for “Capture screen” Pinch out two fingers for “Zoom in” k Pinch in two fingers for “Zoom out” l Swipe up for “Turn volume up” m Swipe backward for “Turn volume down” n Raise fingers for “Pause” o Swipe right for “Next screen” Swipe left for “Previous screen” f g Create a vertical span for “Take call” h Create a horizontal span for “Hang call” i j Point index for “Play” j
- 10. TapStrapGest: Elicitation and Recognition of Ring-based Multi-Finger Gestures 10 • Gesture Classification Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 Q9 Q10 Q11 Q12 Q13 Q14 Q15 Q16 0 5 10 15 20 25 30 Distribution of scores per question I strongly disagree I disagree I am so so I agree I strongly agree System Usefulness (SysUse: 1-6) Information Quality (InfoQual: 7-12) Interface Quality (InterQual: 13-15) Overall Satisfaction (Overall: 1-16) 0 1 2 3 4 5 6 7 5.26 4.60 5.40 5.20 IBM PSSUQ Measures
- 11. TapStrapGest: Elicitation and Recognition of Ring-based Multi-Finger Gestures 11 • Gesture Recognition • Data reduction and Gesture Set • Filtering gesture Set Data (only Accelerometer values) • Dimensionality reduction with Principal Component Analysis (PCA) • Four gesture recognizers: ‐ Linear Discriminant Analysis (LDA) ‐ Quadratic Discriminant Analysis (QDA) ‐ Support Vector Machine (SVM) ‐ Random Forest (RF)
- 12. TapStrapGest: Elicitation and Recognition of Ring-based Multi-Finger Gestures 12 • Data acquisition and set of gestures
- 13. TapStrapGest: Elicitation and Recognition of Ring-based Multi-Finger Gestures 13 • Data reduction and Gesture Set
- 14. TapStrapGest: Elicitation and Recognition of Ring-based Multi-Finger Gestures 14 • Testing procedure ( https://github.com/gcornella/TapStrap2-Hand-Gesture-Recognition-)
- 15. TapStrapGest: Elicitation and Recognition of Ring-based Multi-Finger Gestures 15 • Results & Performance Analysis • Comparative testing of the recognizers LDA QDA SVM RF 0.90 0.95 1.00 0. 9 9 11 0. 9 867 0. 9 933 0. 9 770 Recognizer [name] Precision [0..1] LDA QDA SVM RF 0.90 0.95 1.00 0 .99 04 0 .99 15 0 .99 26 0 .97 52 Recall [0..1] Recognizer [name] LDA QDA SVM RF 0.90 0.95 1.00 0.9 8 96 0.9 8 81 0.9 9 26 0.9 7 41 Recognizer [name] F1-Score [0..1] ** *
- 16. TapStrapGest: Elicitation and Recognition of Ring-based Multi-Finger Gestures 16 • Real-time gesture recognition application
- 17. TapStrapGest: Elicitation and Recognition of Ring-based Multi-Finger Gestures 17 • Design Implications • Suggested implications for designing ring-based multi-finger gesture interfaces Support gesture customizability Finger independence Gesture consistency Feedback and confirmation Customization Accessibility considerations
- 18. TapStrapGest: Elicitation and Recognition of Ring-based Multi-Finger Gestures 18 • Formalizing Ring-Based Multi-Finger Gestures • µGlyph (Chaffangeon Caillet et al., 2023) • Formal notation for microgestures description • Bio-mechanical finger movements • Surface contact information • One- or two-handed interaction (a) (b) (c) (d)
- 19. TapStrapGest: Elicitation and Recognition of Ring-based Multi-Finger Gestures 19 • µGlyph Limitation and Extension • µGlyph only captures finger movements • Cannot represent wrist or arm gestures (e) (f) (g)
- 20. • Limitations • Device constraints limit the range and variety of interaction • Limited comparison data • The notation system cannot capture all gesture types • Advantages • Expanded ring-based device capabilities • A comprehensive solution for gesture interaction • Formalized a wide range of multi-finger gestures TapStrapGest: Elicitation and Recognition of Ring-based Multi-Finger Gestures 20
- 21. Thank you very much for your attention