![The (quadratic) letter assignment problem
Given: n letters – 𝑖, 𝑗 𝜖 𝛴
m gestures – 𝑘, 𝑙 𝜖 𝐺
Let: 𝑥𝑖𝑘 = 1 if letter 𝑖 is assigned to gesture 𝑘, 𝑥𝑖𝑘 = 0 otherwise
𝐗 = {𝑥𝑖𝑘 | ∀ 𝑖 𝜖 𝛴, 𝑘 𝜖 𝐺, 𝑥𝑖𝑘 𝜖 0,1 } characterises the full design space
[Burkhard, 1977]
Formulation](https://image.slidesharecdn.com/dagstuhloptimizationoftextinput-180122154639/85/Optimization-of-Text-Input-8-320.jpg)
![[Zhai, Hunter & Smith, 2000]
[Light & Anderson, 1993]
Formulation
[Oulasvirta & Karrenbauer, 2014]](https://image.slidesharecdn.com/dagstuhloptimizationoftextinput-180122154639/85/Optimization-of-Text-Input-10-320.jpg)
![” It is almost impossible to write correctly French
with a keyboard marketed in France”
French Ministry of Culture and Communications
[Feit, Nancel,Weir, John, Bailly,
Karrenbauer, Oulasvirta, upcoming]
Formulation](https://image.slidesharecdn.com/dagstuhloptimizationoftextinput-180122154639/85/Optimization-of-Text-Input-11-320.jpg)
![[Feit, Nancel,Weir, John, Bailly,
Karrenbauer, Oulasvirta, upcoming]
Formulation
é è à ù ê Ê É È À ç Ç æ Æ œ Œ ß ẞ þ Þ ð Ð ŋ Ŋ ij IJ ə Ə ʒ Ʒ & θ ı İ @ ™ ® ©
ſ º ª · ´ ˋ ˆ ¨ ˉ ̲ ˘ ̑ ˇ ˜ ˙ ̣ ̊ ˝ ˵ ¸ ˛ ̦ ̵ ̷ + < > = ± × ÷ ≤ ≥ ≃ % ‰ √ ∞ ¼ ½ ¾
# / | . , ; : ! ? ¡ ¿ … - - — – _ * † ‡ § ( ) [ ] { } “ ” ‘ ’ « » ‚ „ ‹ › € $ £ ¢ ¤ ¥ ₩
?? ?
> 𝟏𝟎 𝟐𝟏𝟑
𝒂𝒔𝒔𝒊𝒈𝒏𝒎𝒆𝒏𝒕𝒔](https://image.slidesharecdn.com/dagstuhloptimizationoftextinput-180122154639/85/Optimization-of-Text-Input-12-320.jpg)
![[Feit, Nancel,Weir, John, Bailly,
Karrenbauer, Oulasvirta, upcoming]
Formulation
è](https://image.slidesharecdn.com/dagstuhloptimizationoftextinput-180122154639/85/Optimization-of-Text-Input-13-320.jpg)
![Objectives
• Performance – Fitts’ law weighted by letter pair frequency
• QWERTY similarity [Dunlop & Levine 2012]
• Word or gesture clarity [Dunlop & Levine 2012, Smith, Bi & Zhai 2015]](https://image.slidesharecdn.com/dagstuhloptimizationoftextinput-180122154639/85/Optimization-of-Text-Input-14-320.jpg)
![Objectives
[How we type,
Feit, Weir, Oulasvirta, CHI 2016]](https://image.slidesharecdn.com/dagstuhloptimizationoftextinput-180122154639/85/Optimization-of-Text-Input-16-320.jpg)
![Objectives
…
|C6| = 0.38
Middle vs. Ring, participant 2046
Non-instructed:Ring
Instructed: Middle
Gesture performance models
based on Fitts’ law and theories
of motor control
Anatomical comfort:
Individuation index for each
finger
[Investigating the Dexterity of Multi-Finger Input for Mid-AirText Entry,
Sridhar, Feit,Theobalt, Oulasvirta, CHI 2015]](https://image.slidesharecdn.com/dagstuhloptimizationoftextinput-180122154639/85/Optimization-of-Text-Input-17-320.jpg)
![Objectives
[Feit, Nancel,Weir, John, Bailly,
Karrenbauer, Oulasvirta, upcoming]
Standardization committee:
”The new keyboard should facilitate typing of correct french, should be easy to learn and intuitive to use”
• Performance and ergonomics of typing a
special character before or after a letter
• Intuitive and easy to learn:
• Grouping similar characters
• Position similar to QWERTY
• Language statistics take into account different
typing tasks, e.g. programming, social media
usage, formal writing, etc.](https://image.slidesharecdn.com/dagstuhloptimizationoftextinput-180122154639/85/Optimization-of-Text-Input-18-320.jpg)
![Optimization
[Feit, Nancel,Weir, John, Bailly,
Karrenbauer, Oulasvirta, upcoming]
Mathematical solver: Gurobi
• Guarantees to cover the full
design space
• Gives explicit bounds
• Nevertheless: cannot solve to
the global optimum
Challenge: integrate
optimization with stakeholders’
opinions](https://image.slidesharecdn.com/dagstuhloptimizationoftextinput-180122154639/85/Optimization-of-Text-Input-20-320.jpg)
Optimization of Text Input
- 1. Optimization ofText Input Anna Feit, Doctoral student, Aalto University Dagstuhl Seminar on Computational Interactivity, 06.06.2017
- 3. … Mid-air hand gestures for text input
- 4. A B C … Z space ? … 27 letters32 gestures 1033 mappings Which hand gesture to use for which letter?
- 5. Decisions: For each letter and gesture, assign the letter to the gesture or not. Constraints: No letter / gesture is assigned more than once. Evaluation criteria: Evaluate each design and pick the best one. 1033 feasible designs
- 6. Mathematical or algorithmic method to find the best design in the space • Search very large design spaces • Efficient and rigorous process • Quantitative guarantees on the goodness of the outcome • Explicitly trade-off different criteria and constraints Optimization
- 7. Challenges Formulation Objectives Optimization Formulation of the design problem and space, identification of design variables and constraints argmin 𝐾 𝑘𝜖𝐾 𝑙𝜖𝐾 𝐶 𝐾(𝑘, 𝑙) Modeling of evaluation criteria and combination into a fast to compute objective function (interaction cost) Mathematical solver or approximation algorithm to efficiently and thoroughly search the design space Objective 1 Objective2
- 8. The (quadratic) letter assignment problem Given: n letters – 𝑖, 𝑗 𝜖 𝛴 m gestures – 𝑘, 𝑙 𝜖 𝐺 Let: 𝑥𝑖𝑘 = 1 if letter 𝑖 is assigned to gesture 𝑘, 𝑥𝑖𝑘 = 0 otherwise 𝐗 = {𝑥𝑖𝑘 | ∀ 𝑖 𝜖 𝛴, 𝑘 𝜖 𝐺, 𝑥𝑖𝑘 𝜖 0,1 } characterises the full design space [Burkhard, 1977] Formulation
- 9. Goal: Find the assignment of letters to gestures that minimize the cost of typing one character after another Formulation
- 10. [Zhai, Hunter & Smith, 2000] [Light & Anderson, 1993] Formulation [Oulasvirta & Karrenbauer, 2014]
- 11. ” It is almost impossible to write correctly French with a keyboard marketed in France” French Ministry of Culture and Communications [Feit, Nancel,Weir, John, Bailly, Karrenbauer, Oulasvirta, upcoming] Formulation
- 12. [Feit, Nancel,Weir, John, Bailly, Karrenbauer, Oulasvirta, upcoming] Formulation é è à ù ê Ê É È À ç Ç æ Æ œ Œ ß ẞ þ Þ ð Ð ŋ Ŋ ij IJ ə Ə ʒ Ʒ & θ ı İ @ ™ ® © ſ º ª · ´ ˋ ˆ ¨ ˉ ̲ ˘ ̑ ˇ ˜ ˙ ̣ ̊ ˝ ˵ ¸ ˛ ̦ ̵ ̷ + < > = ± × ÷ ≤ ≥ ≃ % ‰ √ ∞ ¼ ½ ¾ # / | . , ; : ! ? ¡ ¿ … - - — – _ * † ‡ § ( ) [ ] { } “ ” ‘ ’ « » ‚ „ ‹ › € $ £ ¢ ¤ ¥ ₩ ?? ? > 𝟏𝟎 𝟐𝟏𝟑 𝒂𝒔𝒔𝒊𝒈𝒏𝒎𝒆𝒏𝒕𝒔
- 13. [Feit, Nancel,Weir, John, Bailly, Karrenbauer, Oulasvirta, upcoming] Formulation è
- 14. Objectives • Performance – Fitts’ law weighted by letter pair frequency • QWERTY similarity [Dunlop & Levine 2012] • Word or gesture clarity [Dunlop & Levine 2012, Smith, Bi & Zhai 2015]
- 15. Objectives Text entry is a complex task involving cognitive and motor processes. Fast performance involves more than quickly pointing from one key to another • Different performance factors • Different tasks • Different skill levels • Different strategies
- 16. Objectives [How we type, Feit, Weir, Oulasvirta, CHI 2016]
- 17. Objectives … |C6| = 0.38 Middle vs. Ring, participant 2046 Non-instructed:Ring Instructed: Middle Gesture performance models based on Fitts’ law and theories of motor control Anatomical comfort: Individuation index for each finger [Investigating the Dexterity of Multi-Finger Input for Mid-AirText Entry, Sridhar, Feit,Theobalt, Oulasvirta, CHI 2015]
- 18. Objectives [Feit, Nancel,Weir, John, Bailly, Karrenbauer, Oulasvirta, upcoming] Standardization committee: ”The new keyboard should facilitate typing of correct french, should be easy to learn and intuitive to use” • Performance and ergonomics of typing a special character before or after a letter • Intuitive and easy to learn: • Grouping similar characters • Position similar to QWERTY • Language statistics take into account different typing tasks, e.g. programming, social media usage, formal writing, etc.
- 19. Optimization Mathematical, exact methods Linear or Integer Programming, Branch and Bound methods Pro: • Explicit bounds and guarantees on optimality • Fast solvers available, e.g. Gurobi, CPLEX (IBM) Con: • Objective function in closed mathematical form • Not so flexible (e.g. noisy input data, interactive optimization, multi-objectives, etc. ) Heuristic approximation algorithms • Simulated annealing, Genetic algorithms, Biology inspired algorithms etc. Pro: • Straightforward to implement and standard implementations available • Flexible, e.g. combine with simulation models Con: • No bounds or guarantee to find the global optimum • Potentially slow • Formulation of design space and constraints
- 20. Optimization [Feit, Nancel,Weir, John, Bailly, Karrenbauer, Oulasvirta, upcoming] Mathematical solver: Gurobi • Guarantees to cover the full design space • Gives explicit bounds • Nevertheless: cannot solve to the global optimum Challenge: integrate optimization with stakeholders’ opinions
- 21. Challenges • Multi-objective optimization: weighted sum versus pareto optimization • No ”one size fits all”: trade-off different tasks, skills, strategies, etc. or optimally adapt • Models and input data: efficient, mathematical models, noisy data
- 23. www.annafeit.de @AnnaFeit Anna Feit Doctoral student, finishing end 2017 Research topics: • Text entry • UI Optimization I also know a bit about: • User modeling • Mid-air input • Eye tracking