Enhancing the interaction space of a tabletop computing system to design paper prototypes for mobile applications

Enhancing the interaction space
of a tabletop computing system
to design paper prototypes
for mobile applications
Master Thesis
Francesco Bonadiman
2016

Blended Prototyping
● tabletop system based on hand-drawn paper sketches
2

Blended Prototyping
○ converted into digital versions
→ projected on the table
→ expanded further
2

○ transformed into testable applications
→ on a mobile device
Blended Prototyping
2

○ transformed into testable applications
→ on a mobile device
● enhances development of mobile applications
○ accelerates early design phases
Blended Prototyping
2

Hardware
● video projector
○ projects mobile frames & prototypes
3

Hardware
● video projector
● webcam
○ recognizes barcodes of screens
3

Hardware
● video projector
● webcam
● DSLR camera
○ shoots HQ pics of the sketches
3

Hardware
● video projector
● webcam
● DSLR camera
○ shoots HQ pics of the sketches
● tablet
○ allows to perform actions on the prototypes
3

Software
● Java application
○ controls behavior of projector & cameras
4

Software
○ identifies barcode markers on paper sheets
4

Software
○ digitize sketches & perform actions on them
4

Software
○ digitize sketches & perform actions on them
● Barcode (similar to QR-code)
○ wider → optimized for webcam
○ gives sheet’s position & rotation on table
4

Advantages of Paper Prototyping
● potential of Paper-Prototyping & mobile devices = combined
6

● first impression of not-yet-developed product
6

● paper → cheap, fast & intuitive (Snyder)
○ no unimportant details → iterative refinement (Nielsen)
6

○ quickly create multiple design alternatives (Landay)
6

○ quickly create multiple design alternatives (Landay)
● same usability issues as Hi-Fi discovered
○ benefits of early usability data = 10+ times bigger (Snyder)
6

● encourages collaboration & critiques →
interdisciplinary & creative (no “gaps”)
Advantages of the System
7

● testing within real-life scenarios (de Sá / Carriço)
→ in thousand of different usage conditions
7

● testing within real-life scenarios (de Sá / Carriço)
→ in thousand of different usage conditions
● add code & functionalities
○ define dynamic interface behavior
○ smooth transition to development
7

Core features
Via TABLET → Determine widgets & semantics
8

Core features
● manually define “hotspots” on prototype
● turn these into different design patterns
8

Core features
● create links between the prototypes
● remove components & connections
8

Core features
● create links between the prototypes
● remove components & connections
● convert into working code
8

Problem
● several devices + media → different degrees of fidelity
○ paper & office supplies → computers & smart objects
9

Problem
BUT
9

Problem
BUT
Changing fidelity of tool & modality of interaction
→ disrupts creative design process
9

Problem
BUT
Changing fidelity of tool & modality of interaction
→ disrupts creative design process
→ shifting continuously confuses users
9

Fidelity Clash
● combination of low- & high-tech solutions = puzzling
10

Fidelity Clash
○ one single user stops ideation process
→ to perform any action
10

Fidelity Clash
○ breaking of collaborative & creative moment (Snyder)
10

Fidelity Clash
● users distracted → do not interact anymore
○ get unfocused → lose the “flow”
10

Fidelity Clash
● users distracted → do not interact anymore
○ get unfocused → lose the “flow”
→ perceived as too technical, isolating & distracting
10

Objectives
● replace high-tech interactions with low-tech approaches
11

Objectives
● new interaction techniques → not to shift fidelity of media
11

Objectives
THEN
11

Objectives
THEN
● find & implement alternative solutions to tablet
11

Objectives
THEN
● find & implement alternative solutions to tablet
● low-tech approaches to define “hotspots” & perform actions
○ keep interaction techniques learnable & usable
11

Tasks
1. digitize a screen by taking a picture
13

Tasks
2. duplicate a whole screen
13

Tasks
3. detect a component (button, image or textbox)
13

Tasks
4. connect two screens (link a button to next screen)
13

Tasks
5. remove a component from a screen
13

Tasks
6. remove a connection between two screens
13

Tasks
6. remove a connection between two screens
7. remove a whole screen
13

Alternatives considered
● Voice recognition
● Object recognition
● Gesture recognition
● Special pen
● Special button
● Colored objects
● Barcode recognition
● Transparent layers
14

Alternatives considered
● Voice recognition
● Object recognition
● Gesture recognition
● Special pen
● Special button
● Colored objects
● Barcode recognition
● Transparent layers
● most approaches = unfeasible
○ noisy / confusing environment
○ need of not-yet-existing hardware
○ time constraints
14

Color
Detection
A
Barcode
Recognition
B
15

A. Color Detection
● no specific hardware
● dynamic & “playful” interaction
16

A. Color Detection
Concept: different color = different component
16

A. Color Detection
→ colored markers + semi-transparent BBPapier
→ users can see through without “ruining” sketches
16

A. Color Detection
→ colored markers + semi-transparent BBPapier
→ users can see through without “ruining” sketches
→ JavaCV & OpenCV
16

→ Procedure
● control card = “toolbox” → paint with markers
● same color → fill up the chosen component
17

→ Procedure
● digitize the sketch → creating threshold
○ average color in toolbox → range (+30/-30)
○ exterior pixels discarded → central values
→ avoid mistakes & colors overlapping
17

→ Procedure
● digitize the sketch → creating threshold
○ average color in toolbox → range (+30/-30)
○ exterior pixels discarded → central values
→ avoid mistakes & colors overlapping
● threshold = calculate component’s RGB value
● any pixel within → associated to specific UI component
17

→ Approximation
Once pixels & colors detected
● algorithm recognizes contours of shape
● approximates it to a rectangle
● component created & projected
18

→ Approximation
Once pixels & colors detected
● algorithm recognizes contours of shape
● approximates it to a rectangle
● component created & projected
Need for filling component
● algorithm not efficient only for outline
● rough drawings typical of sketching
18

B. Barcode Recognition
Barcode marker = tool for specific operations
19

● take pictures of device
● connect screens
● recognize & delete components
● copy screens
19

● take pictures of device
● connect screens
● recognize & delete components
● copy screens
Algorithm continuously running
→ checks if new barcodes are detected
19

Further improvements
Sidebar → alternative approach for some functionalities
21

● rectangular transparent area + at top of table
● if barcode placed inside → operation performed
21

Clock (progress indicator) → added onto center-top area
21

● immediate visual feedback to users
● if appears → users realize they are performing an action
21

● immediate visual feedback to users
● if appears → users realize they are performing an action
● 3 seconds = time-frame users might change their mind
→ effective to avoid accidental mistakes
21

Modality A: Sidebar
● digitize a screen → taking a picture
○ by placing single barcode inside
23

Modality A: Sidebar
● digitize a screen → taking a picture
○ by placing single barcode inside
● duplicate a screen
○ by placing two barcodes inside
○ one is already digitized (source)
○ one is empty (destination)
23

Modality A: Colors
● toolbox projected
→ boxes for button, image, textbox…
24

Modality A: Colors
● BBPapier over toolbox → paint with color
● same color → BBPapier over component
24

Modality A: Colors
● BBPapier over toolbox → paint with color
● same color → BBPapier over component
● digitize screen with toolbox visible on table
→ colored square displayed to represent component
24

Modality A: Colors
To connect two screens
● same + use FROM → TO box
To delete component → digitize again
● physically remove colored BBPapier
● or cover it with white paper
To remove connection
● same with button-connector
25

Modality B
● camera tool to digitize
● copy tool to duplicate
● handles to detect component
● arrow tool to connect screens
● rubber tool to delete
○ component
○ connection
○ screen (only way)
26

Dependent Variables
● measure overall success of new interaction techniques
○ if quick & easy-to-use without distracting user
○ which of the two modalities = most effective
27

Dependent Variables
● prove if efficient & offers satisfactory user-experience
27

Dependent Variables
● prove if efficient & offers satisfactory user-experience
4 parameters:
→ Quickness → Ease-of-use → Distraction → User-Experience
27

Quickness & Ease-of-Use
Quickness
● time for task completion
● evaluation is videotaped
28

Quickness
● Effective Time
○ duration of task
→ without system errors
→ avoids malfunctionings
28

Quickness
● Effective Time
Ease-of-use
● % successfully completed tasks
● ratio calculated by checking
interactions executed perfectly
28

Quickness
● Effective Time
Ease-of-use
● % successfully completed tasks
● ratio calculated by checking
interactions executed perfectly
○ users make no mistakes
○ no significant problems
○ not need any help / hint
28

Distraction
Two factors: quickness & workload index (RTLX)
● unweighted (Raw) version of NASA Task Load Index (TLX)
● 6 subscales:
○ Mental, Physical, Temporal Demand; Performance; Effort; Frustration
29

Distraction
Two factors: quickness & workload index (RTLX)
● unweighted (Raw) version of NASA Task Load Index (TLX)
● 6 subscales:
○ Mental, Physical, Temporal Demand; Performance; Effort; Frustration
● own survey → filled in by users after every task
● 7-points linear scales → then averaged = RTLX
● the lower → the less demanding & distracting task is
29

User-Experience
● dimensions of AttrakDiff Survey → studies by Hassenzahl
○ “how users rate the usability and design of your interactive product”
30

User-Experience
● dimensions of AttrakDiff Survey → studies by Hassenzahl
○ “how users rate the usability and design of your interactive product”
● own custom version → filled in after finishing whole study
● 28 7-points semantic differential scales
○ opposite adjectives at both poles ("good - bad" / "human - technical")
○ implicitly divided into 4 dimensions (Pragmatic Quality, Hedonic
Quality - Identity, Hedonic Quality - Stimulation, Attractiveness)
30

Experiment Design
● two different design solutions → A/B Testing
● “within-subject” design → every user tests both versions
○ no interactions in common → no influence
32

Experiment Design
● two different design solutions → A/B Testing
● “within-subject” design → every user tests both versions
○ no interactions in common → no influence
● to avoid possible bias = AB & BA groups
32

User Demographics
● one user at a time → focus on interactions
○ easier to observe & more difficult to get biased
33

User Demographics
● one user at a time → focus on interactions
○ easier to observe & more difficult to get biased
● 24 participants → 11 females & 13 males
● half aged 18-24, 10 aged 25-34, two older
● 7 background in HCI, 8 IT, rest other expertise
● via Facebook groups & personal connections
● reward = tasty gift or house utensils
33

Results
● reviewed videos from GoPro
● investigate how long took users to complete tasks
● detect when they had troubles & needed help
35

Results
● reviewed videos from GoPro
● investigate how long took users to complete tasks
● detect when they had troubles & needed help
● SPSS → ONE-WAY ANOVA test
○ ideal for type of Dependent Variables (continuous)
○ valid for any group of users of study
○ robust to violations in underlying assumptions
35

Quickness
● Modality B faster than Modality A
○ time for A = 40.1s, B only 23.25s
○ statistically significant (p < .05)
36

Quickness
● Effective Time (without system errors)
36

Quickness
● Effective Time (without system errors)
→ color detection = long process
36

Ease-of-use
● Modality A easier to use than B
37

Ease-of-use
● Modality A easier to use than B
● perfectly accomplished tasks:
○ ratio: A = 84%, B = 60%
37

Distraction
● A almost as demanding as B
○ A RTLX index = 1.90
○ B RTLX index = 1.84
38

Distraction
● A almost as demanding as B
○ A RTLX index = 1.90
○ B RTLX index = 1.84
● On a 7-points scale →
not statistically significant
(F1,286 = 0.289, p = 0.591)
38

User-Experience
● users feel “assisted by the product”
○ usability can be improved
39

User-Experience
● users are “stimulated by the product”
○ users can identify with it
○ room for hedonic improvement
39

User-Experience
● users are “stimulated by the product”
○ users can identify with it
○ room for hedonic improvement
● system considered as “rather desired”
○ attractiveness above average
39

Task 1 Analysis
● sidebar (A) better
○ faster
○ easier to use
○ less demanding
41

Task 1 Analysis
● sidebar (A) better
○ faster
○ easier to use
○ less demanding
● camera tool slower
○ no clue on how /
where to place it
41

Task 2 Analysis
● sidebar (A)
○ easier to use
42

Task 2 Analysis
● sidebar (A)
○ easier to use
○ faster (only E.T.)
○ causes several
system errors →
see later
42

Task 2 Analysis
● sidebar (A)
○ easier to use
○ causes several
system errors →
see later
● RTLX == copy tool
42

Task 3 Analysis
● handles (B)
○ faster
○ less demanding
○ easier to use
43

Task 3 Analysis
● handles (B)
○ faster
○ less demanding
○ easier to use
● color detection
○ 3 tasks in one
○ digitize + create
43

Task 4 Analysis
● arrow (B)
○ faster
○ less demanding
44

Task 4 Analysis
● arrow (B)
○ faster
○ less demanding
○ where?
44

Task 4 Analysis
● arrow (B)
○ faster
○ less demanding
○ where?
● color detection
○ easier to use
○ create + connect
44

Task 5 Analysis
● rubber (B)
45

Task 5 Analysis
● rubber (B)
○ bug found!
45

Task 5 Analysis
● rubber (B)
○ bug found!
● color detection (A)
○ easier to use
○ less demanding
○ more accurate
45

Task 6 Analysis
● rubber (B)
○ faster
○ less demanding
46

Task 6 Analysis
● rubber (B)
○ faster
○ less demanding
○ easier to use
○ experience?
46

Interesting facts
● people with background in HCI
→ performed more easily & faster
● good sketching abilities
→ same + found system more attractive
47

Interesting facts
● people familiar with mobile devices
→ faster + system less distracting
47

Interesting facts
● people familiar with mobile devices
→ faster + system less distracting
● during day-time → better performance & lower RTLX
47

Considerations
● accuracy = ratio of task’s precision
○ global accuracy = 86%
○ color detection = 67%
→ more delicate + error-prone
48

Considerations
● barcodes + hands into digitized image → distraction + inexperience
48

Considerations
● barcodes + hands into digitized image → distraction + inexperience
● if BBPapier wavy → grey area projected
○ projector’s brightness → shadow (“ghost shape”)
48

Projector’s beam
● when toolbox at sides of projection → outer areas appear darker
○ either no color is detected
○ “fake” grey color recognized → multiple components created
49

Projector’s beam
● most system errors
○ excessively intense light causing reflections on table
49

Projector’s beam
→ closest barcodes could not be detected
→ devices continuously refreshing
49

Projector’s beam
→ closest barcodes could not be detected
→ devices continuously refreshing
○ “effective time” → task completed if interaction is correct
49

Which better?
○ easier to use
○ takes longer time
51

Which better?
○ easier to use
● barcode recognition (B)
○ definitely faster
○ harder to understand
51

Which better?
○ easier to use
● barcode recognition (B)
○ definitely faster
○ harder to understand
● both have similar workload
○ around 1.9 out of 7
○ positively low →
not too distracting
51

Ideal implementation
1-2. sidebar → digitize & duplicate screens
52

2. copy tool too → duplicate without moving papers
52

3-4. both approaches to detect components & create connections
a. color detection = from scratch + multiple tasks at same time
52

b. handles & arrow = fast & easy-to-use
52

b. handles & arrow = fast & easy-to-use
5-6-7. rubber tool to remove components, connections & screens
→ implement separate ones to avoid possible mistakes
52

Future Work
● visual feedback when projections disappear from table
53

Future Work
● icon near clock → tells which action is being performed
53

Future Work
● exploring color detection further
○ better algorithms to calculate the threshold
○ better ways to detect shapes through contours
53

Future Work
● barcodes = built with wood / opaque plastic → reduce reflections
53

Future Work
● compare interaction techniques with original system
53

Future Work
● compare interaction techniques with original system
● test system in real-life scenario → inside company or startup
53

Thanks!
Any questions?
images from pixabay.com & papers by Bähr

Enhancing the interaction space of a tabletop computing system to design paper prototypes for mobile applications

More Related Content

Viewers also liked (20)

Similar to Enhancing the interaction space of a tabletop computing system to design paper prototypes for mobile applications (20)

More from Francesco Bonadiman (13)

Recently uploaded (20)

Enhancing the interaction space of a tabletop computing system to design paper prototypes for mobile applications