Model-Based User Interface Optimization: Part III: SOLVING REAL PROBLEMS - At SICSA Summer School on Computational Interaction 2015

Optimization for User Interface Design
Antti Oulasvirta, Anna Feit
Aalto University
SICSA SUMMER SCHOOL ON COMPUTATIONAL INTERACTION
GLASGOW 
JUNE 23, 2015
SOLVING REAL PROBLEMS

GOALS
• Learn to formulate your user interface design
problems as an optimization task

• Get more acquainted with assignment
problems:A common problem type in interface
design

• Appreciate the nature of common design
problems

CONTENTS
• Assignment problems

• Linear assignment, quadratic assignment

• Linear menu, hierarchical menu,  
menus with groups

• Other problem types in HCI

• Transfer

• Functionality selection

• Notiﬁcation scheduling
5 “mini exercises”

YOU WILL NEED
• Pen & paper

• Your group

WHY PROBLEM FORMULATION 
IS VITAL
• If you don’t do it,“optimization” does not make sense

• There’s no way you can avoid it: Your code does it implicitly

• You can find links to known problems and exploit their
solutions

• You can find efficient re-formulations

• Currently, this is the single most important obstacle for
optimization approaches to user interfaces

• It challenges present notions of what user interface design is

Pay attention and take
notes!You will apply
these deﬁnitions in a
minute

LINEAR ASSIGNMENT
• Assign n elements to m slots

• Often n = m
…Elements
…Slots

DEFINITION
• Assign agents to tasks

• Find a bijection with minimum total cost

• Assuming real-valued cost matrix C:
min
X
a2A
Ca,f(a)
f : A ! T

AS A LINEAR PROGRAM
An agent can only be
assigned once
Each task can hold only
one agent
min
X
i2A
X
j2T
C(i, j)xij
s.t.
X
j2T
xij = 1 for i 2 A,
X
i2A
xij = 1 for j 2 T

QUADRATIC ASSIGNMENT
• Assign n elements to m slots on a rectangle-sized
surface. Often n = m
…Elements
…Slots

DEFINITION
• Given P facilities and L locations, a weight function w,
and a distance function d, ﬁnd the bijection 
that minimizes  
 
 
 
 
Intuition:We minimize average distance among
facilities.We try to “pack” together the facilities that
are commonly used with each other
f : P ! L
X
a,b2P
wa,bdf(a),f(b)
w weight

d distance

FORMULATION WITH BINARY
DECISION VARIABLES
• Identify a permulation matrix X of dimensions  
n x n such that

!
• Observations:

• Large problems (n > 30) may be impossible

• Generalization of TSP and therefore NP-complete
min
nX
i,j=1
nX
h,k=1
dihfjkxijxhk
f frequency

d distance

IN-CLASS DISCUSSION
Which of the following involve an assignment problem?
Graphical user interfaces
Consumer electronics
Automotive interfaces
Keyboards
Web user interfaces
Gestural interaction Mobile interfaces
Dialogue interfaces

APPLICATION: MENU SYSTEMS
“Rectangles assigned to
locations on a larger
rectangle”

THE SIMPLEST MENU
slot 1!
slot 2!
…!
slot n
One-dimensional (linear) menu  
a.k.a. array / row facility layout problem
commands

DEFINE THIS DESIGN TASK
Given a frequency distribution for n commands, ﬁnd an
assignment of commands to menu such that it  
minimizes expected search+selection time
Command frequencies
command!
command!
…!
command
Menu
Command
Frequency
You have to make an assumption
about search+selection time

ONE SOLUTION
• Can be solved with the linear program
min
X
i2A
X
j2T
C(i, j)xij
s.t.
X
j2T
xij = 1 for i 2 A,
X
i2A
xij = 1 for j 2 T
Probability x Search time 
Search time from the SDP model  
Cockburn et al. CHI’07]
election times of each item i weighted by their probability
T =
n
Â
i=1
piTi
Selection time Ti of item i is given by
Ti = Tdsi + Tpi
Pointing time Tpi is governed by the Mackenzie varian
aw and is
Tpi = ap + bplog2(Ai/Wi + 1)
where Ai is movement amplitude and Wi is height of item
ically organized menus). ap and bp are empirically deter
i=1
Selection time Ti of item i is given by
Ti = Tdsi + Tpi
Pointing time Tpi is governed by the Mackenzie
law and is
where Ai is movement amplitude and Wi is height
tically organized menus). ap and bp are empirically
constants.
Decision/Search time Tdsi is linear interpolation
and visual search:
law and is
where Ai is movement amplitude and Wi is height of
tically organized menus). ap and bp are empirically d
constants.
Decision/Search time Tdsi is linear interpolation b
and visual search:
Tdsi = (1 ei)Tvsi + eiThhi
where ei[0..1] is an expertise factor ei = 1 1/t i w
number of repetitions with i. As the user becomes m
with a menu, there is a shift from serial visual search
among competing elements Thhi.
and visual search:
Tdsi = (1 ei)Tvsi + eiThhi
where ei[0..1] is an expertise factor ei = 1 1/t i where
number of repetitions with i. As the user becomes more ex
with a menu, there is a shift from serial visual search Tvsi t
Visual search time Tvsi is assumed to be linear function
number of n:
Tvsi = bvsn + avs
Hick-Hyman decision time Thhi is dependent on the entrop
item Hi = log2(1/pi). Here, pi is the probability of the ite
the history of selections, pi = ti/tt where tt is the number
selections in the menu. Now, Thhi is given by:
where ei[0..1] is an expertise factor ei = 1 1/t i where ti is t
number of repetitions with i. As the user becomes more experie
with a menu, there is a shift from serial visual search Tvsi to dec
Visual search time Tvsi is assumed to be linear function of th
number of n:
Tvsi = bvsn + avs
Hick-Hyman decision time Thhi is dependent on the entropy of t
item Hi = log2(1/pi). Here, pi is the probability of the item in
the history of selections, pi = ti/tt where tt is the number of to
selections in the menu. Now, Thhi is given by:
Thhi = bhhHi + ahh
where ahh and bhh are empirically determined constaints.
The model has been validated for a few different types of me
including linear, morphing, split, and rectangular menus. Figur
Fast to solve but trivial: Organizes the elements 
in decreasing order of frequency

T = f(L,T, H, P)
Selection time
Menu length
Target Position
Practice
The model is based on Fitt’s law and Hick-Hyman law
Entropy
THE SEARCH-DECIDE-POINT MODEL

HIERARCHICAL MENU
command!
command!
command!
command
command!
command!
command
command!
command
Title Title Title
command!
command!
command!
command!
command
command!
command!
command

DEFINE THIS DESIGN TASK
Given a bigram distribution for n commands, ﬁnd an
assignment of commands to a menu hierarchy such that
it minimizes expected search+selection time
Bigrams
Menu system
Command
Command
You have to make an
assumption about the cost of
searching submenus

ONE SOLUTION
• k submenus and n items can be organized in  
ways into a tree

• A QAP, but with much larger matrices for cost
(here:A) and permutations P(n, m)

!
• Practically impossible to solve to optimality when  
n >20
(n + k)n+k+2
= [csr] is an m x m matrix and P = [pij] is an n x n
P(n,m) is the set of all 1–1 (injective) mappings
…, n} to {1, …, m}, corresponding to the place-
the letters in these locations; and, !(!) is the loca-
etter i in the current solution ! ∈ !(!, !). Now,
(!) is the cost contribution of assigning letter i to
!(!) and letter j to !(!). This problem can be for-
with a quadratic objective function [33]: Let xij be
variable that is 1 if letter i is assigned to location j,
herwise. Now, Eq. 4 is equivalent to
min !!"!!"!!"!!"
!
!!!
!
!!!
!
!!!
!
!!!
,!!!!!!!!!!!!!!!!!!!!(5)!
" is the cost of using keyslot l after keyslot k.
gn of menu systems is a QAP, too: n commands are
to slots in k submenus that form a tree. Three dif-
must be addressed for treating menu system design
The Ant
Althoug
rapidly (
search g
particula
(ACO)
colony f
produce
Impleme
To our k
We use
ensures
upper an
pheromo
add pher
Bailly et al. UIST’13
p probability

a cost (search time)

MENU WITH GROUPS
command!
command!
command!
command!
command

DEFINE A DESIGN TASK
Given a bigram distribution for n commands, ﬁnd an
assignment of commands to groups in a menu such that
it minimizes expected search+selection time
Relatedness distribution Grouped menu
Command
Command
You have to make an
assumption about how groups
affect search+selection time

SOLUTION
• Add another objective to the QAP:
min
nX
i,j=1
nX
h,k=1
dihfjkxijxhk +
nX
i,j=1
nX
h,k=1
dihrjkxijxhk
Minimizes distance 
among related  
commands
Alternative: You can
formulate this as an additional
constraint
To decide where to put a
divider, you need some
heuristic to do that

SUMMARY
• Assignment tasks are of two kinds

• One-shot cost (linear)

• Transaction cost (quadratic)

• Powerful enough for many spatial design problems

• We considered only single objective: performance.
We will return to the multi-objective case

HTTP://GILLESBAILLY.FR/MENUOPTIMIZER.HTML
VIDEO FROM BAILLY ET AL. PROC. UIST’13

FACILITY LAYOUT PROBLEM
Fig. 4. Multi-ﬂoor layout.
Reviews in Control 31 (2007) 255–267
Drira et al. 2007

DISCUSSION: EXTENSION TO
LAYOUTS
• New design considerations

• Size of elements

• Coloring

• Grid layouts

• Objective function

• Clutter perception

• Visual search time

• Target acquisition time

• Grid quality perception

BEYOND ASSIGNMENT
PROBLEMS
Goal:To sensitize you to
different problem types in HCI

SKILL TRANSFER
• Intuition: Exploit “familiar patterns” in a new user
interface
Old activities
Familiar patterns

The Hughes Printing Telegraph (1855)
PianoText
[Feit DIS’14]

MPC-VCC Review 2014
VIDEO FROM FEIT & OULASVIRTA PROC. DIS’14

DEFINITION AS AN ASSIGNMENT TASK
• The objective minimizes distances among assigned
elements according to a given relatedness
distribution (r)

!
!
• O-oh!This does not work!
min
nX
i,j=1
nX
h,k=1
dihrjkxijxhk

ANOTHER TRY
• Have to consider both source and target domain
probabilities to make this work
max
X
a2A
X
b2A
X
s2T
X
t2T
pa,bps,txa,sxb,t
subject to
X
s2t
xa,s  1 8a 2 A
X
a2A
xa,s = 1 8s 2 T

EXAMPLE FUNCTIONALITIES
File Creation and organization
Close Close the current file
Create notebook Create notebooks to organize the notes
Delete file Delete a file
Delete notebook Delete notebook *
Exit Exit the application
Export Export the current file, e.g. to pdf
Move note to notebook Move a selected note to one of the notebooks.
New Create a new file
Open choose a previously created file to be opened
Open containing folder Open folder that contains the document.
Recently opened choose a recently opened file to be opened
Reload Reload the current file
Rename Rename file.
Rename notebook
Save Save the current file
Structural Elements
Bullet size Change size of bullets.
Bullet type Change bullet type.
Bullets Start bullet list.
Change number size
Change start number
Insert blank page
Insert date and time Insert date and time into the text.
Insert horizontal line
Number style Change style of numbering, e.g. 1. or 2) or iii or IV, etc.
Numbering Start numbered list.
Page break
Page layout
Line number color Customize color of line numbers
Line number continuation Continue line numbers or restart, e.g. at new page
Line number font Customize font of line numbers
Line number size Customize size of line numbers
Line numbers Show or hide numbering of lines
Margins Define the margins of the page (top, bottom, right, left)
Orientation Change the orientation of the page (landscape or portrait)
Page size
Change the page size, e.g. to predefined templates (A4, A5, ...)
or to a custom size.
Text columns Define the number of columns of the layout (e.g. 2-column text)
Link and share
Link to calendar entry Link note to event in calendar
Publish online Publish note online, e.g. on blog, facebook, etc.
Put on homescreen Put the note in form of a widget on desktop or homescreen
Share share notes via another application, e.g. Email, messenger, etc.
View
Close all windows In case of multiple open documents.
Compare documents Compare two documents and show differences.
Full screen Show in full screen mode, hide controls.
Line wrap Wrap text at the end of a line
Move document to other view In case of multiple documents per window (e.g. tabs) or split view.
Save all windows Save all open document.
Show grid
Show ruler
Split view Split view to show two documents at once.
Synchronize scrolling Synchronize scrolling of two open documents for better comparison.
Zoom Zoom in or out the document
Printing
Choose printer Choose the printer for printing the document
Number of copies Change the number of copies to be printed.
Print current page Print only the currently viewed page
Print custom range Define custom range of pages to be printed.
Print document
Print multiple pages per sheet
Print selection Print only selected text
Printer settings
Change setting of printer (e.g. color, quality, duplex etc.)
as given by operating system.
Search printer Search for a connected printer not displayed.
Controls
Copy
Cut
Delete element Delete selection
Find Find a certain text in the document
Mark all text
Mark text
Paste Paste copied or cut text.
Redo Redo last action.
Replace Replace selection by text.
Undo Undo last action.
Input
Change encoding Change encoding of text (e.g. UTF-8)
Draw freehand
Draw shape e.g. rectangle, line, triangle, etc.
Fill color Change fill color of shape
Handwriting recognition Enter text by recognition of handwriting
Insert link
Insert picture
Insert video
Line color Change line color of drawing.
Line style e.g. pen, pencil, crayon etc.
Record audio
Sort elements Sort selected elements alphabetically or numerically
Spell check Check for correct spelling
Type text Entering text via keyboard
Voice recognition Enter text by voice input.
Text style options
Align text Change text alignment, e.g. left, right, middle, etc
Character spacing Change space between characters
Font color Change color of font
Font size Change font size
Indent Change text indent.
Line spacing Change space between text lines
Paragraph spacing Change space between paragraphs.
Text background color Change color of text background.
Text font Change font of text
Text style Change style of text, e.g. bold, italic, underline, etc.
Table
Add column to table
Add row to table
Cell color Custom color of table cell
Cell margins Customize text margins of a cell in the table
Column size Change size of each column
Create table Insert a table with defined number of rows and columns
Merge cells Merge two or more cells into one.
Remove column from table
Remove row from table
Row size Change size of each row
Split cell Split a cell in the table into two.
Table border Customize table border lines
Text alignment in table
Notes application (106 functionalities)

TRADE-OFFS
Few LotsUsefulness
Learnability
Usability
User satisfaction
Costs
Proﬁtability

DEFINE A DESIGN TASK
Given a set of functionalities, as well as their estimated
usefulness values & usability costs, how to select the best
subset of functionalities?

THE NEXT RELEASE PROBLEM
• A software engineering problem
the next release of software.
ormulated as 0/1 knapsack problem: Let x1, ...xn be 0/1 va
ence/presence of requirements in software. The goal is
cted features to customers wi while keeping the total c
max
nX
i=1
wixi
s.t.
nX
i=1
cixi  B
xi 2 {0, 1}
at address a set of customers with different requiremen
xpressing dependencies in the implementation of featur
mplemented without feature A) [Bagnall et al. 2001].
Knapsack formulation
A burglar breaks into a jewelry store
He carries a backpack to store the jewels
Backpack has limited capacity, jewels have a monet
Goal: put in the backpack the combination of items th
max total value
Let’s design a heuristic
WS 2011
But: Inadequate for UI
designers

FORMULATING AN OBJECTIVE
FUNCTION WITH EXPERTS
Participants
“How do you design functionalities in a real and ideal case?”

INTERVIEW: RESULTS
Functionality selection is

•Embedded in user-centered design process

•iterative, corrective, exploratory

•cooperative and collaborative, multiple stakeholders

•Informed by multiple types of data and outputs

•user profiles, scenarios, sketches, prototypes, wireframes

•but designers often focus mainly on one scenario & user profile

•Addressing multiple design objectives
•usability, usefulness, user experience, profitability, implementability,
learnability, …

OBJECTIVE FUNCTION
erms of four objectives called USEP: usefulness U, satisfa
profitability P. These terms form the core of the objectiv
ut our work, supplementing it with case-specific weights, o
hen needed.
maxG = !U U + !SS + !EE + !P P
denote a weight given to an objective,
P
!2⌦ ! = 1.0.
wing, we define each term. Throughout, we denote a function
and the full set of selected functionality . By ”normaliza
al bar, we refer to scaling a variable to range [0, 1].
s
e usefulness of a design U as the sum of usefulness u and
ch selected functionality:
¯U =
1
2
(
X
2
¯u + D )
P. These terms form the core of the objective function we
upplementing it with case-specific weights, objectives, and
axG = !U U + !SS + !EE + !P P (4)
t given to an objective,
P
!2⌦ ! = 1.0.
each term. Throughout, we denote a functionality selected
set of selected functionality . By ”normalization”, marked
r to scaling a variable to range [0, 1].
a design U as the sum of usefulness u and dependency
ctionality:
¯U =
1
2
(
X
2
¯u + D ) (5)
2 [0, 1] and
D =
P
2
P
⌧2 ,⌧
P
i
P
j i,j
(6)
j2[0,1] are directional dependency scores defined by the de-
Usefulness
Dependency
the set of functionality that maximizes ”goodness” G. We
jectives called USEP: usefulness U, satisfaction S, ease-
. These terms form the core of the objective function we
pplementing it with case-specific weights, objectives, and
xG = !U U + !SS + !EE + !P P (4)
t given to an objective,
P
!2⌦ ! = 1.0.
each term. Throughout, we denote a functionality selected
et of selected functionality . By ”normalization”, marked
to scaling a variable to range [0, 1].
a design U as the sum of usefulness u and dependency
ctionality:
¯U =
1
2
(
X
2
¯u + D ) (5)
2 [0, 1] and
D =
P
2
P
⌧2 ,⌧
P
i
P
j i,j
(6)
2[0,1] are directional dependency scores defined by the de-
Satisfaction
3. Users value the loss of an important function more negatively than its inclusion in a design.
tisfaction
fine user satisfaction as a nonlinear function that maps the presence/absence of
tionality to a satisfaction score. This score refers to subjective experience of the
onality, as opposed to the usefulness score. We offer two formulations.
first definition uses a linear sum:
S =
X
2
¯s (7)
mitation is that the absence of a functionality is not accounted for in the score.
second definition builds on prospect theory that predicts that the absence of an
tant functionality is appraised more critically than its presence. This is captured
following mathematical function for prospect ⇡ of item :
⇡i =
⇢ p
si ifxi = 1
2
p
si ifxi = 0
(8)
we reformulate as
Ease-of-use
Note that satisfaction can be negative if a design misses functionali
to have.
This yields a new S⇡:
S =
nX
i=1
¯⇡(i)
S is used in the objective function after normalization.
4.3. Ease-of-use
We provide two definitions for the ease-of-use of a design E. The
quicker computation but simplifies the effect of increasing function
is a decreasing function of the number of selected functionality.
Our first formulation is a
E =
X
62
c
where c is the estimated complexity of .
ACM Transactions on Computer-Human Interaction, Vol. 1, No. 1, Article 11, Publication date: Ja
Profitability
Fig. 4. We assume the ease-of-use of an interface to decrease non-linearly with
Here, y = 1/(x 1).
Our second formulation captures the fact that ease-of-use is c
ing more and more functionalities. Including very few triviall
use. For example, designers group semantically to decrease sel
chical menu [Bailly et al. 2014]. Interface techniques for comma
functionality, toolbars, autocompletion, panels, and tool palette
effect. Ease-of-use would increase linearly only in the case the i
randomly. We also note that some interface techniques allow co
number of features. For example, hotkeys can make access t
but users typically learn only few hotkeys. While we use the re
following, we recognize that the exact shape of this function
including the skills of the design and the interface type. The re
shown in Figure 4 and given as
E =
1
| | + 1
We can also formulate this by introducing a quadratic constrai
4.4. Profitability
We define the profitability of a functionality p in terms of its
and costs c 2 R: p = v c . After normalization of p, we can
¯P =
X
2
p =
X
2
(v c )
Note that if c > v, profitability is negative. Sometimes inclu
negative in profitability is justifiable in the light of other object

BINARY QUADRATIC PROGRAMMING
!
• Find a binary vector of
length n to maximize
quantity q

• Global optimum can be
computed

• Fast!
LP ﬁle generator
Gurobi
Survey data (.csv)
Designers
Design exploration

! Result

EXPLORATIVE OPTIMIZATION
100,000 design task variations
100,000 optimum designs
A
Usefulness
Satisfaction
Ease of use
Profitability
24 functionalities
Add_column_to_table
Add_row_to_table
Align_text
Bullet_size
Bullet_type
Bullets
Cell_color
Cell_margins
Change_encoding
Change_number_size
Change_start_number
Character_spacing
Choose_printer
Close
Close_all_windows
Column_size
Compare_documents
Copy
Create_notebook
Create_table
Cut
Delete
Delete_notebook
Draw_freehand
Draw_shape
Exit
Export
Fill_color
Find
Font_color
Font_size
Full_screen
Handwriting_recognition
Indent
Insert_blank_page
Insert_date_and_time
Insert_horizontal_line
Insert_link
Insert_picture
Insert_video
Line_color
Line_number_color
Line_number_continuation
Line_number_font
Line_number_size
Line_numbers
Line_spacing
Line_style
Line_wrap
Link_to_calendar_entry
Margins
Mark_all_text
Mark_text
Merge_cells
Move_document_to_other_view
Move_note_to_notebook
New
Number_of_copies
Number_style
Numbering
Open
Open_containing_folder
Orientation
Page_break
Page_size
Paragraph_spacing
Paste
Print_current_page
Print_custom_range
Print_document
Print_multiple_pages_per_sheet
Print_selection
Printer_settings
Publish_online
Put_on_homescreen
Recently_opened
Record_audio
Redo
Reload
Remove_column_from_table
Remove_row_from_table
Rename
Rename_notebook
Replace
Row_size
Save
Save_all_windows
Search_printer
Share
Show_grid
Show_ruler
Sort_elements
Spell_check
Split_cell
Split_view
Synchronize_scrolling
Table_border
Text_alignment_in_table
Text_background_color
Text_columns
Text_font
Text_style
Type_text
Undo
Voice_recognition
Add_column_to_table
Add_row_to_table
Align_text
Bullet_size
Bullet_type
Bullets
Cell_color
Cell_margins
Change_encoding
Change_number_size
Change_start_number
Character_spacing
Choose_printer
Close
Close_all_windows
Column_size
Compare_documents
Copy
Create_notebook
Create_table
Cut
Delete
Delete_notebook
Draw_freehand
Draw_shape
Exit
Export
Fill_colorFind
Font_color
Font_size
Full_screen
Indent
Insert_blank_page
Insert_link
Insert_picture
Insert_video
Line_color
Line_number_color
Line_number_font
Line_number_size
Line_numbers
Line_spacing
Line_style
Line_wrap
Margins
Mark_all_text
Mark_text
Merge_cells
New
Number_of_copies
Number_style
Numbering
Open
Orientation
Page_break
Page_size
Paragraph_spacing
Paste
Print_current_page
Print_custom_range
Print_document
Print_selection
Printer_settings
Publish_online
Recently_opened
Record_audio
Redo
Reload
Rename
Rename_notebook
Replace
Row_size
Save
Save_all_windows
Search_printer
Share
Show_grid
Show_ruler
Sort_elements
Spell_check
Split_cell
Split_view
Table_border
Text_columns
Text_font
Text_style
Type_text
Undo
Voice_recognition
File creation and organization
Link and share
Printing
Table
Structural elements
Input
Text style options
Page layout
Controls
View
F
Usefulness
Satisfaction
Ease of use
Profitability
82 functionalities
Add_column_to_table
Add_row_to_table
Align_text
Bullet_size
Bullet_type
Bullets
Cell_color
Cell_margins
Change_encoding
Change_number_size
Change_start_number
Character_spacing
Choose_printer
Close
Close_all_windows
Column_size
Compare_documents
Copy
Create_notebook
Create_table
Cut
Delete
Delete_notebook
Draw_freehand
Draw_shape
Exit
Export
Fill_color
Find
Font_color
Font_size
Full_screen
Indent
Insert_blank_page
Insert_link
Insert_picture
Insert_video
Line_color
Line_number_color
Line_number_font
Line_number_size
Line_numbers
Line_spacing
Line_style
Line_wrap
Margins
Mark_all_text
Mark_text
Merge_cells
New
Number_of_copies
Number_style
Numbering
Open
Orientation
Page_break
Page_size
Paragraph_spacing
Paste
Print_current_page
Print_custom_range
Print_document
Print_selection
Printer_settings
Publish_online
Put_on_homescreen
Recently_opened
Record_audio
Redo
Reload
Rename
Rename_notebook
Replace
Row_size
Save
Save_all_windows
Search_printer
Share
Show_grid
Show_ruler
Sort_elements
Spell_check
Split_cell
Split_view
Table_border
Text_columns
Text_font
Text_style
Type_text
Undo
Voice_recognition
Add_column_to_table
Add_row_to_table
Align_text
Bullet_size
Bullet_type
Bullets
Cell_color
Cell_margins
Change_encoding
Change_number_size
Change_start_number
Character_spacing
Choose_printer
Close
Close_all_windows
Column_size
Compare_documents
Copy
Create_notebook
Create_table
Cut
Delete
Delete_notebook
Draw_freehand
Draw_shape
Exit
Export
Fill_colorFind
Font_color
Font_size
Full_screen
Indent
Insert_blank_page
Insert_link
Insert_picture
Insert_video
Line_color
Line_number_color
Line_number_font
Line_number_size
Line_numbers
Line_spacing
Line_style
Line_wrap
Margins
Mark_all_text
Mark_text
Merge_cells
New
Number_of_copies
Number_style
Numbering
Open
Orientation
Page_break
Page_size
Paragraph_spacing
Paste
Print_current_page
Print_custom_range
Print_document
Print_selection
Printer_settings
Publish_online
Recently_opened
Record_audio
Redo
Reload
Rename
Rename_notebook
Replace
Row_size
Save
Save_all_windows
Search_printer
Share
Show_grid
Show_ruler
Sort_elements
Spell_check
Split_cell
Split_view
Table_border
Text_columns
Text_font
Text_style
Type_text
Undo
Voice_recognition
Link and share
Printing
Table
Structural elements
Input
Text style options
Page layout
Controls
View
…
k Diverse designs
R
Usefulness
Satisfaction
Ease of use
Profitability
106 functionalities
Add_column_to_table
Add_row_to_table
Align_text
Bullet_size
Bullet_type
Bullets
Cell_color
Cell_margins
Change_encoding
Change_number_size
Change_start_number
Character_spacing
Choose_printer
Close
Close_all_windows
Column_size
Compare_documents
Copy
Create_notebook
Create_table
Cut
Delete_element
Delete_file
Delete_notebook
Draw_freehand
Draw_shape
Exit
Export
Fill_color
Find
Font_color
Font_size
Full_screen
Indent
Insert_blank_page
Insert_link
Insert_picture
Insert_video
Line_color
Line_number_color
Line_number_font
Line_number_size
Line_numbers
Line_spacing
Line_style
Line_wrap
Margins
Mark_all_text
Mark_text
Merge_cells
New
Number_of_copies
Number_style
Numbering
Open
Orientation
Page_break
Page_size
Paragraph_spacing
Paste
Print_current_page
Print_custom_range
Print_document
Print_selection
Printer_settings
Publish_online
Put_on_homescreen
Recently_opened
Record_audio
Redo
Reload
Rename
Rename_notebook
Replace
Row_size
Save
Save_all_windows
Search_printer
Share
Show_grid
Show_ruler
Sort_elements
Spell_check
Split_cell
Split_view
Table_border
Text_columns
Text_font
Text_style
Type_text
Undo
Voice_recognition
Zoom
Add_column_to_table
Add_row_to_table
Align_text
Bullet_size
Bullet_type
Bullets
Cell_color
Cell_margins
Change_encoding
Change_number_size
Change_start_number
Character_spacing
Choose_printer
Close
Close_all_windows
Column_size
Compare_documents
Copy
Create_notebook
Create_table
Cut
Delete_element
Delete_file
Delete_notebook
Draw_freehand
Draw_shape
Exit
Export
Fill_colorFind
Font_color
Font_size
Full_screen
Indent
Insert_blank_page
Insert_link
Insert_picture
Insert_video
Line_color
Line_number_color
Line_number_font
Line_number_size
Line_numbers
Line_spacing
Line_style
Line_wrap
Margins
Mark_all_text
Mark_text
Merge_cells
New
Number_of_copies
Number_style
Numbering
Open
Orientation
Page_break
Page_size
Paragraph_spacing
Paste
Print_current_page
Print_custom_range
Print_document
Print_selection
Printer_settings
Publish_online
Recently_opened
Record_audio
Redo
Reload
Rename
Rename_notebook
Replace
Row_size
Save
Save_all_windows
Search_printer
Share
Show_grid
Show_ruler
Sort_elements
Spell_check
Split_cell
Split_view
Table_border
Text_columns
Text_font
Text_style
Type_text
Undo
Voice_recognition
Zoom
Link and share
Printing
Table
Structural elements
Input
Text style options
Page layout
Controls
View
Robust design

NOTIFICATION SCHEDULING
• Assign incoming messages to a computer to one
of the statuses:

• Show immediately with a pop-up

• Push to list of notiﬁcations

• Don’t show

JOB-SHOP SCHEDULING TASK
n jobs
m machines
Find a schedule that 
minimizes total processing 
time

CHARACTERIZATION
• n jobs, each consisting of a chain of operations

• m machines, each handles at most one operation at a time

• Each operation must be processed during an uninterrupted
period of a given length on a given machine

• The objective is to ﬁnd a schedule (an allocation of the jobs
to time intervals to machines) with minimum duration

• In static JSS, n jobs arrive at an idle shop

• In dynamic JSS, jobs arrive intermittently

TASK: A NOTIFICATION BROKER
A set of incoming messages is given, decide to
which modality to channel them and when to
present them.

!
Tip:What is being minimized/maximed here?

SUMMARY
• Design tasks can described in terms of objectives
and constraints

• Assignment tasks are common in user interface
design

• Many optimization tasks in operations research
have applications in HCI

SICSA SUMMER SCHOOL ON COMPUTATIONAL INTERACTION

DAY 2: OPTIMIZATION
EXAMPLE: MID-AIR GESTURE
OPTIMIZATION  
[SRIDHAR ET AL. PROC CHI’15]

Model-Based User Interface Optimization: Part III: SOLVING REAL PROBLEMS - At SICSA Summer School on Computational Interaction 2015

More Related Content

What's hot (20)

Viewers also liked (10)

Similar to Model-Based User Interface Optimization: Part III: SOLVING REAL PROBLEMS - At SICSA Summer School on Computational Interaction 2015 (20)

More from Aalto University (10)

Recently uploaded (20)

Model-Based User Interface Optimization: Part III: SOLVING REAL PROBLEMS - At SICSA Summer School on Computational Interaction 2015