Action-Driven Design of User Interfaces

Designing and Analyzing Interaction Challenges for designers Model driven engineering UML-IDEA Conclusions

Action-driven design of user interfaces

Giorgio Brajnik

Dip. di Matematica e Informatica
University of Udine, Italy
and
School of Computer Science, University of Manchester, UK
www.dimi.uniud.it/giorgio
These slides: http://www.dimi.uniud.it/giorgio/events/aau.pdf

Institut für Informatik-Systeme
Alpen-Adria Universität Klagenfurt
January 25, 2013

c Giorgio Brajnik Action-driven design of user interfaces


Table of contents
Designing and Analyzing Interaction
What designers do
Usability investigations
Primacy of actions
Challenges for designers
Model driven engineering
Early model-based UIs
More modern model-based UIs
Action-Driven Design: the UML-IDEA
State machines
Expected advantages
Limitations
Conclusions



Survey on professional web designers

(Newman and Landay, DIS 2000)

Activities
1. information design (information architecture)
2. navigation design
3. visual design



What web designers do

more than one design shown to clients
different phases:
discovery
design exploration
design reﬁnement
production
iterative process
formal presentations at the end of each cycle



Different design artifacts

Early page sketches



Design artifacts II

Site map sketches



Design artifacts III

Storyboards



External artifacts

Polished site map



External artifacts II

Site map after client’s feedback



External artifacts III

Page schematics



External artifacts IV

Page mockups



External artifacts IV

Executable prototypes
in HTML or Powerpoint
Often mentioned
Usually done at the end, during "Production phase"
Usually developed as "executable specs"



My comments

1. lots of arrows, everywhere, to represent behavior
user actions are important
2. structure of content is important
it determines how content is operated upon
3. different types of artifacts
raw or polished ones
4. executable prototypes developed late
because expensive
their full power is not unleashed



Interaction Units

(Hokyoung and Monk, HCI, 2009)

A usability analysis technique
brief interaction scenarios annotated with user actions
with their effects on the UI
with their ripercussions on user’s goal stack



Monk’s triangular model

shows how interaction cycles through 3 steps
usability problems associated to edges



Embodied interaction
(Dourish, Where the action is, 2001)

Key points
1. Interaction is the means by which work is
accomplished, dynamically and in a context
2. The ways in which we experience the world are
through directly interacting with it
3. We act in the world by exploring the opportunities for
action that it provides to us



Coupling actions to functions
(Wensveen et al., DIS, 2004)

Relevant dimensions

time
location
direction
speed and force
modality
expressivity



My comments

actions and effects are important in design artifacts
they are important for usability
they are even philosophically justiﬁed



Expected results



Programming frameworks

Such as GWT, Spring, PHP, Dojo, jQuery, ...
1. these tools/frameworks promote bottom-up approaches
2. they are good for production
3. they are not suitable for UI prototyping
4. code is very structured, but user actions very opaque



Rich Internet Applications (RIAs) are complex

1. variety of widgets, that have to be choreographed
2. multitargeting (different user platforms)
3. with multimedia content
4. with multimodal interaction (gestures, voice, special
devices)
5. online/ofﬂine capabilities
6. distributed data management (granularity and policy of
syncs)
7. communication with server (pull/push)
8. hard and soft states in browsers



On top of that

Frequent changes are needed
new functional requirements
because new user segments
because new competitors
because new technologies
because of errors
...
Abstraction difficulties
behavior needs to be reified through some artifact
Gap between use cases and user interface specification



Model-driven engineering

Tenets
Modeling lifts the abstraction level from code to models
It provides means to direct the course of
understanding, design, construction, deployment,
operation, maintenance and modiﬁcation

Thus modeling could be a way to overcome some of the
challenges



Goals of MDE for UIs

To support:
abstraction of relevant views of the application
generation of early prototypes
rapid changes
ﬁnding usability problems
elicitation of requirements
seamless transition to implementation
multitargeting, multimodality
test management



State of the terminal

(Parnas, ACM Natl. Conference, 1969)

commands=inputs
system output=effects
He suggested to use them for
speciﬁcation of the UI
design support
to identify usability problems



State of the terminal II

Usability problems that can be discovered
hidden mode
dead ends
loops

After 43yrs, same problems
visibility of state, visibility of operations, user control,
consistency!



UWE: UML Web Modeling

(Koch et al., 2002)

following the MDE school
"evolutionist" approach:
many models
automatic M2M transformations (design patterns)
compatible with manual editing



UWE: metamodels

For a library management system:
use cases
conceptual model (classes)



UWE: metamodels II

navigation space



UWE: metamodels III
navigation structure

presentation model (abstract widgets)



UWE: metamodels IV

task model



My comments

interaction is mainly deﬁned in a data-driven approach
actions are spread over different models (navigation space,
navigation structure and task model)
approach well suited for CRUD applications
roundtrip engineering unsolved
activity diagram has its own interpreter included in
deployed app



RUX: Rich Internet Application User Experience

(Linaje, 2007)

"parassitic" approach:
it relies on abstract models written in WebML, UWE, OOH
then add RIAs models
it produces running code
through many complex M2M transformations



My comments

it addresses multimedia
it addresses responsiveness
through a ﬂat event/condition/action model
still bottom-up, still data-driven



Non-data-driven design

A UI for controlling a classroom
for 3 light sections
for audio-video equipments
for ventilation

Look at the following examples:
How do they differ from each other?
Which design decisions were taken, and why?



One and two

(Gajos, Weld, Wobbrock, SUPPLE, 2010)



Three and four



What do they have in common?

same conceptual model
same functionalities/use cases



Why do they differ?

interaction modality
geometry of the display
aesthetics
which actions
are more frequent

Conclusion:
these designs were not data-driven



What kind of models?

Consider that
UIs are reactive systems
they react to
user actions
server events
other users
UIs are also complex artifacts
require powerful abstractions for controlling complexity



State charts

invented by D. Harel in early ’80
in the context of avionics systems
to cope with "badly organized documentation and mental
models" of engineers
centered on events/conditions/actions rules

Key notions
"device modes" are crucial for understanding
visual language very powerful



State charts II



State machines in UML 2.4

Newer version of statecharts
Extended ﬁnite state machines: variables and operations
on them
Transitions with event triggers, guards, actions
States with entry/exit/do actions
State hierarchy ("superstates" and "superedges")
Concurrent regions
History
Arbitrary plumbing with pseudostates



Example of a TV set



The UI design problem

Deciding
how to locate in space and time information that users
have to interpret and actions they have to take, and
what look and feel to adopt
so that users can carry out their work



UML-based Interaction Design Approach
1. Start from essential use cases (user intentions and system
responsibilities)
2. deﬁne conceptual model of user interface
3. identify interaction spaces
4. model interaction through state machines
5. incrementally add data views models
6. incrementally add widgets/containers
7. automatically generate executable prototypes
for debugging
for user testing
to promote quick changes
to promote exploration of solution space
8. seamlessly evolve them into production code.



Foundation for UML-IDEA

Data views, to represent the objects being manipulated
and their operations;
State Machines and Events, to represent the behavior;
Annotations, to bind the two and providing widgets to
render data and triggers for events;
User platforms, to decide how to render things.



Data views (classroom example)



Behavior model I



Behavior model II



Action Driven Design

Basic assumptions behind ADD
1. States should be identiﬁed by the transitions that leave
them
2. States should be annotated with data and widgets
needed by the user to decide what to do next.



Design principles

1. Each state corresponds to an interaction space
2. Nesting of states corresponds to nesting of interaction
spaces
3. Concurrent regions correspond to sibling interaction
spaces



Results
No annotations With annotations



Annotations

Example that binds a widget to a state and a variable



Variant: one light section visible only



Results



Another design variant



Speed and ﬂexibility

Generation of UIs is automatic
Annotations add widgets and data
Many code hooks to enhance UIs rendering
Changing the dynamics is simple, as well as replacing
widgets
Suitable for exploring the design space
Suitable for user testing
Suitable for parallel design
Specs and code are in sync
Models are platform independent



Approach suitable for mixed-fidelity prototypes

Five dimensions of fidelity
Visual refinement
Breadth of functionality
Depth of functionality
Richness of interactivity
Richness of data
(McCurdy et al, 2006)

In UML-IDEA they are independent.



Powerful UI compiler

Seamless evolution from prototypes to running user
interfaces
Efficient code based on finite-state automata
Clever techniques to avoid combinatorial explosion
(flattening of state machines)



Expected characteristics of resulting UIs

Visibility: all and only needed options and materials are
visible to users
Feedback: users are kept informed of relevant state
changes
Structure: the UI is organized purposefully ... by putting
related things together and separating unrelated things.

(Constantine and Lockwood, 1999)



Compare with this



What did we do?

We separated three aspects



MVC

Model



MVC

Controller



MVC

And the View?



MVC

View
only partially given
in terms of individual widgets/containers
in terms of CSS rules
Not as a structure:

Hence:
V = M + C + widgets



Does MVC make sense at conceptual level?

(In addition to being useful at the programming level?)



Yes, of course!

Different issues can be uncovered
they are independent from each other, orthogonal
Model: what is being shown? does it make sense to
the user? are operations the expected ones? is some
missing?
Controller: are actions the expected ones in that
space? are they complete with respect to model? what
data need to be shown to let users make the
appropriate decision?
View: is the rendering of M or C appropriate?
salience? affordance? consistency?



UICompiler
right now a standalone Java application
capable of handling standard UML state machines
with a dozen types of widgets
a single target platform: HTML, CSS, jQuery

Next
to extend the widget library to handle most of
commonly used ones
to perform studies to make sure of usefulness of
generated prototypes
to target different platforms



Conclusions

1. Events/states/actions for UIs were suggested since 1969
2. SMs can be abstract (wrt platform, modality, development
platform)
3. SMs are expressive enough to generate UIs
4. SMs can be used as primary UI models:
to support elicitation of requirements
to support design
to support generation of prototypes
for usability analysis



Interaction Design Solutions SrL

A spin-off of the University of Udine
We design user interfaces
We do usability investigations
We develop GENIUS - the UML-IDEA generator of
user interfaces



THANK YOU!

Questions?

These slides:
http:
//www.dimi.uniud.it/giorgio/events/aau.pdf

Contact:
brajnik@uniud.it
www.dimi.uniud.it/giorgio


Action-Driven Design of User Interfaces

More Related Content

Viewers also liked (20)

Similar to Action-Driven Design of User Interfaces (20)

More from Förderverein Technische Fakultät (20)

Recently uploaded (20)

Action-Driven Design of User Interfaces