HCI 2015 (10/10) Natural User Interfaces. Ubiquitous Computing

Master on Software Engineering :: Human-Computer Interaction
Dr. Sabin-Corneliu Buraga – www.purl.org/net/busaco
Human-Computer Interaction
from natural interaction
to ubiquitous computing

“The details are not the details.
They make the design.”
Charles Eames

How about interacting with “special” devices
offering non-conventional interfaces?

sensorial experience
Sense Relevant information
Audio Speech, Identity, Intonation,
Precision (timing)
Vision Identity, Facial expression, Body
language, Gesture, Accuracy (spatial)
Tactile/
somesthetic
Tabs/pads/devices, Texture,
Precision (timing), Accuracy (spatial)
Scent Atmosphere, likability
Taste Clarification, enjoyment

The interpretation of sensorial data
depends on user expectations

M. Nielsen, T. Moeslund, M. Storring & E. Granum (2008)

Also, the interpretation of sensorial data
depends on specific input devices (K. Hinckley, ‘08)

Also, the interpretation of sensorial data
depends on specific input devices (K. Hinckley, ‘08)
mouse, trackball, isometric joystick,
touchpad, touchscreen, pen-operated device, watch,
camera, microphone, indirect table,…

Interaction modes
specify how an user interface responds to user actions

Interaction modes
specify how an user interface responds to user actions
for a particular manner of performing a task,
UI adopts a specific mode if the interpretation of this task
is always the same

Interaction modes
when a task has a different interpretation,
the UI offers another interaction mode

Interaction modes
when a task has a different interpretation,
the UI offers another interaction mode
modal vs. modeless interaction

Interaction modes
an interface could be modal for a given action,
and modeless for another (Jef Raskin, 2000)
if an UI is considered modeless,
then that UI must not be modal for any other action

Interaction modes
changing the interaction mode should have
a solid motivation
…even in the context of (mobile) Web

Haptic  Gesture  Locomotion
Auditory & Voice  Tangible
natural interaction

natural interaction
Haptic UI
(O’Malley & Gupta, 2008)
based on tactile sensations to provide information

natural interaction
Haptic UI
(O’Malley & Gupta, 2008)
based on tactile sensations to provide information
typical examples:
haptic screen for “touching” objects
natural output – e.g., vibrations (via tactons)

natural interaction
Haptic UI
cutaneous touch versus kinesthetic touch

natural interaction
Haptic UI
used in various contexts & domains:
CAD (Computer-Aided Design)
design prototyping
product evaluation
simulation of exceptional situations
remote task execution (tele-operation)
mobile & wearable computing

natural interaction
Haptic UI
usually, a haptic environment is defined by a formal model

natural interaction
example:
virtual wall – used in 3D games/simulations

natural interaction
Haptic UI
fixed haptic interfaces
haptic joysticks
pen-based haptic devices
floor- and ceiling-mounted interfaces

M. O’Malley, A. Gupta, “Haptic Interfaces”,
in P. Kortum (Ed.), HCI Beyond the GUI, Elsevier, 2008

natural interaction
Haptic UI
portable haptic interfaces
exoscheletons

natural interaction
Light Exoskeleton (L-Exos)
described by Frisoli et al. (2005)

natural interaction
PowerArm HCI project: Arduino, Android & Unity3D
Dan-Gabriel Bratu, Daniel Vicol, Sergiu Recean (FII, 2013)
http://profs.info.uaic.ro/~stefan.negru/studentprojects/powerarm.html
case study

natural interaction
Haptic UI
tactile interfaces
could detect heat, pressure, vibrations, pain
the feedback is usually provided by the user fingers

natural interaction
Haptic UI
tactile interfaces
vibrotactile interfaces
wearable tactile interfaces

natural interaction
Haptic UI
tactile interfaces
vibrotactile interfaces
wearable tactile interfaces
useful in the context of user accessibility

natural interaction
Moose haptic interface (O’Modhrain & Gillespie, 1995)
https://ccrma.stanford.edu/files/papers/stanm95.pdf
case study

natural interaction
AHNE – Audio-Haptic Navigation Environment
(SOPI Research Group, Aalto University, 2011)
user tracking via Kinect + OpenNI & OSCeleton
http://vimeo.com/28447850
case study

natural interaction
Haptic UI
aspects regarding human factors:
understanding bio-mechanical, sensorial-motor,
and cognitive abilities of users (Tan et al., 1994)

natural interaction
Haptic UI
accuracy of movement measurement
(position, velocity, acceleration)

natural interaction
Haptic UI
accuracy of movement measurement
(position, velocity, acceleration)
in the mobile Web context, see HTML5 APIs – e.g., Touch,
Proximity, Vibration, Screen Orientation, Gamepad, Media Capture
http://www.w3.org/Mobile/mobile-web-app-state/

natural interaction
Haptic UI
design principles:
base your mechanical design
on the inherent capabilities of the human operator

natural interaction
Haptic UI
design principles:
consider human sensitivity to tactile stimuli

natural interaction
Haptic UI
design principles:
use active rather than passive movement

natural interaction
Haptic UI
design principles:
minimize confusion and control instabilities

natural interaction
Haptic UI
design principles:
ensure realistic display of environments
with tactile devices

natural interaction
Haptic UI
specific uses:
(tele-)medicine
e.g., laparoscopic surgery simulator, palpation simulator

natural interaction
Haptic UI
specific uses:
3D surface modeling
example: Geomagic OpenHaptics Toolkit (C++)
http://www.geomagic.com/en/products/open-haptics

natural interaction
Haptic UI
specific uses:
learning the interaction with touchscreen devices

natural interaction
Haptic UI
specific uses:
digital art
dance – Edinburgh experiments (2012): http://tinyurl.com/k7kcap4
cinema – GATech PhD thesis (2011): http://hdl.handle.net/1853/39500
installation – Middlesex (2014): http://youtu.be/_3ECYqu61ZQ
sculpture – University of Guelph (2012): http://dhlguelph.com/art/

natural interaction
Pseudo-Haptic UI
(software) emulation of a haptic device

natural interaction
Pseudo-Haptic UI
sensorial substitution (Lenay et al., 2003)
a haptic stimulus is substituted by another sense
(e.g., an audio signal simulating
the contact with virtual objects)

natural interaction
Pseudo-Haptic UI
offering pseudo-haptic feedback (Anatole Lecuyer, 2007)
simulation of stiffness via a combination of
the interaction with the input device & the visual feedback
pseudo-haptic simulation of textures

natural interaction
Simulation-based Palpation using Pseudo-haptic
Feedback (Min Li et al., 2012)
www.youtube.com/watch?v=OcIczYCUkmc
case study

natural interaction
Gesture interaction
inspired by non-verbal interactions between people

natural interaction
Gesture interaction
inspired by non-verbal interactions between people
alternative or complementary to traditional interactions
“tap is the new click”

natural interaction
Gesture interaction
“While natural gestures are often subtle,
gesture interfaces rely on emphasized gestures.”
M. Nielsen et al., 2008

natural interaction
Gesture interaction
types:
hand gestures for pointing out objects
or controlling a mouse-like pointer

Interacţiune neconvenţională

natural interaction
Gesture interaction
types:
motion-tracking interfaces
used by several game consoles or laptops

natural interaction
Gesture interaction
types:
facial gestures
useful in detection of reactions and emotions

natural interaction
Gesture interaction
usually, based on specific input devices
mechanical/magnetic – e.g., mouse, data glove, Wii,…
single/multi-point touch
example: iPhone

natural interaction
Gesture interaction
could be “invisible” for the user
detecting gestures via computer vision:
segmentation – extracting the relevant fragments
tracking – detecting the movements of interest
classification – obtaining the desired information
(e.g., via machine learning, neural nets,…)

general architecture of a gesture-based interactive system
(Moustakas et al., 2006)

natural interaction
Gesture interaction
gesture classification:
semantic – the meaning of a gesture
functional – what the gesture does in an interface
descriptive – how the gesture is performed

natural interaction
Gesture interaction
functionally, gestures can signify:
command – Quit, Undo, Configure
pointing – e.g., selecting an item of interest
manipulation – example: scaling/rotating an image
control – emulates the control over an entity

natural interaction
dynamic & spatial-temporal “classical” gestures
performed via an input device: mouse, stylus,…

natural interaction
common gestures for Android devices:
touch, long press, swipe (drag), double touch, pinch
http://developer.android.com/design/patterns/gestures.html
case study

natural interaction
material design patterns (Google)
http://www.google.com/design/spec/patterns/
case study

natural interaction
Gesture interaction
the interaction vocabulary should be limited
we must assure an optimal interaction space (spatial zone)

natural interaction
“special” interactive zones having specific semantics

natural interaction
Gesture interaction
technology- versus human-based gestures
causes of discomfort:
stressful/fatigue producing for the user
nearly impossible for some people to perform
illogically imposed functionality

some gestures easily
recognizable by software
meanings/actions difficult
to be understood/performed
by the human users:
T0: without signification
(residue)
T1: select
T2: copy & paste
T3: delete
T4, T5: menu/release

natural interaction
Gesture interaction
a gesture-based UI must:
be intuitive
take into consideration the user mental model
be easy to learn & deployed (ergonomics)

natural interaction
Gesture interaction
practical design advices:
be tolerant to human errors
beware of cultural factors
consider the context
provide positive feedback

natural interaction
Gesture interaction
patterns for touchscreens & interactive surfaces
(Dan Saffer, 2009)
tap to open/activate
tap to select
drag to move object
slide to scroll, spin to scroll

natural interaction
Gesture interaction
patterns for touchscreens & interactive surfaces
(Dan Saffer, 2009)
slide and hold for continuous scroll
tap to stop
pinch to shrink and spread to enlarge
two fingers to scroll

natural interaction
gesture notation (Luke Wroblewski)

natural interaction
Cue – a set of icons to build a standard visual language of
touch-based interactions (P.J. Onori)
http://somerandomdude.com/work/cue/

natural interaction
Gesture interaction
patterns for free-form interactive gestures
(Dan Saffer, 2009)
move body to activate
point to select/activate
wave to activate
place hands inside to activate

natural interaction
Gesture interaction
patterns for free-form interactive gestures
(Dan Saffer, 2009)
rotate to change state
step to activate
shake to change
tilt to move

locomotion design
natural interaction

natural interaction
Locomotion interfaces
interfaces that both enable users to move around
in real or virtual spaces and make users feel
as if they are moving
Mary C. Whitton & Sharif Razzaque, 2008

natural interaction
could be virtual, not only physical

natural interaction
could be virtual, not only physical
main problem:
to capture the user’s intent using data that can be derived
from sensing the pose (position and orientation)
and movement of the user’s body

discussion

natural interaction
perceiving self-motion
each sensorial channel (visual, auditory, vestibular,…) is
used in detection of user or surrounded objects movement
Dichgans & Brandt, 1977

natural interaction
detection of pose and motion
artificial sensors (trackers)
optical motion capture
mechanical solutions

optical motion
capture
optical motion
capture
exoskeleton
trackers

Konami DanceDance Revolution
in the e-learning context
www.ddrclassroomedition.com/
case study

natural interaction
providing feedback to user(s)
visual information:
head-mounted displays
surround-screen displays

natural interaction
walking simulation
treadmills (Brooks, 1998; Hollerbach et al., 2000)
walking-in-place (WIP) interfaces

natural interaction
VirtuSphere
www.virtusphere.com
case study

natural interaction
design principles (Whitton & Razzaque, 2008):
making the interface feel natural
higher-fidelity reproduction of human way of walking
avoiding simulator sickness – cyber-sickness

natural interaction
issues that can be detected in the testing phase:
realism + preservation of spatial understanding
cognitive distraction
break-in-presence (BIP) events

natural interaction
an action that causes break-in-presence
(virtual scenephysical location)

natural interaction
advice:
match the locomotion metaphor to the interface goals
walking metaphor versus vehicle metaphor

natural interaction
practical considerations:
Is viewing direction independent of motion direction?
Are hands used for locomotion?
Able to move in any direction and change direction easily?
Can walking speed or step length be set for each user?

natural interaction
practical considerations:
Are tracker sensors immune to other signals in the room?
Are wireless trackers used?
What parts of body will have sensors/markers on them?
Cables – if any – interfere with user’s motion?

auditory & vocal interactions

natural interaction
Auditory interfaces
using non-speech audio information for interaction
for details, see (S. Camille Peres et al., 2008)

natural interaction
Auditory interfaces
motivations and uses:
presenting information to visually impaired people
providing an additional information channel
alerting people to error or emergency states of a system
providing information via devices with small screens

natural interaction
Auditory interfaces
why an audio interaction?
reducing visual overload
reinforcing visual messages
when eyes are elsewhere
when audio is more informative
conveying emotion – e.g., for electronic games

natural interaction
Auditory interfaces
sonification = using non-speech sound to render data
the simplest method is audification

natural interaction
Auditory interfaces
representational
auditory icons
abstract
earcons
Stephen Brewster (2008)

natural interaction
Auditory interfaces

natural interaction
Auditory interfaces
problems regarding the audio interaction:
annoyance
privacy
auditory overload
interference/masking
low resolution
lack of familiarity

natural interaction
Auditory interfaces
aspect of interest:
the tasks performed by the user via an audio interface

natural interaction
Auditory interfaces
aspect of interest:
what (kind of) information could be exclusively
represented by sounds?

natural interaction
Auditory interfaces
aspect of interest:
the suitable format audio – lossless vs. lossy – to be used
FLAC, PCM, WAV versus AAC, OGG, MP3

natural interaction
Auditory interfaces
aspect of interest:
producing the sound
thinking about sound as information
designing the sound – consider auditory content as
a kind of sound ecology (Walker & Kramer, 2004)

natural interaction
audio-only games – http://www.audiogames.net/list-games/
case study

natural interaction
Voice user interfaces
facilitating the conversation between software and user

natural interaction
important aspect: speech recognition

natural interaction
user perceptions regarding a VUI are fragile
lack of trust (Susan Hura, 2008)

natural interaction
users interact generally via prompts
“Hello, you have to choose from the following options…”

natural interaction
several discourse makers are needed for:
confirmation – “yes, of course”, “alright”
changing the topic of conversation
emphasizing a statement – “as we previously said…”
narrative sequences – “so,…”

natural interaction
problem: the lack of concision
“I can help you with the following five options. You can
interrupt me and speak your choice at any time. Please,
select one of the following: sign up for new service, add
features to my service, move my existing service, problems
with my satellite service, or ask a billing question.”

natural interaction
problem: dealing with errors
“I’m sorry. I did not understand your response.
I can help you with the following five options…”
inducing the feeling of user culpability

natural interaction
problem: increasing the user discomfort
“Please say or enter your account number!”
the user cannot control the interface

natural interaction
problem: using the same message (in different contexts)
“Attention! I don’t understand! Please, indicate the correct answer!...
Attention! I don’t understand! Please, indicate the correct answer!...
Attention! I don’t understand! Please, indicate the correct answer!...”

natural interaction
speech recognition – main phases (Cohen et al., 2004)

natural interaction
design guidelines:
present menu options in a way
that makes them comprehensible and easy to retain

natural interaction
design guidelines:
consider error sources when
writing error-handling prompts

natural interaction
design guidelines:
provide context
VUI: You can say: account balances, last contribution, transfers,
allocations, or life insurance.
User: Transfers.
VUI: Transfers. You can say: hear pending transfers
or make a transfer.

natural interaction
design guidelines:
use hints and reminders
“Next time, you can say ‘order status’ at the main menu
to get here quicker.”
“You can speak your account number or key it in.”

natural interaction
design guidelines:
favor models over instructions
learning by doing

natural interaction
design guidelines:
use discourse markers liberally
using in conversation common expressions
like “all right”, “next”, “thanks”,…

natural interaction
design guidelines:
allow for conversational repair

natural interaction
design guidelines:
be careful with terminology

natural interaction
recent examples in the mobile context:
Cortana (Windows Phone)
Google Now (Android)
Siri (iOS)
SpeakToIt Assistant (Android, iOS, Windows Phone,…)

natural interaction
case study

tangible interfaces

natural interaction
Tangible interfaces
digital informationphysical form (representation)
Hiroshi Ishii
digital content could be directly perceived & manipulated

natural interaction
Tangible interfaces
key components:
control – permits users to manipulate the content
external representations – perceived by human senses

natural interaction

natural interaction
Tangible interfaces

natural interaction
Tangible
Manipulation
haptic
direct
manipulation
lightweight
interaction
isomorph effects
Spatial
Interaction
inhabited space
configurable
materials
visual access
full body
interaction
performative
action
Embodied
Facilitation
embodied
constraints
multiple access
points
tailored (adaptive)
representations
Expressive
Representation
representational
significance
externalization
perceived coupling
Eva Hornecker (2010)
see also www.ehornecker.de/TangiblesFramework.html

natural interaction
Tangible Interaction Conceptual Framework
responding to key design questions via cards
Can users grab, feel and move “the important stuff”?
Do people and objects meet and invite into interaction?
Can you create a meaningful place with atmosphere?
Are representations legible, meaningful, and expressive?
https://personal.cis.strath.ac.uk/eva.hornecker/tang_framework_cards.pdf

natural interaction
Tangible interfaces
for other resources & demos, consult:
www.interaction-design.org/encyclopedia/tangible_interaction.html
http://mas834.media.mit.edu/
http://vimeo.com/search?q=tangible

multimodal interaction

Processing two or many inputs – voice, gestures,
tactile, etc. – in a coordinated manner in order to
obtain a complex output (e.g., multimedia, 3D) or
to perform a task
Sharon Oviatt, 2008
Paulo Barthelmess & Sharon Oviatt, 2008

“The situation where the user is provided with
multiple modes for interacting with a system.”
Abdo El Ali, 2011
http://www.slideshare.net/Abd0/multimodal-interaction-an-introduction

Interacţiune multimodală

Diverse contexts are considered
from mobile interfaces to collaborative systems
virtual and/or augmented reality

Advantages:
error reduction via mutual disambiguation
increasing user performance

Issues to be resolved:
linguistic factors
redundancy & complementarity
choosing the “right” multi-/mono-modal input
cognitive load

natural interaction
discussion
designing multimodal interaction for info kiosks

natural interaction
discussion
designing multimodal interaction for info kiosks
aspects of interest (M. Maguire, 1999):
choosing a suitable location
encouraging use
providing instructions for using the system
user privacy
multiple input methods: touch, speech, keypad
proper output: language, color, images, feedback,…
navigation possibilities
http://ui4all.ics.forth.gr/UI4ALL-97/maguire.pdf

ubiquitous computing

“Invisibly enhancing the world that already exists.”
(Mark Weiser, 1988)

(Mark Weiser, 1988)
ubicomp
pervasive computing
ambient intelligence (AmI)
physical computing
everyware

(Mark Weiser, 1988)
can occur using any device,
in any location,
and in any format
computers disappear in the environment

Related topics of research:
context awareness
distributed & mobile computing
Internet of Things (IoT) + wireless sensor networks

Scales of devices
Mark Weiser’s initial proposals (1991)
tabs – centimeter-sized devices:
PDAs, voice recorders, smart-phones, smart-watches,…
vision: individuals own many of them and they can all
communicate with each other and environment

natural interaction
interacting with a smartwatch via widgets
example: SmartWatch (Sony)
http://developer.sonymobile.com/tag/smartwatch/
discussion

Scales of devices
pads – handheld decimeter-sized devices:
tablets, laptops, e-readers, etc.
vision: an individual owns several,
but it’s not assumed to be always with them

Scales of devices
boards – meter-sized interactive devices:
mounted walls, e-tables, smart info kiosks,…
vision: buildings or institutions own them and
lots of people used them

natural interaction
Virtual Reality
term coined by Jaron Lanier (1989)
a computer-based simulated 3D environment
intended for its users to inhabit and to interact via avatars
an interface between the “physical” reality
and a virtual environment
discussion

natural interaction
Augmented Reality
term coined by Thomas Caudell (1990)
the use of technology which allows the perception of
the physical world to be enhanced or modified by
computer-generated stimuli perceived
with the aid of special equipment
generation of virtual objects merged into real scenes
discussion

natural interaction
reality-virtuality continuum (Milgram & Kishino, 1994)
see also www.slideshare.net/busaco/from-virtual-to-augmented-reality
discussion

Scales of devices
new proposals (Stefan Poslad, 2009)
dust – miniaturized devices without visual output
smartdust: a system of many tiny micro-
electromechanical systems (MEMS) that can detect light,
temperature, vibration, magnetism, chemicals,…

Scales of devices
skin – fabrics based upon light emitting, conductive,
polymers, organic computer devices
smart clotheswearable computing

natural interaction
T-Shirt-OS – 1024 pixels grid controlled by an app +
built-in micro-camera, mic, accelerometer & speakers
case study

Scales of devices
clay – ensembles of MEMS that can be formed into
arbitrary 3D shapes, resembling physical objects
remember tangible
interfaces?

M. Melnykowycz, Wearable Sensors and UX Development (2014)
www.slideshare.net/Boltzmann/wearable-sensors-and-ux-development

For various projects and demos,
see UbiComp conference proceedings
UbiComp 2013:
www.ubicomp.org/ubicomp2013/adjunct/forms/adjunct.htm
UbiComp 2014:
ubicomp.org/ubicomp2014/proceedings/ubicomp_adjunct/demos.html

ubiquitous computing: future
Brain-Computer Interface
direct communication between the brain and
an external device
assisting, augmenting, or repairing human cognitive or
sensory-motor functions

Alexis Ortiz-Rosario, Hojjat Adeli, “Brain-Computer
Interface Technologies: from Signal to Action”,
Reviews in the Neurosciences, 24 (5), 2013

example: Brainstorm – a collaborative, open-source
application for MEG/EEG/sEEG/ECoG data analysis
http://neuroimage.usc.edu/brainstorm/

“Conclusion”
The most profound technologies are those
that disappear. They weave themselves into
the fabric of everyday life
until they are indistinguishable from it.
Mark Weiser, “The Computer for the 21st Century”,
Scientific American, 265 (3), 1991

sensorial experience  natural user interfaces
multimodal interaction  ubiquitous computing
“Conclusion”

Good luck!

HCI 2015 (10/10) Natural User Interfaces. Ubiquitous Computing

More Related Content

Viewers also liked (12)

Similar to HCI 2015 (10/10) Natural User Interfaces. Ubiquitous Computing (20)

More from Sabin Buraga (20)

Recently uploaded (20)

HCI 2015 (10/10) Natural User Interfaces. Ubiquitous Computing