Gestural Interaction With In-vehicle Audio and Climate Control

Gestural Interaction With In-Vehicle Audio and Climate Controls

Chongyoon Chung1 and Esa Rantanen

Rochester Institute of Technology, Rochester, NY
1
Now at Samsung Electronics, Seoul, South Korea

Among the most distractive in-vehicle interactions are audio and climate controls. If these interactions
were as easy and spontaneous as natural language, driving could be much safer. Through this research it
was found that drivers preferred gestural language to voice language when the control was simple and re-
petitive; subsequently, gestural interactions with secondary in-vehicle tasks were investigated. Following
the principle of “eyes on the road and hands on the wheel”, a steering wheel design with two touch pads on
the wheel to recognize gestures was conceived. The physical design of the steering wheel incorporated
good ergonomics and anthropometric data, while gesture stereotypes assigned to a number of in-vehicle
controls were determined empirically by two experiments. The new steering wheel design does not have
any buttons, which may contribute to driver distraction, yet it incorporates 19 functions through natural
thumb-gestures. This compares favorably with most current steering wheel designs, which have more than
11 buttons and 13 functions on the average.

INTRODUCTION functions on their steering wheels. These values are undenia-
bly high.
Modern automobiles have a myriad of manual controls To reduce driver distraction due to operation of in-vehicle
for increasingly complex auxiliary systems that have the po- systems, their control interfaces should be as intuitive and
tential of distracting drivers from their primary task of driving. easy to use as possible. The most natural modes of communi-
Car audio, climate control, and navigation systems have stea- cation and control are language and gestures. Speech recogni-
dily increased in sophistication, and CD players and mp3 tion, such as Microsoft’s Sync system in new Ford automo-
player interfaces are now common even in least expensive biles, has recently been introduced to consumers. Speech rec-
models. Manual control of all these systems requires drivers to ognition has limitations, however, as it is vulnerable to noises,
take their eyes off the road and traffic environment, which has dialects, and individual voice differences. Moreover, current
obvious safety implications. Pickering (2005) found that an technology recognizes only specific words preprogrammed in-
average glance time to control a radio was 1.2 seconds; in that to the system.
time a car travels over 50 ft at 30 mph. Summala, Lamble, and
Laakso (1998) showed that ambient vision was not sufficient Gestural Control Interfaces
for hazard detection: response times increased significantly
with increasing eccentric viewing by up to 2.9 seconds, sug- Gesture control has yet to be implemented in automobiles
gesting that timely hazard detection required some degree of but gesture interfaces have been very successful in many mo-
focal visual resources. Driver distraction has recently received bile communications and computing devices with touch-
long overdue attention as a major contributor in accidents. Ad- screen interfaces. Similar interfaces could also be designed
justing radio, cassette, and CD players has been estimated to and implemented for in-vehicle systems control in automo-
cause 11.4% of drivers’ distractions (Stutts, Reinfurt, Staplin, biles. Gesture recognition interaction addresses many of the
& Rodgman, 2001). In addition, it is best for safe driving to problems associated with voice control and allows for reduc-
keep both hands on the steering wheel in case of sudden ma- tion of both visual and cognitive load of drivers. Gestures can
neuvers are needed to avoid road hazards. also be viewed as an integral part of natural language and
Current automobile interfaces can be very confusing with therefore they would be easy for drivers to learn.
too many functions per control. For example, the BMW 7 se- There are two main techniques for gesture recognition,
ries driver-controlled systems have over 700 functions (Gil- camera recognition and touch sensor recognition. Camera-
bert, 2004). Many controls are hard to find and even invisible. based gesture recognition has serious spatial limitations, how-
In an attempt to make in-vehicle controls better accessible to ever. A camera requires certain distance to recognize a driver's
drivers and to allow them to keep their hands on the steering gesture, but most interiors of automobiles are quite small,
wheel most of the time, most recent car models have placed making it difficult to install a camera that would accurately
many controls on the steering wheel spokes as push-buttons or detect and recognize gestures. Another problem is the posi-
toggle switches. Unfortunately, steering wheels can become tioning of the camera. As the majority of drivers are right-
very crowded with nonstandard ways of grouping and assign- handed it seems natural that gestures could be performed by
ing functionality to the buttons. A survey of eight car models the right hand. The best location for a gesture recognition
(ranging from luxury to compact cars) showed that modern camera would therefore be to the right side of drivers. How-
automobiles have on the average 11.62 buttons and 13.86 ever, the camera could be confused by a passenger’s gestures,

as the right side of the driver could be shared with a front pas- ledge or gesture stereotypes were then applied to controls in
senger. an automobile. To simplify this analysis, only audio and venti-
Touch sensor-based gesture recognition system is a good lation controls were considered. Finally, a prototype steering
solution that circumvents the spatial limitations of camera- wheel accommodating touch pads for gesture control was de-
based systems. Touch-based gesture recognition requires no signed. Note that only the initial design process is described
distance from drivers. There are two feasible locations for here. The steering wheel prototype is not functional and so no
touch-sensitive surfaces that could recognize gestures, on the usability testing of how well this design would work in an ac-
center console or on the steering wheel. If a touch pad is in- tual driving environment has yet been conducted.
stalled on the center console, however, drivers will need to
take their hand off the steering wheel to operate the system, PRACTICE INNOVATION
defeating the “eyes on the road and hands on the wheel” de-
sign principle. Therefore, the best place for touch pads is on There were two main considerations in the design. First,
the steering wheel. anthropometric principles and measures had to be considered
There are also two kinds of touch recognition systems, in the physical placement and size and shape of the touch pads
pressure and twist recognition system, and surface touch rec- on the steering wheel. Second, the actual control gestures
ognition system. For automobile applications, pressure points would have to be intuitive to the users. Anthropometric di-
could be incorporated in any place along the steering wheel mensions could be found in literature but gesture stereotypes
rim while a touch sensor area could be located in the hub or had to be determined empirically.
spokes of the steering wheel (Figure 1).
Anthropometric Considerations

Recognizes Because the steering wheel-mounted touch pad would
pressure necessarily be operated by thumbs, the anthropometric meas-
and twist ures of hand and in particular thumbs were the take as a start-
Recognizes ing point of the design. The average length of a male thumb
touch on the from crotch to end is 2.3 in, with a range from 2.1 to 2.9 in,
surface
and the average width 0.9 in, ranging from 0.55 to 1.25 in. For
female thumbs, the average length is 2.1 in, with a range of
1.7 to 2.5 in. The average width of a female thumb is 0.75 in,
ranging from 0.63 to 0.87 in. The lateral movement range of
the thumb is from 80 degrees abduction to 45 degrees of ad-
duction (Tilley, 1993).

Figure 1. Possible locations of pressure points and touch- Experiment 1
sensitive surfaces on a steering wheel.
Materials. To research gesture stereotypes of people for
Examples of how to use pressure and twist recognition certain functions, a mock-up steering wheel with touch pads at
system include a twist out to turn climate control on while two the 10:2 positions was created (Figure 2). The touch pads
twists out would turn climate control off; a twist in could be were made out of thin white fabric, through which partici-
air conditioner on and two twists in could be air conditioner pants’ thumb gestures could be videotaped. The length of the
off. Available mappings of gestures to pressure and twisting sides of the pads was 2.5”, or 95% women’s thumb length.
motions are very limited, however. A surface touch based rec- The angle between the two sides of the touch pad was 80˚,
ognition system would allow for a wider range of gestures to which is the radial abduction angle. The participants were also
be used as well as have other benefits as it is incorporated into queried about their preferred gestures by a questionnaire. The
the steering wheel design. Two touch pads on the wheel questions concerned which side would be better for audio con-
would also overcome some of the problems with depth of the trols and asked the participants to draw a preferred gesture for
menu. For example, one side would be for climate controls common control actions.
and another would be for audio controls. In addition, the best Procedure. Nineteen people volunteered for the experi-
hand position of drivers while driving is 10–2 position and ment, 13 males and 6 females. Everyone had a driver’s li-
therefore touch pads on the steering wheel should be located cense. Most participants were young college students, with the
close to these locations. exception of one person. The participants sat facing the expe-
rimenter and a video camera. The experimenter asked the par-
Purpose of the Research ticipants to perform a control action on the steering wheel
mock-up and they complied by making a gesture they felt was
This paper describes the design process for a gestural vo- the most intuitive for the required control with their thumbs on
cabulary interface for selected in-vehicle tasks. How people the “touch pads”. The gestures could be videotaped through
perceive and understand the meaning of certain most com- the fabric mimicking the touch pads. After this the participants
monly used gestures was investigated. This gestural know- drew gestures they made on the paper.

Two problems with gestures were discovered in this ex- last three functions, defrost, air from dashboard and air from
periment. One is that the participants had trouble imagining underneath were a tap or a tap that has spatial meaning such as
gestures in front of a camera and became nervous or rushed to top, center or bottom of the touch pad.
figure them out. Another problem was that they used all im- In the videotaped gestures, 10 participants gestured
aginable good, easy, and spontaneous gestures to “Turn the thumb up and down for volume up and down, and 6 of them
controls on and off.” Hence, they ran out of ideas for good did thumb right and left for change station to higher and lower
gestures for the remaining functions. station. However, 5 answers had directional meaning from
right to left for changing radio station to higher station and left
to right for changing station to lower station. Twelve people
performed a tap for mute, 16 did a tap for pause Mp3 player
and 14 a tap for play. Eleven people gestured thumb right
from left for forward to next and 13 moved their thumbs from
right to left for previous song. Ten participants did thumb up
and down for temperature up and down, and 4 of them did
thumb right and left for fan intensity up and down; 4 other
gestured thumb up and down for fan intensity up and down.
Nine participants tapped for defrost and 3 participants tapped
for air form dashboard; 2 others used a tap that had spatial
meaning like top and bottom. For example, for air from un-
derneath, 3 participants used a tap of the bottom of a touch
pad, one tapped the bottom left of the touch pad; and another
just tapped once.

Experiment 2

Figure 2. A mock-up steering wheel with touch pads for the Since it was possible that the relatively small touch pads
study of gesture stereotypes. Positions of thumbs making the used in Experiment 1 (see Figure 2) may have constrained the
gestures were videotaped through the thin material in the gestures performed by the participants and limited the variety
mock touch pads. of gestures, a second experiment was run with a new steering
wheel mock-up that had larger “touch pads” and with a new
Consequently, the order of testing was changed to have group of participants.
participants first draw gestures on paper with time to imagine Materials. The length of the new mock-up touch pad was
the gestures for the controls, and then perform the gestures for 2.7 in, longer than the average length of male thumb. The
the camera. This arrangement helped most participants to width of the touch pad now fully accommodated thumb
make more gestures that were also more variable, but many movement of 45° adduction and 80° abduction angle.
still ran out of ideas for “defrost” and “air from underneath”, Because the participants in the first experiment were pre-
for example. Regardless, they were asked to just do something dominantly young college students, older participants were re-
for each control action. cruited for the second experiment. Fourteen people volun-
Findings. The results between the drawings of gestures on teered, 5 males and 9 females. Their ages ranged from twen-
paper and those performed for video were almost identical. ties (one participant) to fifties. All had driver’s licenses and
Fourteen (out of 19) people chose right side as audio control. drove daily.
Over half of the participants offered same gestures for each The paper-and-pencil questionnaire was also slightly
function, with the exception of controls for fan intensity and modified for the second experiment: questions about prefe-
airflows. In the drawing task, 11 out of 19 participants sug- rences between gestural and voice command were added and
gested thumb up and down for volume up and down, 9 did the boxes for drawing gestures were eliminated not to con-
thumb right and left for change station to higher and lower strain free drawing of gestures in any way. Finally, a rating
station. Eleven participants offered a tap for mute, 9 drew a scale (1-5) was added to gauge how good, easy, and sponta-
tap to pause Mp3 player, and 11 a tap for play. Even though neous the most common gestures were.
these three functions shared the same gesture, it would work Procedure. With the exception of research material and
because radio and Mp3 player are different features and play questionnaires, the procedure was identical to that in Experi-
and pause are opposite functions. Twelve people gestured ment 1. One participant only answered the questionnaire and
thumb right and left for forward to next and previous song. was not videotaped.
One participant suggested using a long tap as turn on and off Findings. The large surfaces of the touch pads confused
like on a cellular phone, which would be easy to learn and some participants who thought that a big surface had spatial
perform based on this experience. meaning; they tried to push or touch a certain points as if
Ten participants offered thumb up and down for tempera- pushing imaginary buttons in the pad instead of making ges-
ture up and down, and 6 of them did thumb right and left for tures. This group as a whole could not carry out gestures as
fan intensity up and down. The most common answers of the

commands, having been accustomed to controlling devices neath and from air from underneath to defrost is a circulating
with buttons for most of their lives. single tap.
All 14 participants chose a conventional blinker over
voice command; 8 preferred a conventional wiper control and DISCUSSION: FINAL DESIGN
6 preferred voice control, suggesting a preference for gesture
to voice as a control of a simple and repeated function. Eleven Based on the anthropometric data, an ergonomically de-
participants selected right side for audio controls, 2 people se- signed shape of a touch pad that looks similar to a piece of pie
lected left side. Most of the gestures both drawn and per- was conceived. The sides of the touch pad are is 2.3 in long,
formed by the second group of participants corresponded to which corresponds to the average thumb length of males. The
those discovered in Experiment 1. On average, over a half of sides form an angle of 80°, which is the abduction angle of
the participants preferred the same gestures as the first group. thumb. Similar touch pads are placed on both sides of the
Gestures for air flow controls were most variable, as was the steering wheel (Figure 3).
case in Experiment 1.

FINDINGS

There was much commonality in how people imagined
gestures to perform common in-vehicle control tasks. Howev-
er, the number of different gestures offered by the participants
was limited, and some less used controls yielded diverse ges-
ture suggestions.
There were also some meaningful differences between the
participants in the two experiments. Participants in the first
experiment consisted of young college students in their twen-
ties whereas participants in the second experiment were older
adults. Younger people are accustomed to digital devices, es-
pecially touch screens and pads and were very good at making
various gestures on touch pad. The gestures gathered from the
second experiment were less variable; some of them tried push
imaginary buttons in their minds. All participants seemed to Figure 3. A technical drawing of the steering wheel design.
run out of imagination for gestures for airflow control. In both The dimensions are in inches and the angle between the two
experiments participants sometimes used their thumbs as if sides of the touch pad in degrees.
drawing a gesture on the touch pad. From these results most
common gestures could nevertheless be identified and used
for assigning controls to gestures on a touch pad. These ges-
tures for the right side, or audio controls were:

Tap and hold: Audio system on/off
Up/down movement: Volume up or down
Left/right movement: Radio station, previous or next
A single tap, or index
finger on backside: Radio to Mp3 player switch
Left/right movement: Previous or next song
A single tap: Pause/play

For the left side, or climate controls the gestures were:

Tap and hold: Climate system on/off
Up/down movement: Temperature up or down
Left/right movement: Fan slower or faster
Figure 4. Computer rendering of the final steering wheel
A single tap: Defrost
design.
A single tap: Air from dashboard
A single tap: Air from underneath.
The two touch pads on the wheel appeared to surround
and wrap the wheel. This appearance was carried over as an
Interaction needs to be consistent to the user. Switching
overall design concept, which underwent several iterations.
play and stop or pause is a single tap on right side for audio
The first draft had two spokes on wheel, which started from
control. Therefore commands of airflow from defrost to air
the pads. To differentiate the design from others, a one-spoke
from dashboard, from air from dashboard to air from under-

design instead of the two-spoke design was adopted. This ini- that gestural control interfaces that are already ubiquitous in
tial steering wheel design was full circular. However, to pre- many mobile communication and computing devices (e.g., so-
vent the two touch pads, which were protruding into the inside called smart phones, Apple’s iPhone, iPod, and iPad, and most
of the wheel, from occluding the instrument cluster, the steer- laptop computers) can find wider applications in automobiles.
ing wheel was stretched laterally by one inch. The final design
is depicted in Figure 4. ACKOWLEDGMENTS
On the steering wheel, most automobiles have average
11.6 buttons and 13.7 functions. In this design there are no This paper is based on the first author’s Master of Fine
buttons. Consequently, drivers do not need to look for Arts thesis for the graduate Industrial Design program at the
grouped and small buttons. Audio and climate controls cause Rochester Institute of Technology. Many thanks are due to his
the most drivers’ distractions among the secondary in-vehicle chief adviser, Prof. David Morgan for his help and support
tasks. If these control interactions were as easy as our daily throughout this program. Dr. Rantanen served as an associate
language, it would be very easy for drivers to operate in- adviser in the thesis committee. The helpful suggestions by a
vehicle tasks. “Eyes on the road and hands on the wheel” is third thesis committee member, Dr. Michelle Harris, are also
the maxim for safe driving. Automobile interactions must sa- gratefully acknowledged.
tisfy this for safe drive. This imposed the least visual and cog-
nitive load when controlling of in-vehicle systems. The pro- REFERENCES
posed gesture-based interaction satisfies all the requirements
stated above, potentially allowing drivers to drive less dis- Tilley, A. R. (1993). The measure of man and woman. Human
tracted and more safely. Factors in Design. New York: Henry Dreyfuss Asso-
Note that only the initial design process has been de- ciates.
scribed in this paper, culminating in a non-functional proto- Pickering, C. A. (2005). Interacting with the car. IEE Compu-
type steering wheel. To carry the development of this product ting & Engineering, 16(1), 26.
further, many engineering problems (e.g., materials of the Summala, H., Nieminen, T., & Punto, M. (1996). Maintaining
touch pads, their sensitivity, and gesture-recognition algo- lane position with peripheral vision during in-vehicle
rithms and software) would have to be solved. An extensive tasks. Human Factors, 38 (3), 442-451.
usability study would also need to be conducted to investigate Stutts, J .C., Reinfurt, D. W., Staplin, L., & Rodgman, E. A.
how well drivers could learn the gestures mapped to various (2001). The role of driver distraction in traffic crashes.
control actions and perform them reliably while driving in ac- AAA Foundation for Traffic Safety. Washington, D.C.
tual traffic environments. Despite these limitations, however, http://www.aaafoundation.org/pdf/distraction.pdf
this study revealed many new and potentially significant as- Gilbert, R. K. (2004). BMW i-drive. INFSCI 250. Pittsburgh,
pects about drivers’ interactions with ever-increasing in- PA: University of Pittsburgh
vehicle technologies and functions. Our research also suggests

Gestural Interaction With In-vehicle Audio and Climate Control

More Related Content

Similar to Gestural Interaction With In-vehicle Audio and Climate Control (20)

Gestural Interaction With In-vehicle Audio and Climate Control