Yamamoto.2011.hyakunin eyesshu a tabletop hyakunin-isshu game with computer opponent by the action prediction based on gaze detection

Hyakunin-Eyesshu: a Tabletop Hyakunin-Isshu Game
with Computer Opponent by the Action Prediction
Based on Gaze Detection
Michiya Yamamoto
Kwansei Gakuin Univ.
2-1 Gakuen,
Sanda, Hyogo
669-1337, Japan
+81-79-565-7121
michiya.yamamoto@
kwansei.ac.jp
Munehiro Komeda
Kwansei Gakuin Univ.
2-1 Gakuen,
Sanda, Hyogo
669-1337, Japan
+81-79-565-7121
aqg77000@kwansei.
ac.jp
Takashi Nagamatsu
Kobe Univ.
5-1-1 Fukaeminami,
Higashinada, Kobe
658-0022, Japan
+81-78-431-6297
nagamatu@kobe-u.ac.jp
Tomio Watanabe
Okayama Pref. Univ.
111 Kuboki,
Soja, Okayama
719-1197, Japan
+81-866-94-2105
watanabe@cse.
oka-pu.ac.jp
ABSTRACT
A tabletop interface can enable interactions between images and
real objects using various sensors; therefore, it can be used to
create many works in the media arts field. By focusing on gaze-
and-touch interaction, we proposed the concept of an eye-tracking
tabletop interface (ETTI) as a new type of interaction interface for
the creation of media artworks. In this study, we developed
“Hyakunin-Eyesshu,” a prototype for ETTI content that enables
users to play the traditional Japanese card game “Hyakunin-
Isshu” with a computer character. In addition, we demonstrated
this system at an academic meeting and obtained user feedback.
We expect that our work will lead to advancements in interfaces
for various interactions and to various new media artworks with
precise gaze estimation.
Categories and Subject Descriptors
H5.2 [Information interfaces and presentation]: User Interfaces. -
Graphical user interfaces.
General Terms
Design, Measurement, Human Factors
Keywords
Eye tracking, tabletop, media arts, human interaction
1. INTRODUCTION
Human beings obtain a considerable amount of information
through their eyes. Thus, various artworks have been created to
appeal to our sense of sight [1,2]. At the same time, it is necessary
to increase the area of interaction of a tabletop interface [3,4]. A
game that uses eye tracking has already been developed by Li [5].
Moreover, the concept of a gaze-aware tabletop interface was
proposed by Holman [6]. We have already developed an eye-
tracking pen display and tabletop interface (ETTI) that enables
gaze-and-touch interaction, thus representing a novel interaction
interface for media artworks [7,8]. In this study, we developed
Hyakunin-Eyesshu—a tabletop Hyakunin-Isshu game that can
control the level through action prediction based on gaze
detection using the ETTI. In addition, we evaluated the ETTI by
demonstrating Hyakunin-Eyesshu at an academic meeting.
2. ETTI
2.1 Estimation of the optical axis of the eye
In our previous study, we developed an eye-tracking pen display
and realized eye tracking across a measurement range of
approximately 60°. The ETTI was developed by introducing a
new aspherical model of the cornea for more precise eye tracking
on a tabletop interface [9].
Here, we estimated the optical axis by using two cameras and two
IR LEDs. First, we calibrated the external parameters of the
cameras to acquire the relative 3D positions of the cameras and
displays. Next, based on the image processing results, we detected
the position of the pupil and two bright points as a Purkinje image
(reflections of the LEDs on the cornea). We assumed that the light
source and camera center were at the same position. Then, we
obtained a plane that contained vectors A and B , as shown in
Figure 1, by using the expression
where X is a point on the plane. One camera and
one LED were used for determining a plane that contained the
optical axis. Therefore, the optical axis could be obtained as the
intersection of two planes by using the two cameras and two
LEDs. Then, the user gazed at a point on the tabletop interface for
calibration. The difference between the optical axis and the visual
Permission to make digital or hard copies of all or part of this work for
personal or classroom use is granted without fee provided that copies are
not made or distributed for profit or commercial advantage and that
copies bear this notice and the full citation on the first page. To copy
otherwise, or republish, to post on servers or to redistribute to lists,
requires prior specific permission and/or a fee.
NGCA '11, May 26-27 2011, Karlskrona, Sweden
Copyright 2011 ACM 978-1-4503-0680-5/11/05…$10.00. Figure 1. Estimation of the Optical Axes.
 ）（ jjjj CP'B'C )(
0)(  jCX ,

axis of the eye was revised by carrying out this calibration. The
cross point of the visual axis and tabletop was estimated as the
gaze point.
2.2 System configuration
Figure 2 shows the system configuration of the ETTI for
“Hyakunin-Eyesshu.” The image on the tabletop interface was
generated by the PC and projected using an LCD TV (SHARP,
LC-52RX5) and short-focus projector (NEC, WT615). Two
cameras and two IR LEDs were arranged for the aspherical model.
We used two digital monochrome cameras (Point Gray, FFMV-
03MTM) equipped with 35-mm lenses and IR filters. An IR LED
was attached to each camera. The image was processed using an
eye-tracking PC (HP, xw4600 Workstation). The gaze point was
estimated and sent to two application PCs (HP, EliteBook 8730w).
We used a magnetic sensor (POLHMUS, Fastrak System) to
detect the user’s hand motion. The software was developed using
OpenCV 1.0 and Microsoft DirectX 9.0. It uses UDP
communication for data communication among three PCs.
2.3 Gaze detection accuracy
We performed two experiments to analyze the accuracy of the
ETTI. First, we selected a task by assuming that a user pushed a
button on a table. We prepared a 100 × 100 pixel square on the
tabletop and asked a user to gaze at the squares one by one for 5 s.
Figure 3 shows an example of the results. With the exception of
the screen edge, the measurement results for the square showed an
accuracy of almost 100%. The screen edge results had an
accuracy of more than 90%.
Next, we measured the traceability by moving the target on the
tabletop. Here, four users were asked to follow the cursor, which
was moved at a speed of 28 pixels per second along a spiral orbit
on the screen. Figure 4 shows an example of the results. The error
between the cursor and the gaze point was 38.7 pixels (16 mm,
1.5°), on an average, in the case of the subject with the best
accuracy and 53.8 pixels (22 mm, 2.1°) in the case of a subject
with the worst accuracy. There was a large deviation from the
cursor position; however, this deviation was attributed to the gap
caused by blinking.
3. HYAKUNIN-EYESSHU
3.1 Hyakunin-Isshu
“Hyakunin-Isshu” is a traditional anthology style of compiling
Japanese waka poetry. It also refers to a very popular card game
in Japan (upper part of Figure 5), which uses a deck composed of
poems (lower-left part of Figure 5) from one such anthology [10].
In this card game, cards with lines from poems are used (lower
right) and two players compete to find cards matching a poem
being read aloud. Because the player who gets the most cards
wins the game, it is necessary to find and snatch cards very
quickly.
3.2 Concept
“Hyakunin-Eyesshu” is a tabletop Hyakunin-Isshu game that
makes it possible to play with a computer character (Figure 6).
The ETTI can determine the moment that the user finds the
correct card by analyzing the user’s gaze point, because human
beings look proactively during such situations [11]. The quick
snatching motion is detected before the card is touched by using
the Fastrak. The computer character, i.e., the cat, might snatch
away the card at the user’s gaze point just as the user is about to
reach for it, thus making the game challenging. Furthermore, the
Figure 4. Results of traceability evaluation.
Figure 3. Results of selection evaluation.
Figure 2. System configuration.
Figure 5. Hyakunin-Isshu.

game’s level of difficulty can be adjusted by analyzing the user’s
gaze-and-touch timing. Because the computer controls all of the
positions for the cards and the correct answer, the previous
computer version of the game was not very exciting or enjoyable.
We were able to develop a real-time Hyakunin-Isshu game by
simply introducing gaze-and-touch interaction. Because the size
of each card was approximately 74 mm × 53 mm, a precise and
easily calibrated ETTI was necessary for the game.
3.3 Game outline
We developed three opponent characters, as shown in Figure 7.
One of the opponent characters is displayed on the screen
opposite the user, and the cards appear on the tabletop. The first
player to take three cards wins.
The blue ninja cat was designed for beginners (top part of Figure
7). This cat provides hints on the position of a card if a player
takes too long to find it. The yellow ninja cat was designed for
middle-grade players (middle part of Figure 7). This cat analyzes
the gaze and motion of the player and takes the correct card at an
appropriate time. The player can thus enjoy a challenging game.
The boss cat is designed for experts (bottom part of Figure 7).
This cat snatches the correct card with its tail just as the player
finds it and reaches for it. In this manner, using the ETTI, a
variety of computer characters was realized for Hyakunin-
Eyesshu.
4. DEMONSTRATION EXPERIMENT
Hyakunin-Eyesshu was publicly demonstrated on October 23–24,
2010, at Entertainment Computing 2010 (EC2010), held at the
Kyoto Institute of Technology (Figure 8). At this demonstration,
we first explained the system to visitors and calibrated it. Next,
we invited visitors to play the game. Finally, we asked them to fill
out a questionnaire form with four items rated on a five-grade
bipolar rating system: “I enjoyed the game,” “I want to play
again,” “I became interested in Hyakunin-Isshu,” and “The eye
tracking was precise.” There was also space for them to provide
their comments.
There were approximately 200 paid participants and 150 general
participants at this event, from which 73 played our system.
Unfortunately, gaze detection failed with 13 participants because
of diffused reflections from their glasses or the influence of
mascara. However, completed forms were obtained from the
remaining 60 participants. Figure 9 shows the results of their five-
grade evaluations. In response to “I enjoyed the game” and “I
want to play again,” 90% of the participants selected four or five.
Almost 90% of the participants selected three or higher in
Figure 8. Hyakunin-Eyesshu in EC2010.
Figure 6. Concept of Hyakunin-Eyesshu.
Figure 7. Computer characters.
Boss cat
（Expert）
Blue ninja cat
（Beginner）
Yellow ninja cat
（Middle-grade player）
Boss cat takes the card just before
the user’s hand reaches it
The user looks at the
correct card
One of the cards is read aloud
Yellow ninja cat starts taking the card
YES
NO
The user looks at the
correct card
The user moves hand
The user can
take the card
Blue ninja cat gives hints
about the correct card position
YES YES
NO
The user looks at
the correct card
The user can’t
find the card
for a while NO
Figure 9. Results of questionnaire.
(1) I enjoyed the game
５
62％(37)
4
27％(16)
3
3％(2)
2
5％(3)
1
3％(2)
(2) I want to play again
1
2％(1)
５
56％(34)
4
32％(19)
3
10％(6)
2
0％(0)
(3) I became interested
in Hyakunin-Isshu
1
7％(4)
５
30％(18)3
32％(19) 4
25％(15)
2
6％(4)
5
18％(11)
4
38％(23)
3
22％(13)
2
10％(6)
1
2％(1)
(4) The eye tracking
was precise

response to “I became interested in Hyakunin-Isshu.” Moreover,
56% selected four or five in response to “The eye-tracking was
precise.”
Finally, we asked the participants who played against the blue
ninja cat, yellow ninja cat, and boss cat for their responses to
“Hints were comprehensible,” “I played an enthusiastic game,”
and “The motion of the boss cat was quick enough,” respectively.
Figure 10 shows the results.
We also divided these results according to whether each subject
rated the eye-tracking precision as four or five and then applied
Friedman’s test. Figure 11 shows the results. We found that the
more precisely the gaze was detected, the higher the game was
rated.
The users provided many positive comments such as “the boss cat
was challenging” and “it was exciting that the cat snatched the
card at the last moment.” However, some users commented that
“the accuracy was insufficient,” and some players tried to win the
game using their peripheral vision.
5. CONCLUSION
In this study, we proposed the concept of “Hyakunin-Eyesshu,”
the first Hyakunin-Isshu game with computer opponent characters.
We developed a prototype of the system and confirmed its
effectiveness through a demonstration. Our future objectives are
to realize more precise and robust eye tracking and to improve the
opponent characters.
6. ACKNOWLEDGEMENTS
This work under our project “Embodied Communication Interface
for Mind Connection” has been supported by “New IT
Infrastructure for the Information-explosion Era” of MEXT
Grant-in-Aid for Scientific Research on Priority Areas. Also, our
project "Generation and Control Technology of Human-entrained
Embodied Media" has been sup-ported by CREST (Core
Research for Evolution Science and Technology) of JST (Japan
Science and Technology Agency).
7. REFERENCES
[1] Geller, T. 2006. Interactive Tabletop Exhibits in Museums and
Galleries. In IEEE Computer Graphics and Applications, 26, 5, 6-
11.
[2] Fujiki, J. 2007. OLE coordinate system. In ACM SIGGRAPH
2007 Art Gallery, 197.
[3] Schick, A., van de Camp, F., Ijsselmuiden, J., and
Stiefelhagen, R. 2009. Extending touch: towards interaction with
large-scale surfaces. In Proceedings of the ACM international
Conference on interactive Tabletops and Surfaces, 117-124.
[4] Yoo, B., Han, J., Choi, C., Yi, K., Suh, S., Park, D., and Kim,
C. 2010. 3D user interface combining gaze and hand gestures for
large-scale display. In Proceedings of the 28th of the international
Conference Extended Abstracts on Human Factors in Computing
Systems, 3709-3714.
[5] LI J., and JAMES J. C. 2008. Video Game Design Using an
Eye-Movement-Dependent Model of Visual Attention. In
Proceedings of ACM Transactions on Multimedia Computing,
Communications and Applications, Vol. 4, No. 3, Article 22.
[6] Holman, D. 2007. Gazetop: interaction techniques for gaze-
aware tabletops. In CHI '07 Extended Abstracts on Human
Factors in Computing Systems, 1657-1660.
[7] Yamamoto, M., Nagamatsu, T., and Watanabe, T. 2010.
Development of eye-tracking pen display based on stereo bright
pupil technique. In Proceedings of the 2010 Symposium on Eye-
Tracking Research & Appli-cations, 165-168.
[8] Yamamoto, M., Komeda, M., Nagamatsu, T., and Watanabe,
T. Development of Eye-Tracking Tabletop Interface for Media
Art Works. Proceedings of the ACM International Conference on
Interactive Tabletops and Surfaces 2010, pp.295-296
[9] Nagamatsu, T., Iwamoto, Y., Kamahara, J., Tanaka, N., and
Yamamoto, M. 2010. Gaze estimation method based on an
aspherical model of the cornea: surface of revolution about the
optical axis of the eye. In Proceedings of the 2010 Symposium on
Eye-Tracking Research & Applications, 255-258.
[10] http://en.wikipedia.org/wiki/Hyakunin_Isshu
[11] Land, M. F. 2006. Eye movements and the control of actions
in everyday life, In Progress in Retinal and Eye Research, 25, 3,
96-324.
Figure 10. Results of cats questionnaire.
1
9％(1)
５
18％(2)
4
46％(5)
3
18％(2)
2
9％(1)
(5) Hints were
comprehensible
1
3％(1)
５
19％(6)
4
34％(11)
3
28％(9)
2
16％(5)
(6) I played an
enthusiastic game
(7) The motion of the boss cat
was quick enough
1
7％(3)
５
15％(6)
4
44％(18)3
15％(6)
2
19％(8)
Figure 11. Divided by eye-tracking precision.
***
** ** P < 0.01
* P < 0.05
1
2
3
4
5
higher than three
three and under
(1) (2) (6)(5)(3) (7)
(1) I enjoyed the game.
(2) I want to play again.
(6) I played an enthusiastic game.
(5) Hints were comprehensible.
(3) I became interested in Hyakunin-Isshu.
(7) The motion of the boss cat was quick enough.

Yamamoto.2011.hyakunin eyesshu a tabletop hyakunin-isshu game with computer opponent by the action prediction based on gaze detection

More Related Content

Similar to Yamamoto.2011.hyakunin eyesshu a tabletop hyakunin-isshu game with computer opponent by the action prediction based on gaze detection (20)

More from mrgazer (10)

Recently uploaded (20)

Yamamoto.2011.hyakunin eyesshu a tabletop hyakunin-isshu game with computer opponent by the action prediction based on gaze detection