Computer Input with Human Eyes-Only Using Two Purkinje Images Which Works in a Real-Time Basis without Calibration

Kohei Arai and Makoto Yamaura
International Journal of Human Computer Interaction (IJHCI), Volume(1): Issue (3) 71
Computer Input with Human Eyes-Only Using Two Purkinje
Images Which Works in a Real-Time Basis without Calibration
Kohei Arai arai@is.saga-u.ac.jp
Department of Information Science
Saga University
1 Honjo, Saga, 840-8502, Japan
Makoto Yamaura makoto@ip.is.saga-u.ac.jp
Department of Information Science
Saga University
1 Honjo, Saga, 840-8502, Japan
Abstract
A method for computer input with human eyes-only using two Purkinje images
which works in a real time basis without calibration is proposed. Experimental
results shows that cornea curvature can be estimated by using two light sources
derived Purkinje images so that no calibration for reducing person-to-person
difference of cornea curvature. It is found that the proposed system allows users’
movements of 30 degrees in roll direction and 15 degrees in pitch direction
utilizing detected face attitude which is derived from the face plane consisting
three feature points on the face, two eyes and nose or mouth. Also it is found that
the proposed system does work in a real time basis.
Keywords: computer input with human eyes only, cornea curvature, eyeball rotation angle estimation,
Purkinje image
1. INTRODUCTION
Now a day, several techniques for measuring eye movement behaviour are available [1],[2],[3].
Also “double-Purkinje-image (DPI) eye tracking system is proposed [4]. This technique is based
on capturing reflected infrared light that is projected on the eye. Other than these, there are
Leuven dual-PC eye-tracking system [5],[6]. As for the computer input system with human eyes
only based on an image-analysis method, many methods have being proposed so far. Matsuda et
al. makes the line of sight which connects eyeball rotation center coordinates and a pupil center,
and is performing gaze measurement [7]. Eyeball rotation center coordinates are searched for by
moving an eyeball in the various directions before gaze measurement. Therefore, since there is
no necessity of showing an index, gaze measurement is possible in all places, but preparation
takes time and there is a fault of not permitting head movement. Moreover, Ono et al. makes the
line of sight which connects cornea center-of-curvature coordinates and a pupil center, and is
performing gaze measurement [8]. Ebisawa, et. al. proposed 3D eyes detection together with
gaze estimation with two cameras for acquiring 3D data so that users’ movement can be
estimated [9]. The proposed system allows users’ movement. Meanwhile, Tanaka et. al.
proposed double Purkinje method for eye detection and tracking [10]. All these methods and
systems need some time consumable calibration process which allows geometric relation among
eyes, display and cameras.

Cornea center-of-curvature coordinates are the light source of one point, and installing a camera
on the optical axis, and are searched for using the general cornea radius-of-curvature value.
However, since they assumed models, such as Japanese typical eyeball form, the error remained
in the direction estimation of a gaze not a little, and if these methods did not perform a calibration,
when there were, they did not become. That is, in order to cancel the gaze estimation error based
on the gap of a central fovea to an eyeball center, the refraction in a cornea, and the individual
difference concerning the form of a cornea, the calibration which draws a gaze correction
coefficient needed to be performed by gazing at two or more indices displayed on the display one
by one. Moreover, since it was not what permits a motion of a user, the burden has been forced
upon the user. All these systems insist users to fix their face. More than that, these systems need
some time consumable calibrations in advance to use the syetms in order to make sure that the
gaze locations.
The former presumes the point of regard on the display at the point of the look obtained from
gaze estimation (three dimension measurement) of not only the direction of a gaze but both eyes,
and uses the pupil center and the corneal reflex center for gaze estimation. The latter presumes a
cornea center of curvature using two Purkinje images, makes a user gaze at the index of three
points, presumes an eyeball rotation center, is the method of making a gaze the straight line
which connects these, and is verifying accuracy by the experiment using a model eye. In this
paper, an expensive stereo camera is not needed, but only a cheap simple eye camera permits a
motion of a user, and the method of determining the direction of a look from a pupil center and a
cornea center of curvature is proposed without the calibration which forces a user a gaze of three
points. By specifically measuring an eyeball cornea curvature radius simply, the degree
estimation of eyeball rotation angle which does not need a calibration is performed, details are
extracted from a face picture, the posture of a head is detected from those relative spatial
relationships, and a motion of a head is permitted. The light source of two points was used for
measurement of the cornea curvature radius of an eyeball, and two Purkinje images obtained
from the cornea surface were used for it. At this time, it decided to also use together the near-
infrared light source which a camera has using the near-infrared camera which became budget
prices, and to acquire the clear Purkinje image in recent years. When five subjects estimate the
direction estimation accuracy of a look for this, a motion of the head of the 30 roll directions and
the 15 directions of a pitch is permitted, and since it checked that the direction of a gaze could be
presumed without a calibration with the error of 0.57 to 0.98 degrees, it reports here.
The angle estimation method of eyeball rotation angle of using two light sources and one near-
infrared camera is proposed first, the head angle detection method using the details in a face
picture is described, and this paper estimates the validity of the proposed method by the direction
estimation experiment of a look by five subjective examiners.
2. PROPOSED METHOD
Angle estimation of eyeball rotation angle and direction estimation method of look is as follows,
originally, a look is a straight line which connects a central fovea, the node of a lens, and the point
of regard, as shown in Figure 1, but since these position specification is difficult, it considers the
straight line going through a cornea center of curvature and a pupil center to be a look. Four kinds
of Purkinje images used for look estimation exist, as shown in Figure 2, but in this paper, the 1st
two Purkinje images with the biggest amount of catoptric light from two light sources are used. It
projects on right and left in every one front of the Purkinje cornea from a camera optic-axis
center. The example is shown in Figure 3. Next, the procedure of presuming the point of regard
on a display from two Purkinje images is shown.

FIGURE 1: Definition of eye gaze and line of sight
FIGURE 2: Purkinje images
FIGURE 3: An example of Purkinje image
First, as shown in Figure 4 (a), the geometric spatial relationships of the Purkinje image on two a
light source, one camera, and a simple eye cornea is considered. Two Purkinje images exist on
the line which divides into two equally the angle which the straight line which connects a cornea
center of curvature to a light source and a camera, respectively makes. Extraction of the Purkinje
image and the distance between two Purkinje images are calculable with image processing.
Since the Purkinje image exists as catoptric light on a cornea, compared with the circumference,
a luminosity value becomes remarkably high. Therefore, if there are some which show a high
luminosity value, let this be the candidate of the Purkinje image. Moreover, since near a pupil
exists, the Purkinje image extracts two candidates Purkinje who exist in the position nearest to a
pupil. Purkinje is made into the distance between Purkinje, the center of gravity of two obtained
candidates. This process is shown in this Figure (c).
In Figure 4 (a), a formula (1) can express a cornea curvature radius rE.
(1)
where L denotes the distance between two light sources, H means the distance between camera
and eyeball surface, and g denotes the measured distance between two Purkinje images. Thus, a
cornea curvature radius is searched for.
Next, a pupillary zone is extracted using a dark pupil method, ellipse approximation of the pupil
form is carried out, and a pupil center is searched for. With a dark pupil method, if an infrared
floodlight is installed in the position distant from the lens optic axis of a camera and an eyeball is
illuminated, the reflection of light of the pupil portion of the illuminated eyeball will be lost, and the
pupil portion of the eyeball picture caught with the camera will use the character to become dark
here. Image processing extracts this dark portion. Moreover, in the extracted pupillary zone, since

reflected lighting and the up-and-down portion of a pupil hide and are missing by the eyelid, the
pupil may not be able to be extracted correctly. Therefore, edge is detected from the obtained
temporary pupillary zone, and the edge concerned is approximated to an ellipse by a least-
squares method. However, when it calculates using except a pupil outline, there is a possibility of
resembling the mistaken ellipse. Therefore, pupil ellipse approximation using the character of an
ellipse was performed. That is, in Figure 5, straight lines l, m, and n shall be parallel, and l and n
shall exist in the equal distance from m.
(a) Method for gaze direction estimation
(b) Estimation of cornea curvature radius (Purkinje image is situated on the bisector between light
source and cornea curvature center
(c) Process flow of the method for gravity center estimation using two candidates of the detected
Purkinje images
FIGURE 4: Method for estimation of cornea curvature radius using the distance between two
Purkinje images.
FIGURE 5: Extraction of sample points from the assumed ellipsoid.

The intersection of an ellipse and a straight line l is set to a and b, the intersection of an ellipse
and a straight line n is set to c and d, and the middle point of the intersection of an ellipse and a
straight line m is set to o. When the middle point of a line which connects the middle point of a
and b and the middle point of c and d is set to o0, there is character in which o0 overlaps with o.
N which is in the equal distance about a temporary pupillary zone from the straight line m drawn
in the center. The point of hitting oi0 (i = 1-N) of the group of the parallel lines is searched for. The
point searched for is distributed on a straight line m. When there are few noises enough, the
position in which most many points o0 gathered is equivalent to the position of o. Since the point
which is distant from the position will include the point which is not on the locus of an ellipse, it
accepts them. o point ai which is alike and has sufficiently near oi0; bi; ci; Since di is a point
on an ellipse, ellipse approximation of an exact pupil outline can be performed by using them. In
addition, the Canny filter [11] was used for edge detection. A pupil outline can be extracted by
approximating an ellipse from the pupil ellipse sample point acquired by the above-mentioned
method. The process of old processing is shown in Figure 6.
(a) Original (b) Binalized (c) Noise reduced (d) Edge (e) Extracted (f) Approximated
image image image detected samples ellipsoid
FIGURE 6: Method for estimation of approximated ellipsoid of cornea shape.
Next, a cornea center of curvature is searched for from the spatial relationship of the acquired
cornea curvature radius and the light source of Figure 7, an eyeball, and a camera, and the
geometric expression of relations in a Figure. A cornea center of curvature exists here on the
bisector which ties a camera, the Purkinje image, and lighting. It asks for a look vector from a
pupil center and a cornea center of curvature.
Finally the point of regard on a display is computed from a look vector. The vector which passes
along two points, a cornea center of curvature and a pupil center, is made into a look vector, and
the point of regard on a display is computed.
FIGURE 7: Geometric relation among light source, camera and eyeball
It is considered as a camera position (0, 0, 0), and is considered as the main coordinates (0, 0z)
of the picture picturized with the camera. If a look vector is made into v = (xv; yv; zv), a camera,
lighting, and a display assume that it is being fixed and the head is also being fixed, and are z.

The distance of the direction of an axis is known. When distance of an eyeball and a display is set
to zh, point-of-regard coordinates t = on a display (xt; yt) is,
(2)
so that the gaze fixed point can be calculated as is shown in Figure 8.
(a) Calibration point locations on the computer screen for evaluation of gaze position estimation
accuracy (red point shows the estimated gaze fixed point)
(b) The estimated cornea center and pupil center
FIGURE 8: Method for gaze fixed position estimation.
How to permit a motion of a head finally is shown. The both ends of eye and a mouth, the middle
point between two ends of eye, and the middle point of the both ends of a mouth are detected
from a face picture. Template matching is used for pursuit of details at these details extraction
using OpenCV [10]. By the middle point of both eyes, and the middle point of a mouth, three
square shapes can be constituted and a plane can be defined. The normal direction of this plane
is judged to be the posture of a head. Once extracting these details, by using template matching
which can pursue most in a short time, it is devising so that the pursuit in real time may be
possible. An example of details extraction and pursuit is shown in Figure 9. The distance between
the details in a Figure is found and the degree of rotation angle of a head can be presumed in
comparison with the position of the details extracted in early stages by OpenCV. The blue line
segment of Figure 9 is a head posture (the plane normal direction is shown.). Moreover, a green
line segment connects the middle point between the middle points of both eyes, and the middle
point of the both ends of a mouth.
The geometric relation between the rotation center of a head and an eyeball rotation center is
shown in Figure 10, and can presume an eyeball rotation center position after this. If the method
of presuming a look from an eyeball rotation center position is considered to be the same thing as
the above-mentioned method, a motion of a head will be permitted and the direction estimation of
a look will be attained. At this time, that the both ends of both eyes and the both ends of a mouth
are not occluded are the conditions which presume a head rotation center and an eyeball rotation
center, and 30 degrees and about 15 degrees are rotation allowable limits in the roll direction and
the direction of a pitch.

FIGURE 9: Method for head pose angle estimation using the extracted feature points from the
acquired face image.
FIGURE 10: Geometric relation between two coordinates of head and eyeball
3. EXPERIMENTS
3.1 System Configuration
The measurement equipments used is the followings,
PC: Dell Computer Optiplex 755 Core 2 Quad 2.66 GHz CPU with 2 MB RAM of main memory
OS: WindowsXP home Service Pack2IR
Camera: (640 by 480 pixels) of DC-NCR 131 type manufactured by NetCowBoy and frame
rate: 10 frames-per-second
Infrared floodlight: KMT-7787.
These measurement equipments are installed as shown in Figure 11.
3.2 System Parameters
The parameters of the experimental conFigureuration are as follows,
L = 1000 mm; HC = 670 mm; HT = 150 mm; HD = 380mm
Software development environment is as follows,
Microsoft Visual C++
Y
X
Z
O1(x, y, z)
O2(x, y, z)
Y
Z
X
H
E

Microsoft Visual studio.NET2
OpenCV 1.0
Moreover, the picture acquired from the camera on the occasion of look measurement was
processed in real time, and performed accuracy verification.
(a) Experimental configuration (b) Experimental layout from the top view
FIGURE 11: Experimental layout and configuration
3.3 Cornea Curvature Measurements
The measurement result of a cornea curvature radius is shown in Figure 12.
FIGURE12: An example of experimental results of cornea curvature radius estimation.
The spike in a Figure is based on incorrect detection of the Purkinje image, and these can be
accepted from measurement of a cornea radius. Moreover, two lines in a Figure are the
maximum errors of the cornea curvature-radius point estimate considered when movement of ±
1.0cm has a head in the display direction. The average of the cornea curvature-radius point
estimate except an edge portion is used for the point estimate of the cornea curvature radius
used for the actual degree estimation of eyeball rotation angle. The value of a cornea curvature
radius was set to R = 7.92 mm from this result.
3.4 Eyeball Rotation Angle Estimation Accuracy
In the experiment, eyeball angle estimation accuracy was checked by seeing five indices
displayed on the display one by one, where a head is fixed simply. The presumed result of the
eyeball angle detection from which a cornea curvature radius differs is shown in Table 1. It is
delta x and y when a cornea curvature radius is set to R = 7.92 mm at this time. ∆ xy = it is 1.128.

TABLE 1: Mean of error in eyeball rotation angle estimation for the various cornea curvature
radius (five people of examiners)
Near the measured cornea curvature radius has the highest degree detection accuracy of eyeball
rotation angle. Therefore, it has checked that rather than led which uses the measured cornea
curvature radius to improvement in the degree detection accuracy of eyeball rotation angle using
the standard value of a cornea curvature radius.
Change of the look estimation result of survey by head parallel translation and rotation is shown
in Table 2.
(a) Head translation in x direction (b) Head translation in y direction
(c) Head translation in z direction (d) Head rotation in roll direction
(e) Head rotation in yaw direction
TABLE 2: Eyeball rotation angle estimation errors (Mean of estimation errors for five examiners)
Here, movement and rotation (circumference of y and a z axis) for all directions of x, y, and z
were investigated, respectively. When Table 2 (a) is seen, it turns out that the error generated by
head movement of a x direction affects only a x direction. It is the same thing as is shown in
Table 2 (b) and (c), it turns out that the error generated by head movement of y and the direction
of z affects it only in y and the direction of z. This cause is because depth is given as a constant,
and in order to solve this, the distance between a head and a camera just measures it on real
time. If Table 2 (d) is seen, the fall for presumed accuracy with big head rotation to the roll

direction will be seen. Moreover, in Table 2 (e), the big fall for presumed accuracy was expected
for the head rotation to the direction of yaw not to have the influence of head rotation looked at by
presumed accuracy, but to exceed a certain range in a certain fixed range. This is considered to
originate in the cornea form of the human being that a cornea curvature radius becomes large
steeply by the nose side gently-sloping by the ear side.
In measurement of a cornea curvature radius, it seems that the big accuracy fall has occurred
when this is exceeded since it has measured in 3.0-5.0mm from the cornea center-of-curvature
part. Moreover, in the influence on the presumed accuracy by the head rotation to the direction of
a pitch, since the influence of eyelashes becomes large in downward rotation of a head, pupil
detection becomes difficult (measurement is impossible), and since the influence of upward
rotation of a head of eyelashes decreases, its pupil detection accuracy improves. From this, the
camera position at the time of measurement is understood that it is desirable to install by
arrangement which looks up at a head.
3.5 Head Pose Angle Estimation
As 3.4 described, movement in the direction of z is compared with movement to x and a y
direction, and has serious influence on eyeball angle detection. Therefore, I need to get a user to
maintain distance with a computer display to some extent (separating beyond the defined
distance and twisting like). As Chapter 2 described, to rotation of x of a head, the parallel
translation of a y direction, a roll, a pitch, and the direction of Yaw, it is detectable. As for parallel
translation, it is highly accurate and can presume correctly about 100%. Therefore, the degree of
setting head rotation angle and average estimation error by five subjects were checked by
experiment here. An experimental result is shown in Figure. 13. Since the Yaw rotation of a head
is rotation in a head plane, a presumed error is comparatively small. The presumed error of the
direction of a pitch was the largest, and it turned out that the presumed error of about 5 times is
produced also in rotation of about 1.0 degrees in the pitch rotation to a direction which especially
bows. Even if it is relatively stout and rotates about 30 degrees to the pitch rotation to a direction
which raises the head, the presumed error is settled in about 1.7 degrees. Moreover, the
presumed error over rotation of the head of the roll direction was located in that middle, and when
it was ±30 roll rotations, it has checked fitting in the presumed error of less than 1.7 degrees.
Furthermore, one of eyes occlusion roll rotation exceeding 40 degrees, and a presumed error
becomes large rapidly. Therefore, ±30-10~+30 degrees, even if the head rotated ±30 degrees in
a roll, a pitch, and each direction of Yaw, it checked fitting in the presumed error of less than 1.7
degrees to them.
3.6 Overall Performance
A cornea curvature radius is presumed using the Purkinje image of two points obtained by using
the two-point light source proposed in this paper, the proposed method of eyeball rotation angle
estimation which does not need the calibration for individual difference dissolution of cornea
curvature are 0.57 degree and y to the direction of x in the state where the head was fixed in
simple. It turned out that the degree of eyeball rotation angle can be presumed with the error of
about 0.98 degree in a direction. Moreover, since the degree estimation method of head rotation
angle using the details of the face proposed in this paper is the presumed error of about 1.7
degrees under restriction of the degree of rotation angle, it assumes that rotation of these heads
and rotation of an eyeball are independent, and it is RSS: Root Sum Square is taken, it turns out
that the eyeball angle estimation accuracy in the case of permitting rotation of a head is 1.988
degrees in a x direction at 1.821 degrees and a y direction.
If a look is presumed and rotation is permitted for a head in the above-mentioned restriction range
from the face picture acquired from the camera which separated 150mm, the direction of a look
can be presumed in the error of about 2 times. When an intersection with a look is put on the
computer display which left this 300mm, it is equivalent to about 10.472mm look and a display
intersection position estimation error. When a with a pixel size (4 pixels / 10mm) computer display
is assumed, this intersection position estimation error will be equivalent to about 4 pixels. Figure
14 sets up a target on 7 pixels from a center as an example of look stability, gets five subjects to
gaze at the target concerned for 30 seconds, and shows what plotted the presumed intersection
position. Therefore, by making distance between adjacent keys into 10 pixels or more showed

that rotation of a head was permitted and the degree estimation method of eyeball rotation angle
which does not need a calibration could be realized.
FIGURE13: Head rotation angle estimation error in pitch, roll and yaw directions.
FIGURE14: Stability of the estimated gaze intersection on computer display. (Gaze position is set
at 7 pixels from the center of the computer display as an example. The stability is within a couple
of pixels mostly, -7 to +6 pixels error is happened occasionally though.)
4. CONCLUDING REMARKS
This paper proposed the estimation method of eyeball rotation angle which does not need the
calibration for individual difference dissolution of cornea curvature by presuming a cornea
curvature radius using the Purkinje image of two points obtained by using a two-point light
source. As a result, in the state where the head was fixed in simple, they are 0.98deg(s) to
0.57deg(s) and the direction of y in the direction of x. The degree estimation accuracy of eyeball
rotation angle was acquired with the error of a grade. Moreover, by the angle estimation method
of the plane defined by the details of the face proposed since rotation of a head was permitted,
when preparing restriction in each rotation of a roll, a pitch, and a yaw directions, it checked that
the degree of rotation angle could be presumed with the error of about 0.5 degree. When it
assumed that rotation of these heads and rotation of an eyeball were independent, it turned out
that the eyeball angle estimation accuracy in the case of permitting rotation of a head is 1.988
degrees in a x direction at 1.821 degrees and a y direction. That is, since the degree of viewing
angle beyond this error was detectable, when detaching and setting up the interval of a key rather
than this, it turned out that it can guarantee that discernment of a key is possible.

5. REFERENCES
1. Carpenter, R. H. S. (1988). Movements of the eyes. London: Pion.
2. Collewijn, H. (1999). Eye movement recording. In R. H. S. Carpenter & J. G. Robson
(Eds.), Vision research: A practical guide to laboratory methods (pp. 245-285). Oxford:
Oxford University Press.
3. Crane, H. D., & Steele, C. M. (1985). Generation-V dual-Purkinje-image eyetracker.
Applied Optics, 24, 527-537.
4. Deubel, H., & Bridgeman, B. (1995a). Fourth Purkinje image signals reveal eye-lens
deviations and retinal image distortions during saccades. Vision Research, 35, 529-538.
5. Deubel, H., & Bridgeman, B. (1995b). Perceptual consequences of ocular lens overshoot
during saccadic eye movements. Vision Research, 35, 2897-2902.
6. Van Rensbergen, J., De Troy, A., Cavegn, D., De Graef, P., van Diepen, P. M. J., & Fias,
W. (1993). The consequences of eye-lens movement during saccades for a stable retinal
image. Poster presented at the Seventh European Conference on Eye Movements.
Durham, UK.
7. K. Matsuda, T. Nagami and S. Yamane, Development of gaze measurement system,
Technical Report of the Institute of Electronics, Information and Communication
Engineers, TL2002-2, 2000.
8. T. Ohno, N. Takegawa, and A. Yoshikawa, Gaze estimation method based on eyeball
shape model, Proc. of the 8
th
Image Sensing Symposium, pp.307-312, 2002.
9. Y. Ebisawa, 3D gaze measurement equipment, Japanese Patent No.2005-198743, 2005.
10. H. Tanaka, S. Hikita, K. Kasai, and A. Takeda, Gaze estimation method based on cornea
curvature center estimation with two light sources, Journal of the Institute of Electronics,
Information and Communication Engineers, 108, 479, MBE2008-128, 117-180, 2009.
11. J. Canny: A Computational Approach to Edge Detection, IEEE Trans. Pattern Analysis
and Machine Intelligence, .8, 6, .679-698, 1986.
12. Intel Crop.: OpenCV (Inte Open Computer Vision Library),
http://www.intel.com/technology/computing/opencv/

Computer Input with Human Eyes-Only Using Two Purkinje Images Which Works in a Real-Time Basis without Calibration

More Related Content

What's hot (18)

Viewers also liked (20)

Similar to Computer Input with Human Eyes-Only Using Two Purkinje Images Which Works in a Real-Time Basis without Calibration (20)

More from Waqas Tariq (11)

Recently uploaded (20)

Computer Input with Human Eyes-Only Using Two Purkinje Images Which Works in a Real-Time Basis without Calibration