![A Vector-based, Multidimensional Scanpath Similarity Measure
Halszka Jarodzka∗ Kenneth Holmqvist†
Knowledge Media Research Center Marcus Nystr¨ m‡
o
Tuebingen, Germany Lund University Humanities Lab
Lund, Sweden
Figure 1: Are these scanpaths similar? Scanpaths from two participants looking at the same stimulus.
Abstract The data emerging from an eye-tracker recording are very informa-
tion dense. Before data can be sensibly interpreted by eye-tracking
A great need exists in many fields of eye-tracking research for a researchers, this rich information must be simplified. For instance,
robust and general method for scanpath comparisons. Current mea- area of interest (AOI) measures replace precise fixation positions
sures either quantize scanpaths in space (string editing measures with imprecise AOI hits, in effect a quantization of space. Attention
like the Levenshtein distance) or in time (measures based on atten- maps and other point-based measures entirely collapse data over
tion maps). This paper proposes a new pairwise scanpath similarity time, losing all sequence information.
measure. Unlike previous measures that either use AOI sequences
or forgo temporal order, the new measure defines scanpaths as a Although previous measures that severely simplify the scanpaths
series of geometric vectors and compares temporally aligned scan- provide valuable results for some research questions, they are not
paths across several dimensions: shape, fixation position, length, sufficient for others. Thus, many researchers stress the urgent
direction, and fixation duration. This approach offers more multi- need to investigate scanpaths [Underwood et al. 2008; Ehmke and
faceted insights to how similar two scanpaths are. Eight fictitious Wilson 2007; Josephson and Holmes 2006; Tzanidou et al. 2005;
scanpath pairs are tested to elucidate the strengths of the new mea- Josephson and Holmes 2002]. In practice, the need to compare and
sure, both in itself and compared to two of the currently most pop- analyze scanpaths often results in considerable manual data anal-
ular measures - the Levenshtein distance and attention map corre- ysis and description of surface features of single scanpath visual-
lation. izations [Ehmke and Wilson 2007; Josephson and Holmes 2006].
Statistical testing of scanpath similarity between groups has been
CR Categories: I.5.3 [Pattern Recognition]: Clustering— considered difficult to the level of unachievable [Tzanidou et al.
Similarity Measures; I.5.4 [Pattern Recognition]: Applications— 2005], and the solutions recently invented are bordering to compu-
Signal Processing J.4 [Social and Behavioral Sciences]: Psychol- tationally intractable [Feusner and Lukoff 2008].
ogy Previous attempts to replace manual analysis with computer al-
gorithms dominantly fall into one of two categories of which the
Keywords: scanpath, sequence analysis, Levenshtein distance, string-edit (a.k.a. the Levenshtein) measure [Levenshtein 1966] is
string edit, vector the most common. It quantizes spatial information by replacing fix-
ations with a character for the AOI they hit. Thus, the representation
1 Introduction of a scanpath is a character string. Similarity is reduced to count-
ing edit operations (insertions, deletions, or substitutions) in pairs
∗ e-mail: h.jarodzka@iwm-kmrc.de of character strings. This measure produces only one value, which
† e-mail:kenneth@lucs.lu.se can be normalized on the longer, the shorter, or on both strings.
‡ marcus.nystrom@humlab.lu.se The attention map measures are a broad class of related measures
which all ignore sequence information. Calculation of similarity
can be done in a number of ways and results in only one value.
In contrast, we propose a measure that represents scanpaths with
Copyright © 2010 by the Association for Computing Machinery, Inc.
geometric vectors. This allows the scanpath to retain a sequence of
Permission to make digital or hard copies of part or all of this work for personal or fixations and saccades and measure similarity using geometry. This
classroom use is granted without fee provided that copies are not made or distributed approach was not used previously. Moreover, a vector represen-
for commercial advantage and that copies bear this notice and the full citation on the tation of scanpaths gives the opportunity to simplify scanpaths in
first page. Copyrights for components of this work owned by others than ACM must be
honored. Abstracting with credit is permitted. To copy otherwise, to republish, to post on
more controlled and flexible ways then previous, cruder methods.
servers, or to redistribute to lists, requires prior specific permission and/or a fee.
Request permissions from Permissions Dept, ACM Inc., fax +1 (212) 869-0481 or e-mail
What the string-edit method and the new measure presented in this
permissions@acm.org. paper have in common is that they align the entities (characters
ETRA 2010, Austin, TX, March 22 – 24, 2010.
© 2010 ACM 978-1-60558-994-7/10/0003 $10.00
211](https://image.slidesharecdn.com/jarodzka-avector-basedmultidimensionalscanpathsimilaritymeasure-100326152700-phpapp02/85/Jarodzka-A-Vector-Based-Multidimensional-Scanpath-Similarity-Measure-1-320.jpg)
![A third, less common way to define AOIs is through clustering of
eye-movement data [Santella and DeCarlo 2004]. Then the AOIs
comprise regions that contain proportionally many fixations from
both scanpaths. The disadvantage is that those AOIs may be rather
used for post-hoc data explorations than for hypothesis testing.
In the string M – M – T – C – C – H – G – M, we have repetitive
AOIs for M and C (‘Crawler’), since our sample scanpath has two
(a) Using gridded AOIs to (b) Semantic AOIs used by successive fixations in each of ‘Main’ and ‘Crawler’. Brandt and
approximate fixation position Josephson and Holmes [2006] Stark [1997] use a variety of the AOI-string representation of scan-
with AOI hits. The first fixation to approximate fixation position
paths that ignores repetitions, or in other words: They represent
is approximated with the AOI with AOI hits. Both first and
dwells in AOIs rather than fixations. The dwell-based AOI-string
A6. last fixation are approximated
with AOI “Main”.
for the same scanpath would be M – T – C – H – G – M. Remov-
ing repetitions, sometimes referred to as compressing the string, is
in fact a sort of clustering method: Several fixations (C – C) are
Figure 2: Gridded vs. semantic AOIs for representing scanpaths. replaced by one dwell (C).
The major advantages of the AOI string representation of scanpaths
is that it retains an approximate sequence of fixations, and is a rough
or fixations/saccades, respetively) in one scanpath with the corre-
representation of the shape, length and direction of the scanpath.
sponding entities in the other scanpath. The representation format
Fixation durations are difficult to represent. The major drawback, is
and the alignment process are closely interlinked, and together de-
the reliance on AOI segmentation, in essence a spatial quantization
termine the nature of the measure.
which necessarily introduces artefacts in the measures relying on it.
It can be argued that the long-term goal for scanpath similarity mea-
sures includes a sensitivity for differences in at least shape1 , posi- 3 Attention maps and other point-based
tion, length, direction and duration between two scanpaths. Two
obvious prerequsites of such measures are, first of all, that the scan-
methods
path retains information along these dimensions, and second, that
the information is used in the calculation of the measures. Without movement forward, can there be a scanpath? The temporal
sequence, the order in which the scan progresses, suggests itself as
a necessary property of scanpaths. Without it, two scanpaths with
2 String-based methods reversed order but otherwise identical would be indistinguishable.
Nevertheless, the attention map measures and others that ignore or-
Several scanpath measures, the most known of which is the Lev- der – such as the Mannan distance measure [Mannan et al. 1996] –
enshtein string-edit measure, represent scanpaths with a string of are often referred to as “scanpath metrics” in literature [Cerf et al.
AOI hits. When using string-based methods, it is important to dis- 2009; Boccignone et al. 2005]. Similarly, in their overview over
tinguish between gridded AOIs and semantic AOIs. Figure 2 shows what they refer to as scanpath metrics, Underwood et al., [2008]
both varieties. The gridded AOIs are constructed by putting a grid discuss both the Mannan distance measure and fixation maps, be-
of equally sized areas across the stimulus, ignoring whatever se- fore settling for the Levenshtein string-edit analysis.
mantics there is in the stimulus. When a scanpath runs over the An attention map is calculated on the basis of a set of points (raw
gridded AOIs, each fixation is replaced by the name of the AOI it data sample or fixations) with no order. Typically, the superposition
hits. For example, the scanpath in figure 2(a) will be represented by of Gaussian functions centred on these points forms a probability
the string A6 – C5 – F0 – I1 – J1 – K2 – I3. landscape and, when represented with colour, a “heat map”. There
Each such AOI name is an approximation of a fixation position are many measures for calculating similarities between two such
point by a whole area, or in other words: We have quantized space. landscapes, heat maps, or sets of points, including simple subtrac-
By definition, quantization introduces noise; a small difference in tion [Wooding 2002], the earth mover distance [Dempere-Marco
gaze pattern would be enough to alter the string, but interestingly, et al. 2006], the correlation coefficient [Ouerhani et al. 2003; Ra-
other small differences can result in the same string. That is, some jashekar et al. 2008], the Kullback-Leibler Divergence [Rajashekar
differences matter, but other differences are ignored. et al. 2004; Tatler et al. 2005], and average landing altitude [Itti
2005]. All these methods reduce similarity to a single scalar.
A division of stimulus space into semantic AOIs adapts to the natu-
Attention maps represent the spatial positions and the scanpath
ral meaningful parts of the stimulus. In figure 2, the semantic AOIs
shape (except sequence). The length and direction of the scanpath
are taken from Josephson and Holmes [2006], who used them to
are ignored, as their calculation would depend on sequence. Fixa-
analyse viewing behaviour on a television screen. Semantic AOIs
tion durations may or may not be included in the representations,
have different sizes, and therefore the approximation of fixation po-
depending on how the attention maps are precisely calculated.
sition can be very coarse. In figure 2, the scanpath example will be
represented with the string M – M – T – C – C – H – G – M (where
M = ‘Main’ etc.). The three M’s in the string represent fixations 4 The proposed method
with very different positions in the stimulus, which means we re-
tain a very coarse position approximation. This can be motivated if 4.1 Representing Scanpaths as Geometrical Vectors
the ‘Main’ AOI is a semantically homogeneous area from the view-
point of the theory, hypothesis and analysis in the study using the We propose to represent the scanpath with vector mathematics. A
measure. Otherwise, very different scanpaths will be represented as vector u = [x, y] is a mathematical entity with direction and length,
equal when using semantic AOIs. and thus suitable for representing an idealized saccade2 . The vec-
tor representation does keep the length and the direction of each
1 Shape in this paper refers to a vector u, and the difference in shape
refers to the difference between two geometric vectors, u − v 2A saccade that takes the shortest distance between two points.
212](https://image.slidesharecdn.com/jarodzka-avector-basedmultidimensionalscanpathsimilaritymeasure-100326152700-phpapp02/85/Jarodzka-A-Vector-Based-Multidimensional-Scanpath-Similarity-Measure-2-320.jpg)
![saccade fully intact. An advantage of this representation is that fix- This type of clustering extends prior methods for clustering of local
ations can be represented inbetween those saccadic vectors. Thus, and global scanpaths [Groner et al. 1984], because saccadic vectors
the position of fixations are the precise start and ending positions are not only clustered according to their distance, but also according
of the saccades. Fixations can of course have durations. Together, to their direction.
fixations and saccades can be used to represent a scanpath, and is
very analogue to the scanpath itself. 4.3 Temporal Alignment of Simplified Scanpaths
The representation of scanpaths with vector mathematics preserves:
In this step the scanpaths are temporally aligned to each other based
1. the shape (vector representation) of the scanpath. on their shape. In order to do that, we used the clustered (simplified)
scanpaths. Thus, we compare two scanpaths that are characterized
2. the length of the scanpath saccades (ignoring saccadic curva- as a series of fixations and vectors.
ture).
Comparison between two scanpaths, S1 = {u1 , u2 , . . . , um } and
3. the direction of the scanpath saccades. S2 = {v1 , v2 , . . . , vn }, is performed in the following steps (illus-
trated in Figure 4):
4. the position of fixations.
1. According to a similarity metric, calculate how similar each
5. the duration of fixations. element i, e.g., saccade amplitude, in one scanpath is com-
pared to each element j in the other scanpath. Let ω(i−1)m+j
In the following sections, we will implement and evaluate the pro-
denote the similarity between elements i and j. Collect the
posed scanpath comparison method using vector representations.
results in a matrix M (i, j).
4.2 Scanpath simplification 2. Create an adjacency matrix A(k, l), {k, l} = 1, 2, . . . , mn
for M (i, j), i.e., define a set of rules of how matrix elements
Besides providing an intuitive description of a scanpath, a vector in M (i, j) are connected. Associate each connection with a
based representation makes it easy to simplify a scanpath, so that it weight ω(i−1)m+j .
retains information that facilitates the alignment of, and comparison 3. Create a graph representation from the adjacency matrix,
between two scanpaths. where matrix elements are called nodes.
For example, a scanpath representation that aims at assessing the 4. Find the shortest path from the node 1 to (the last) node mn
overall shape of a scanpath should not include every single saccadic in the Graph using Dijkstra algorithm [Dijkstra 1959].
vector, especially if a whole series of small saccades looks like a
single point or a single line. String editing techniques for scanpath 5. Align scanpaths along the shortest path.
comparison, for example, are often preceded by simplification pro- In this paper, we always align scanpaths based on their shapes.
cedures where repetitions in letter strings are removed [Brandt and However, the alignment may be performed on other dimensions,
Stark 1997]. This section describes how to cluster a vector based if they are of greater interest in a particular study.
scanpath representation from a local (fine) to a global (coarse) level,
but with the constraint of removing as little vital information about
the scanpaths as possible. Obviously, this is a trade-off; if too much 4.4 Scanpath comparison
is removed, the scanpath looses its meaning, but if too little is re-
moved, the scanpaths may be difficult to align with each other, and For each pair of fixation and saccade vectors in the simplified and
small saccades can become unproportionally influential in the sim- aligned scanpaths, the following measures are computed (average):
ilarity calculations. To meet this trade-off, we present a two-step 1. difference in shape between saccade vectors ui − v j ,
method: First, within a scanpath, several small saccadic vectors
usually occur in one area, which differ in their directions. In this 2. difference in amplitude (length) between saccade vectors
case, the purpose is not to focus on those differences, but rather ||ui − v j ||,
to focus on the fact that the person looked at this area. Thus, any 3. distance between fixations,
group of consecutive saccade vectors {u1 , u2 , . . . , um }, m > 1
with amplitudes smaller than a threshold, Tamp are replaced with a 4. difference in direction (angle) between saccade vectors,
new vector u = u1 + u2 + . . . + um (amplitude-based cluster-
ing). Consequently, very short vectors that have different directions 5. difference in duration between fixations.
are clustered, because they are likely to represent the inspection Consequently, we end up with five measures indicating how similar
of one stimulus feature and therefore do not belong to the overall two scanpaths are along different dimensions. To obtain similarity
shape of the scanpath. Second, within a scanpath several consec- values between [0, 1], the first three measures are normalized by the
utive saccades often have similar directions. Analogous with the screen diagonal. Direction is normalized by π, whereas the differ-
amplitude based clustering, any two consecutive saccades that are ence for each fixation duration is normalized against the maximum
directed at most Tφ radians apart will be added into one larger sac- duration of the two durations being compared.
cade (direction-based clustering). As a consequence, consecutive
vectors having a very similar direction are added, because they do
not by themselves contribute to the overall global shape. 5 Results and Discussion
Fixation durations associated with the merged vectors are still rep- The proposed algorithm is compared to the Levenshtein distance
resented in the simplified scanpath; each fixation at the beginning [Levenshtein 1966] and the correlation coefficient r between atten-
of a merged vector is added to the beginning of a global vector re- tion maps, which are two of the most common measures for scan-
placing it. This way, the scanpaths retain most of the information path comparison. The comparison is made on eight pairs of ficti-
about the dimensions used to evaluate their similarity. Figure 3 tious scanpaths constructed to illustrate where the proposed mea-
illustrates what amplitude and direction based clustering do to a sure could be particularly useful. Prior to computing the Leven-
hypothetic scanpath. shtein distance, fixations are quantized on a 5×5 grid, and the re-
213](https://image.slidesharecdn.com/jarodzka-avector-basedmultidimensionalscanpathsimilaritymeasure-100326152700-phpapp02/85/Jarodzka-A-Vector-Based-Multidimensional-Scanpath-Similarity-Measure-3-320.jpg)
![sulting string of letters is ’compressed’ [Brandt and Stark 1997]. imagery scanpaths to real object inspection scanpaths. Before an
The attention maps (AM) are built from 2D Gaussian functions observer starts to describe the already seen stimulus, s/he might
need some time to organize her/his thoughts before s/he starts the
(x − x0 )2 (y − y0 )2
„ «
description and thus the actual scanpath. The organizing phase is
G(x, y) = exp + (1) hence off-topic to the actual scanpath. Such an off-topic beginning
2σ 2σ
does not occur in the inspection of the stimulus, because it can be
where (x0 , y0 ) represents one fixation position in the scanpath. inspected immediately. Thus, the mental/visual imagery scanpath
σ = 0.10W was set to span 10% of the screen width (W ). To is shifted in time compared to the real stimulus inspection scanpath.
generate an attention map, Gaussian functions from all fixations in A comparable pattern occurs for differences in viewing time or fix-
a scanpath are superimposed. Thus, the attention maps do not take ation number, which easily can occur in scanpaths. Both result in
fixation duration into account. A minimum fixation duration of 40 a different scanpath length. Therefore, the occurrence of different
ms was used in the constructed scanpaths. Furthermore, Tamp and scanpath lengths is almost inevitable, and a very important property
Tφ were set to 0.2W and π/4 deg, respectively. The exact val- of a scanpath similarity measure is to deal with scanpaths of differ-
ues do not critically effect the results in this paper, but should be ent lengths [Josephson and Holmes 2006; Zangemeister and Liman
selected with care when applied to real eye-tracking data. 2007]. Results show that the attention-map-based measure assumes
the two scanpaths to be relatively similar (0.62). In contrast, the
Table 1 shows the results. For all three measures the similarity val- Levenshtein distance confound similarity completely with string
ues are between 0 and 1. However, a high value in the correlation length. That is, a long scanpath is per default dissimilar to a short
coefficient between attention maps corresponds to a high similarity one, even though the shorter is a substring of the longer. Moreover,
between scanpaths, whereas for the remaining two measures a low the string-edit measure has no possibility to deal with a temporal
value corresponds to a high similarity between scanpaths. offset of even one character but, as example four illustrates, judges
such strings as being quite different (0.40). The new measure pro-
Example 1 is composed of two identical scanpaths, an observation
vides a broader perspective on the type of (dis)similarity: the sac-
that all three measures robustly confirm. However, scanpaths may
cade lengths of both scanpaths are highly similar (0.01) whereas
have a similar location, but occur in a reversed order (example 2).
their shapes (0.11), the positions of their fixations (0.16), and the di-
Since only one letter is shared between letter strings in this case,
rections of their saccades (0.12) differ more. For the high difference
a string-editing measure finds such scanpaths to be very different
in fixation durations (0.32) see example 8. As already described
(here: 0.80). In contrast, attention-map-based measures assume
above, the researcher has to decide which difference is meaningful
such scanpaths to be identical (1.00). The new measure provides
for his specific research question.
a more detailed picture of the (dis)similarity of the two scanpaths.
We see the highest similarity for saccade amplitude (0.16), whereas Section 2 discussed the problem of setting AOIs, in particular as
the similarities for shape (0.31), position (0.34), direction (0.26), grids, in detail. Example 5 provides an extreme case, where two
and duration (0.43) – all compared in reversed order – are lower. scanpaths look almost identical, but reside in very different AOIs,
because the fixations are located close to AOI boarders. While the
One application that causes severe problems for current spatial sim- attention-map based measure estimates the scanpaths as being very
ilarity measures is the comparison of visual/mental imagery scan- similar (0.90), the Levenshtein distance estimates them to be to-
paths (example 3). Such scanpaths may be very similar in their tally different (1.00). Thus, the Levenshtein measure is completely
shape by preserving the overall pattern. At the same time, they biased by the definition of AOIs. The new vector-based measure
may introduce some variability and distortion and hence be very instead estimates the scanpaths as very similar on the following
different on a coordinate level [Brandt and Stark 1997; Laeng and dimensions: shape (0.05), position (0.04), length (0.04), direction
Teodorescu 2002; Zangemeister and Liman 2007]. One problem (0.01). The difference that the new measure detects is in the fix-
that may occur in comparing such scanpaths is offset in the spatial ation durations (0.57, for a discussion on differences in durations
dimension. In that, the scanpath that occurred during inspecting the see example 8). Another bias that may occur is an exaggeration
stimulus may be centered, whereas the scanpath that occurred dur- or a miniaturization of one scanpath compared to another (example
ing mental imagery may be shifted to one side of the screen. This 6). In that, the imagery scanpath may be expanded compared to
is due to the fact that the eyes have no spatial reference point while the original stimulus inspection scanpath, or vice versa. Moreover,
looking at a blank screen. Although both scanpaths have identical a mental/visual imagery scanpath may be more locally restricted
shapes, current spatial similarity measures estimate them to be quite compared to global viewing of real pictures [Zangemeister and Li-
different. Results show that both the attention-map-based measure man 2007]. Both common measures estimate the scanpaths as very
(-0.23) and the Levenshtein distance (1.00) estimate the two scan- dissimilar: an attention-map correlation coefficient of r =-0.36 and
paths as very dissimilar. Again, the new measure provides a more a Levenshtein distance of 1.00. While the new measure detects a
multifaceted description: the shape, the length, and the directions dissimilarity, it is far smaller for position (0.28), length (0.32), du-
of both scanpaths are very similar (better than 0.01). What does dif- ration (0.32), and shape (0.32). Most notably, the measure detects
fer in both scanpaths are the positions of their fixations. Now, the a perfect similarity in direction of the saccades (0.01).
researcher must decide whether this difference is meaningful for
his research question (e.g., comparing two persons’ scanpaths on a Example 7 illustrates two scanpaths where long (global) saccades
picture with many closely located details) or not (e.g., in a mental are used to get to a new region, whereafter this region is inspected
imagery study). Moreover, both scanpaths differ in terms of fixa- in more detail using smaller (local) saccades. Intuitively, such typ-
tion duration. Again, the research question should decide whether ical scanpaths are quite similar. However, the two scanpaths are
this observation is relevant (cf. example 8). estimated as very dissimilar by the attention-map-based measure (-
0.05) and the Levenshtein measure (0.76). The new measure better
Another problem that might occur in calculating scanpath similar- captures this intuitive similarity in all dimensions.
ity is temporal offset (cf. example 4). Temporal offset stands for the
fact that a part of one scanpath is very similar to one within another Scanpaths may differ in terms of the duration of certain fixations.
scanpath, however later in time. For instance, by inspecting static While solving a mathematical problem, for instance, consider a per-
pictures one person might orient her/himself longer than another, son that makes two short fixations on a formula, followed by two
but starting with the same inspection strategy after a longer orien- longer fixations on its corresponding graph. Then another person
tation phase. Similar problems occur in comparing mental/visual makes four fixations in the same order, but instead directs the longer
215](https://image.slidesharecdn.com/jarodzka-avector-basedmultidimensionalscanpathsimilaritymeasure-100326152700-phpapp02/85/Jarodzka-A-Vector-Based-Multidimensional-Scanpath-Similarity-Measure-5-320.jpg)
![Metric 1 Equal 2 Reversed 3 Spatial offset 4 Temporal offset
r(AM1 , AM2 ) 1.00 1.00 -0.23 0.62
Levenshtein distance 0.00 0.80 1.00 0.40
Proposed:
Shape 0.00 0.31 0.01 0.11
Position 0.00 0.34 0.15 0.16
Length 0.00 0.16 0.00 0.01
Direction 0.00 0.26 0.00 0.12
Duration 0.00 0.43 0.32 0.32
Metric 5 AOI border problem 6 Scaled 7 Local/Global 8 Duration
r(AM1 , AM2 ) 0.90 -0.36 -0.05 1.00
Levenshtein distance 1.00 1.00 0.76 0
Proposed:
Shape 0.05 0.32 0.29 0.00
Position 0.04 0.28 0.39 0.00
Length 0.04 0.32 0.23 0.00
Direction 0.01 0.01 0.33 0.00
Duration 0.57 0.32 0.33 0.88
Table 1: Similarity between fictitious scanpaths using the 1) correlation between attention maps (r ∈ [−1, 1]) 2) Levenshtein distance and
3) the method proposed in this paper. For the two latter methods, ‘0’ indicates a high similarity. The proposed method outputs differences
with respect to certain aspects of the scanpaths: Shape (difference between saccade vectors), (fixation) position, saccade length, saccade
direction, and fixation duration. All differences are computed on all pairs of aligned saccades and fixations, and only the average is reported.
In all examples, alignment of scanpaths is done according shape. All scanpaths except the reversed one in example 2 start in the upper left
corner.
216](https://image.slidesharecdn.com/jarodzka-avector-basedmultidimensionalscanpathsimilaritymeasure-100326152700-phpapp02/85/Jarodzka-A-Vector-Based-Multidimensional-Scanpath-Similarity-Measure-6-320.jpg)
![fixations toward the formula. The first person is rather studying since most scanpaths are of different lengths. Second, it carries all
the graph, whereas the second person is rather studying the for- the problems that are associated with using AOI-based string input
mula. Such a situation is presented in example 8. The attention- as described above. Another measure uses quantized saccade direc-
map-based measure considers both scanpaths to be exactly similar tion and amplitude rather than stimulus space. Gbadamosi [2000]
(1.00)3 . A string-editing measure may be even more biased; not as well as Zangemeister and Liman [2007] developed a version of
only does it consider the two scanpaths completely similar (0), but the string representation for scanpaths, in which each saccade is
it would still do so if the difference in fixation time increased even represented with a pair of (hexadecimal) numbers where the first
further. The new measure sees the similarity of both scanpaths in number is for direction and the second for amplitude. Such a rep-
shape, position, length, and direction (all 0). However, it detects resentation of scanpaths does not require a segmentation of space
the high difference in fixation durations of both scanpaths (0.88), into AOIs; in fact, it completely abandons position information, and
which may be very important in certain scenarios. instead focusses on representing the overall shape. It is much more
precise than the original AOI string representation, more true to our
subjective perception of scanpath shape, but still based on segmen-
6 Conclusions and Future Work tation and therefore sensitive to quantization effects.
This paper presents an alternative to common measures that assess Still, the new measure proposed in this paper has at least two draw-
the similarity of two scanpaths, like the Levenshtein distance or backs. First, this measure compares only two scanpaths. Some-
correlating attention-maps. The new measure uses a representation times, the overall aim is to compare whole groups of participants
of scanpaths as mathematical vectors. This type of representation with each other. One solution for performing the according sta-
preserves important features of the real scanpath: the overall shape, tistical comparison is delivered by Feusner and Lukoff [2008].
the direction and the amplitude of its saccades, the position and the They present a method to combine many pairwise comparisons in a
duration of its fixations. In a second step the measure simplifies the meaningful and computational efficient way. Second, the new mea-
scanpath representation by amplitude- and direction-based cluster- sure presumes fixations and saccades to occur, however, smooth
ing. The simplified scanpath representation is compared to another pursuit is not handled. Indeed, the representation of scanpaths with
one with regard to their shape. This is done by calculating vector geometrical vectors allows in principle to handle smooth pursuit. In
differences within a recurrence matrix. The found path of least re- such case, smooth pursuit would have to be represented as a series
sistance within this matrix is the basis for further calculations. On of short vectors, which must not be clustered into one long vector.
this path, the position and the duration of the fixations of both scan-
paths are compared. Moreover, the length and the direction of the Despite these drawbacks, the proposed measure is capable of deal-
saccades of both scanpaths are compared. As a result the measure ing with important issues concerning scanpath similarity. However,
delivers five normalized values, one for each scanpath dimension. open questions for future research still remain. One question is
about simplifying the scanpath representation. In this paper, thresh-
This new measure was compared to the currently most common olds for amplitude- and direction-based clustering are set rather ar-
measures: the Levenshtein distance and correlation of attention bitrarily. Of course, a change in these would influence the final
maps. In general, the presented examples demonstrate that the output. It is difficult to decide from a theoretical point of view how
new measure provides more detailed information on the type of to decide the thresholds. Thus, further research that quantifies the
(dis)similarity of two scanpaths according to the five dimensions. influence of different thresholds is required.
In particular, the new measure provides more valid results for spe-
cific cases compared to the other measures. Since the new measure Moreover, the comparison dimensions need to be investigated in
does not require to define AOIs, it does not get biased by fixations depth. Some dimensions are not important for certain tasks and thus
falling into AOI border regions as it is the case for the Levenshtein should play a minor role in the overall judgement. For instance,
distance. Moreover, the new measure is capable of describing the the results from the reversed order example show a relative low
(dis)similarity of spatially shifted or scaled scanpaths. In contrast, similarity between the two scanpaths. However, for certain studies
the other two measures simply estimate such cases as dissimilar. this might not be important. Thus, in further it should be possibile
Furthermore, the new measure deals also with temporal issues. Not to match two scanpaths in the other direction for specific research
only does it take into account fixation durations, but it also deals questions. On the other hand, other dimensions should sometimes
successfully with shifts in time and different scanpath lengths. Note be handled more strictly. Temporal alignment is not always appro-
that the difference in scanpath length is a severe problem, in partic- priate. For instance, in comparing scanpaths on dynamic material,
ular for string-editing. Some measures are often not applicable at like movies. In that, temporal misalignment is a meaningful dif-
all, like the Hamming distance, or “unique assignment” variants of ference between two scanpaths. Imagine a dynamic scene, where
linear distance metrics [Henderson et al. 2007]. several persons are walking across a street. If one participant looks
at person A and another participant looks at person B, both will pro-
It must be mentioned that neither the Levenshtein distance mea- duce scanpaths that are quite similar, but temporally shifted. This
sure nor the correlation of attention maps are the newest or the best temporal shift is, however, meaningful saying that both participants
measures available. We nevertheless chose those two, because they look at different objects. In this case temporal alignment would lead
are currently the most popular ones. Still, several measures exist to almost identical scanpaths, although they are from a semantical
that have solved some of the addressed problems. One attempt to point of view diverse. Thus, the temporal shift is not a bias in this
face the problem of how to take temporal shift into account, is the case. In future, the scanpath similarity measure should allow for
ClustalG method [Wilson et al. 1999]. This algorithm only makes weighting of sub-measures influencing the similarity index.
binary comparison between AOIs: A0 is just as far away from A1
than it is to F6, so A0 = A1 is just as false as A0 = F6. Thereby, the Irrespective of the content of the single dimensions, they need to
scanpaths are temporally aligned based on their most similar part. be investigated further as a total. The measure presented so far de-
Although this measure faces successfully the problem of temporal livers five separate values for five dimensions of the scanpath that
shift, it still has two main disadvantages. First, it does not take dif- can be used independently to compare different aspects of scanpath
ferent scanpath lengths into account, which is a severe drawback, similarity. Alternatively, the five dimensions could be added up
to give one precise value for scanpath similarity. However, in this
3 It should be noted, however, that certain attention map implementations case further research is needed on how those dimensions should be
take fixation duration into account. weighted. Most probably, different kinds of weighting are neces-
217](https://image.slidesharecdn.com/jarodzka-avector-basedmultidimensionalscanpathsimilaritymeasure-100326152700-phpapp02/85/Jarodzka-A-Vector-Based-Multidimensional-Scanpath-Similarity-Measure-7-320.jpg)
Jarodzka A Vector Based, Multidimensional Scanpath Similarity Measure
- 1. A Vector-based, Multidimensional Scanpath Similarity Measure Halszka Jarodzka∗ Kenneth Holmqvist† Knowledge Media Research Center Marcus Nystr¨ m‡ o Tuebingen, Germany Lund University Humanities Lab Lund, Sweden Figure 1: Are these scanpaths similar? Scanpaths from two participants looking at the same stimulus. Abstract The data emerging from an eye-tracker recording are very informa- tion dense. Before data can be sensibly interpreted by eye-tracking A great need exists in many fields of eye-tracking research for a researchers, this rich information must be simplified. For instance, robust and general method for scanpath comparisons. Current mea- area of interest (AOI) measures replace precise fixation positions sures either quantize scanpaths in space (string editing measures with imprecise AOI hits, in effect a quantization of space. Attention like the Levenshtein distance) or in time (measures based on atten- maps and other point-based measures entirely collapse data over tion maps). This paper proposes a new pairwise scanpath similarity time, losing all sequence information. measure. Unlike previous measures that either use AOI sequences or forgo temporal order, the new measure defines scanpaths as a Although previous measures that severely simplify the scanpaths series of geometric vectors and compares temporally aligned scan- provide valuable results for some research questions, they are not paths across several dimensions: shape, fixation position, length, sufficient for others. Thus, many researchers stress the urgent direction, and fixation duration. This approach offers more multi- need to investigate scanpaths [Underwood et al. 2008; Ehmke and faceted insights to how similar two scanpaths are. Eight fictitious Wilson 2007; Josephson and Holmes 2006; Tzanidou et al. 2005; scanpath pairs are tested to elucidate the strengths of the new mea- Josephson and Holmes 2002]. In practice, the need to compare and sure, both in itself and compared to two of the currently most pop- analyze scanpaths often results in considerable manual data anal- ular measures - the Levenshtein distance and attention map corre- ysis and description of surface features of single scanpath visual- lation. izations [Ehmke and Wilson 2007; Josephson and Holmes 2006]. Statistical testing of scanpath similarity between groups has been CR Categories: I.5.3 [Pattern Recognition]: Clustering— considered difficult to the level of unachievable [Tzanidou et al. Similarity Measures; I.5.4 [Pattern Recognition]: Applications— 2005], and the solutions recently invented are bordering to compu- Signal Processing J.4 [Social and Behavioral Sciences]: Psychol- tationally intractable [Feusner and Lukoff 2008]. ogy Previous attempts to replace manual analysis with computer al- gorithms dominantly fall into one of two categories of which the Keywords: scanpath, sequence analysis, Levenshtein distance, string-edit (a.k.a. the Levenshtein) measure [Levenshtein 1966] is string edit, vector the most common. It quantizes spatial information by replacing fix- ations with a character for the AOI they hit. Thus, the representation 1 Introduction of a scanpath is a character string. Similarity is reduced to count- ing edit operations (insertions, deletions, or substitutions) in pairs ∗ e-mail: h.jarodzka@iwm-kmrc.de of character strings. This measure produces only one value, which † e-mail:kenneth@lucs.lu.se can be normalized on the longer, the shorter, or on both strings. ‡ marcus.nystrom@humlab.lu.se The attention map measures are a broad class of related measures which all ignore sequence information. Calculation of similarity can be done in a number of ways and results in only one value. In contrast, we propose a measure that represents scanpaths with Copyright © 2010 by the Association for Computing Machinery, Inc. geometric vectors. This allows the scanpath to retain a sequence of Permission to make digital or hard copies of part or all of this work for personal or fixations and saccades and measure similarity using geometry. This classroom use is granted without fee provided that copies are not made or distributed approach was not used previously. Moreover, a vector represen- for commercial advantage and that copies bear this notice and the full citation on the tation of scanpaths gives the opportunity to simplify scanpaths in first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, to republish, to post on more controlled and flexible ways then previous, cruder methods. servers, or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from Permissions Dept, ACM Inc., fax +1 (212) 869-0481 or e-mail What the string-edit method and the new measure presented in this permissions@acm.org. paper have in common is that they align the entities (characters ETRA 2010, Austin, TX, March 22 – 24, 2010. © 2010 ACM 978-1-60558-994-7/10/0003 $10.00 211
- 2. A third, less common way to define AOIs is through clustering of eye-movement data [Santella and DeCarlo 2004]. Then the AOIs comprise regions that contain proportionally many fixations from both scanpaths. The disadvantage is that those AOIs may be rather used for post-hoc data explorations than for hypothesis testing. In the string M – M – T – C – C – H – G – M, we have repetitive AOIs for M and C (‘Crawler’), since our sample scanpath has two (a) Using gridded AOIs to (b) Semantic AOIs used by successive fixations in each of ‘Main’ and ‘Crawler’. Brandt and approximate fixation position Josephson and Holmes [2006] Stark [1997] use a variety of the AOI-string representation of scan- with AOI hits. The first fixation to approximate fixation position paths that ignores repetitions, or in other words: They represent is approximated with the AOI with AOI hits. Both first and dwells in AOIs rather than fixations. The dwell-based AOI-string A6. last fixation are approximated with AOI “Main”. for the same scanpath would be M – T – C – H – G – M. Remov- ing repetitions, sometimes referred to as compressing the string, is in fact a sort of clustering method: Several fixations (C – C) are Figure 2: Gridded vs. semantic AOIs for representing scanpaths. replaced by one dwell (C). The major advantages of the AOI string representation of scanpaths is that it retains an approximate sequence of fixations, and is a rough or fixations/saccades, respetively) in one scanpath with the corre- representation of the shape, length and direction of the scanpath. sponding entities in the other scanpath. The representation format Fixation durations are difficult to represent. The major drawback, is and the alignment process are closely interlinked, and together de- the reliance on AOI segmentation, in essence a spatial quantization termine the nature of the measure. which necessarily introduces artefacts in the measures relying on it. It can be argued that the long-term goal for scanpath similarity mea- sures includes a sensitivity for differences in at least shape1 , posi- 3 Attention maps and other point-based tion, length, direction and duration between two scanpaths. Two obvious prerequsites of such measures are, first of all, that the scan- methods path retains information along these dimensions, and second, that the information is used in the calculation of the measures. Without movement forward, can there be a scanpath? The temporal sequence, the order in which the scan progresses, suggests itself as a necessary property of scanpaths. Without it, two scanpaths with 2 String-based methods reversed order but otherwise identical would be indistinguishable. Nevertheless, the attention map measures and others that ignore or- Several scanpath measures, the most known of which is the Lev- der – such as the Mannan distance measure [Mannan et al. 1996] – enshtein string-edit measure, represent scanpaths with a string of are often referred to as “scanpath metrics” in literature [Cerf et al. AOI hits. When using string-based methods, it is important to dis- 2009; Boccignone et al. 2005]. Similarly, in their overview over tinguish between gridded AOIs and semantic AOIs. Figure 2 shows what they refer to as scanpath metrics, Underwood et al., [2008] both varieties. The gridded AOIs are constructed by putting a grid discuss both the Mannan distance measure and fixation maps, be- of equally sized areas across the stimulus, ignoring whatever se- fore settling for the Levenshtein string-edit analysis. mantics there is in the stimulus. When a scanpath runs over the An attention map is calculated on the basis of a set of points (raw gridded AOIs, each fixation is replaced by the name of the AOI it data sample or fixations) with no order. Typically, the superposition hits. For example, the scanpath in figure 2(a) will be represented by of Gaussian functions centred on these points forms a probability the string A6 – C5 – F0 – I1 – J1 – K2 – I3. landscape and, when represented with colour, a “heat map”. There Each such AOI name is an approximation of a fixation position are many measures for calculating similarities between two such point by a whole area, or in other words: We have quantized space. landscapes, heat maps, or sets of points, including simple subtrac- By definition, quantization introduces noise; a small difference in tion [Wooding 2002], the earth mover distance [Dempere-Marco gaze pattern would be enough to alter the string, but interestingly, et al. 2006], the correlation coefficient [Ouerhani et al. 2003; Ra- other small differences can result in the same string. That is, some jashekar et al. 2008], the Kullback-Leibler Divergence [Rajashekar differences matter, but other differences are ignored. et al. 2004; Tatler et al. 2005], and average landing altitude [Itti 2005]. All these methods reduce similarity to a single scalar. A division of stimulus space into semantic AOIs adapts to the natu- Attention maps represent the spatial positions and the scanpath ral meaningful parts of the stimulus. In figure 2, the semantic AOIs shape (except sequence). The length and direction of the scanpath are taken from Josephson and Holmes [2006], who used them to are ignored, as their calculation would depend on sequence. Fixa- analyse viewing behaviour on a television screen. Semantic AOIs tion durations may or may not be included in the representations, have different sizes, and therefore the approximation of fixation po- depending on how the attention maps are precisely calculated. sition can be very coarse. In figure 2, the scanpath example will be represented with the string M – M – T – C – C – H – G – M (where M = ‘Main’ etc.). The three M’s in the string represent fixations 4 The proposed method with very different positions in the stimulus, which means we re- tain a very coarse position approximation. This can be motivated if 4.1 Representing Scanpaths as Geometrical Vectors the ‘Main’ AOI is a semantically homogeneous area from the view- point of the theory, hypothesis and analysis in the study using the We propose to represent the scanpath with vector mathematics. A measure. Otherwise, very different scanpaths will be represented as vector u = [x, y] is a mathematical entity with direction and length, equal when using semantic AOIs. and thus suitable for representing an idealized saccade2 . The vec- tor representation does keep the length and the direction of each 1 Shape in this paper refers to a vector u, and the difference in shape refers to the difference between two geometric vectors, u − v 2A saccade that takes the shortest distance between two points. 212
- 3. saccade fully intact. An advantage of this representation is that fix- This type of clustering extends prior methods for clustering of local ations can be represented inbetween those saccadic vectors. Thus, and global scanpaths [Groner et al. 1984], because saccadic vectors the position of fixations are the precise start and ending positions are not only clustered according to their distance, but also according of the saccades. Fixations can of course have durations. Together, to their direction. fixations and saccades can be used to represent a scanpath, and is very analogue to the scanpath itself. 4.3 Temporal Alignment of Simplified Scanpaths The representation of scanpaths with vector mathematics preserves: In this step the scanpaths are temporally aligned to each other based 1. the shape (vector representation) of the scanpath. on their shape. In order to do that, we used the clustered (simplified) scanpaths. Thus, we compare two scanpaths that are characterized 2. the length of the scanpath saccades (ignoring saccadic curva- as a series of fixations and vectors. ture). Comparison between two scanpaths, S1 = {u1 , u2 , . . . , um } and 3. the direction of the scanpath saccades. S2 = {v1 , v2 , . . . , vn }, is performed in the following steps (illus- trated in Figure 4): 4. the position of fixations. 1. According to a similarity metric, calculate how similar each 5. the duration of fixations. element i, e.g., saccade amplitude, in one scanpath is com- pared to each element j in the other scanpath. Let ω(i−1)m+j In the following sections, we will implement and evaluate the pro- denote the similarity between elements i and j. Collect the posed scanpath comparison method using vector representations. results in a matrix M (i, j). 4.2 Scanpath simplification 2. Create an adjacency matrix A(k, l), {k, l} = 1, 2, . . . , mn for M (i, j), i.e., define a set of rules of how matrix elements Besides providing an intuitive description of a scanpath, a vector in M (i, j) are connected. Associate each connection with a based representation makes it easy to simplify a scanpath, so that it weight ω(i−1)m+j . retains information that facilitates the alignment of, and comparison 3. Create a graph representation from the adjacency matrix, between two scanpaths. where matrix elements are called nodes. For example, a scanpath representation that aims at assessing the 4. Find the shortest path from the node 1 to (the last) node mn overall shape of a scanpath should not include every single saccadic in the Graph using Dijkstra algorithm [Dijkstra 1959]. vector, especially if a whole series of small saccades looks like a single point or a single line. String editing techniques for scanpath 5. Align scanpaths along the shortest path. comparison, for example, are often preceded by simplification pro- In this paper, we always align scanpaths based on their shapes. cedures where repetitions in letter strings are removed [Brandt and However, the alignment may be performed on other dimensions, Stark 1997]. This section describes how to cluster a vector based if they are of greater interest in a particular study. scanpath representation from a local (fine) to a global (coarse) level, but with the constraint of removing as little vital information about the scanpaths as possible. Obviously, this is a trade-off; if too much 4.4 Scanpath comparison is removed, the scanpath looses its meaning, but if too little is re- moved, the scanpaths may be difficult to align with each other, and For each pair of fixation and saccade vectors in the simplified and small saccades can become unproportionally influential in the sim- aligned scanpaths, the following measures are computed (average): ilarity calculations. To meet this trade-off, we present a two-step 1. difference in shape between saccade vectors ui − v j , method: First, within a scanpath, several small saccadic vectors usually occur in one area, which differ in their directions. In this 2. difference in amplitude (length) between saccade vectors case, the purpose is not to focus on those differences, but rather ||ui − v j ||, to focus on the fact that the person looked at this area. Thus, any 3. distance between fixations, group of consecutive saccade vectors {u1 , u2 , . . . , um }, m > 1 with amplitudes smaller than a threshold, Tamp are replaced with a 4. difference in direction (angle) between saccade vectors, new vector u = u1 + u2 + . . . + um (amplitude-based cluster- ing). Consequently, very short vectors that have different directions 5. difference in duration between fixations. are clustered, because they are likely to represent the inspection Consequently, we end up with five measures indicating how similar of one stimulus feature and therefore do not belong to the overall two scanpaths are along different dimensions. To obtain similarity shape of the scanpath. Second, within a scanpath several consec- values between [0, 1], the first three measures are normalized by the utive saccades often have similar directions. Analogous with the screen diagonal. Direction is normalized by π, whereas the differ- amplitude based clustering, any two consecutive saccades that are ence for each fixation duration is normalized against the maximum directed at most Tφ radians apart will be added into one larger sac- duration of the two durations being compared. cade (direction-based clustering). As a consequence, consecutive vectors having a very similar direction are added, because they do not by themselves contribute to the overall global shape. 5 Results and Discussion Fixation durations associated with the merged vectors are still rep- The proposed algorithm is compared to the Levenshtein distance resented in the simplified scanpath; each fixation at the beginning [Levenshtein 1966] and the correlation coefficient r between atten- of a merged vector is added to the beginning of a global vector re- tion maps, which are two of the most common measures for scan- placing it. This way, the scanpaths retain most of the information path comparison. The comparison is made on eight pairs of ficti- about the dimensions used to evaluate their similarity. Figure 3 tious scanpaths constructed to illustrate where the proposed mea- illustrates what amplitude and direction based clustering do to a sure could be particularly useful. Prior to computing the Leven- hypothetic scanpath. shtein distance, fixations are quantized on a 5×5 grid, and the re- 213
- 4. (a) Original scanpath (b) Simplified scanpath Figure 3: Example of scanpath simplification. Notice how saccade vectors with similar direction and/or short saccade vectors at similar positions are merged. Fixation durations associated with small, merged vectors are added to the beginning of a new, larger vector. Scanpath 2 Scanpath 2 Scanpath 1 1 2 1 2 3 Scanpath 1 (a) Hypothetical scanpaths (b) Matrix M (i, j) with six elements and weights indicating the similar- ity between saccade vectors. Allowed transitions (adjacency indicator) are indicated by smaller arrows. Node 1 1.1928 0.75599 0.43522 Node 2 Node 4 0.16924 0.43522 0.43522 0.42922 Node 3 Node 5 0.42922 0.42922 Node 6 (c) Adjacency matrix, A(k, l) (d) Graph build according to the adjacency matrix with the shortest path highlighted Figure 4: General principle of the proposed method for scanpath comparison. (a) The scanpaths. (b) A matrix where adjacency is indicated by small arrows. Each matrix element is a node i, which contains a value ωi indicating the similarity between two saccade vectors. To align two scanpaths, Dijkstra algorithm is used the find the minimum cost path from node i = 1 to node i = 6. As a result, all saccade vectors in one scanpath are matched with saccade vectors in the other scanpath. Aligned saccade vectors (and fixation) can be then be compared across different features. The similarity metric in this example was saccade vector difference (i.e, ‘shape’). 214
- 5. sulting string of letters is ’compressed’ [Brandt and Stark 1997]. imagery scanpaths to real object inspection scanpaths. Before an The attention maps (AM) are built from 2D Gaussian functions observer starts to describe the already seen stimulus, s/he might need some time to organize her/his thoughts before s/he starts the (x − x0 )2 (y − y0 )2 „ « description and thus the actual scanpath. The organizing phase is G(x, y) = exp + (1) hence off-topic to the actual scanpath. Such an off-topic beginning 2σ 2σ does not occur in the inspection of the stimulus, because it can be where (x0 , y0 ) represents one fixation position in the scanpath. inspected immediately. Thus, the mental/visual imagery scanpath σ = 0.10W was set to span 10% of the screen width (W ). To is shifted in time compared to the real stimulus inspection scanpath. generate an attention map, Gaussian functions from all fixations in A comparable pattern occurs for differences in viewing time or fix- a scanpath are superimposed. Thus, the attention maps do not take ation number, which easily can occur in scanpaths. Both result in fixation duration into account. A minimum fixation duration of 40 a different scanpath length. Therefore, the occurrence of different ms was used in the constructed scanpaths. Furthermore, Tamp and scanpath lengths is almost inevitable, and a very important property Tφ were set to 0.2W and π/4 deg, respectively. The exact val- of a scanpath similarity measure is to deal with scanpaths of differ- ues do not critically effect the results in this paper, but should be ent lengths [Josephson and Holmes 2006; Zangemeister and Liman selected with care when applied to real eye-tracking data. 2007]. Results show that the attention-map-based measure assumes the two scanpaths to be relatively similar (0.62). In contrast, the Table 1 shows the results. For all three measures the similarity val- Levenshtein distance confound similarity completely with string ues are between 0 and 1. However, a high value in the correlation length. That is, a long scanpath is per default dissimilar to a short coefficient between attention maps corresponds to a high similarity one, even though the shorter is a substring of the longer. Moreover, between scanpaths, whereas for the remaining two measures a low the string-edit measure has no possibility to deal with a temporal value corresponds to a high similarity between scanpaths. offset of even one character but, as example four illustrates, judges such strings as being quite different (0.40). The new measure pro- Example 1 is composed of two identical scanpaths, an observation vides a broader perspective on the type of (dis)similarity: the sac- that all three measures robustly confirm. However, scanpaths may cade lengths of both scanpaths are highly similar (0.01) whereas have a similar location, but occur in a reversed order (example 2). their shapes (0.11), the positions of their fixations (0.16), and the di- Since only one letter is shared between letter strings in this case, rections of their saccades (0.12) differ more. For the high difference a string-editing measure finds such scanpaths to be very different in fixation durations (0.32) see example 8. As already described (here: 0.80). In contrast, attention-map-based measures assume above, the researcher has to decide which difference is meaningful such scanpaths to be identical (1.00). The new measure provides for his specific research question. a more detailed picture of the (dis)similarity of the two scanpaths. We see the highest similarity for saccade amplitude (0.16), whereas Section 2 discussed the problem of setting AOIs, in particular as the similarities for shape (0.31), position (0.34), direction (0.26), grids, in detail. Example 5 provides an extreme case, where two and duration (0.43) – all compared in reversed order – are lower. scanpaths look almost identical, but reside in very different AOIs, because the fixations are located close to AOI boarders. While the One application that causes severe problems for current spatial sim- attention-map based measure estimates the scanpaths as being very ilarity measures is the comparison of visual/mental imagery scan- similar (0.90), the Levenshtein distance estimates them to be to- paths (example 3). Such scanpaths may be very similar in their tally different (1.00). Thus, the Levenshtein measure is completely shape by preserving the overall pattern. At the same time, they biased by the definition of AOIs. The new vector-based measure may introduce some variability and distortion and hence be very instead estimates the scanpaths as very similar on the following different on a coordinate level [Brandt and Stark 1997; Laeng and dimensions: shape (0.05), position (0.04), length (0.04), direction Teodorescu 2002; Zangemeister and Liman 2007]. One problem (0.01). The difference that the new measure detects is in the fix- that may occur in comparing such scanpaths is offset in the spatial ation durations (0.57, for a discussion on differences in durations dimension. In that, the scanpath that occurred during inspecting the see example 8). Another bias that may occur is an exaggeration stimulus may be centered, whereas the scanpath that occurred dur- or a miniaturization of one scanpath compared to another (example ing mental imagery may be shifted to one side of the screen. This 6). In that, the imagery scanpath may be expanded compared to is due to the fact that the eyes have no spatial reference point while the original stimulus inspection scanpath, or vice versa. Moreover, looking at a blank screen. Although both scanpaths have identical a mental/visual imagery scanpath may be more locally restricted shapes, current spatial similarity measures estimate them to be quite compared to global viewing of real pictures [Zangemeister and Li- different. Results show that both the attention-map-based measure man 2007]. Both common measures estimate the scanpaths as very (-0.23) and the Levenshtein distance (1.00) estimate the two scan- dissimilar: an attention-map correlation coefficient of r =-0.36 and paths as very dissimilar. Again, the new measure provides a more a Levenshtein distance of 1.00. While the new measure detects a multifaceted description: the shape, the length, and the directions dissimilarity, it is far smaller for position (0.28), length (0.32), du- of both scanpaths are very similar (better than 0.01). What does dif- ration (0.32), and shape (0.32). Most notably, the measure detects fer in both scanpaths are the positions of their fixations. Now, the a perfect similarity in direction of the saccades (0.01). researcher must decide whether this difference is meaningful for his research question (e.g., comparing two persons’ scanpaths on a Example 7 illustrates two scanpaths where long (global) saccades picture with many closely located details) or not (e.g., in a mental are used to get to a new region, whereafter this region is inspected imagery study). Moreover, both scanpaths differ in terms of fixa- in more detail using smaller (local) saccades. Intuitively, such typ- tion duration. Again, the research question should decide whether ical scanpaths are quite similar. However, the two scanpaths are this observation is relevant (cf. example 8). estimated as very dissimilar by the attention-map-based measure (- 0.05) and the Levenshtein measure (0.76). The new measure better Another problem that might occur in calculating scanpath similar- captures this intuitive similarity in all dimensions. ity is temporal offset (cf. example 4). Temporal offset stands for the fact that a part of one scanpath is very similar to one within another Scanpaths may differ in terms of the duration of certain fixations. scanpath, however later in time. For instance, by inspecting static While solving a mathematical problem, for instance, consider a per- pictures one person might orient her/himself longer than another, son that makes two short fixations on a formula, followed by two but starting with the same inspection strategy after a longer orien- longer fixations on its corresponding graph. Then another person tation phase. Similar problems occur in comparing mental/visual makes four fixations in the same order, but instead directs the longer 215
- 6. Metric 1 Equal 2 Reversed 3 Spatial offset 4 Temporal offset r(AM1 , AM2 ) 1.00 1.00 -0.23 0.62 Levenshtein distance 0.00 0.80 1.00 0.40 Proposed: Shape 0.00 0.31 0.01 0.11 Position 0.00 0.34 0.15 0.16 Length 0.00 0.16 0.00 0.01 Direction 0.00 0.26 0.00 0.12 Duration 0.00 0.43 0.32 0.32 Metric 5 AOI border problem 6 Scaled 7 Local/Global 8 Duration r(AM1 , AM2 ) 0.90 -0.36 -0.05 1.00 Levenshtein distance 1.00 1.00 0.76 0 Proposed: Shape 0.05 0.32 0.29 0.00 Position 0.04 0.28 0.39 0.00 Length 0.04 0.32 0.23 0.00 Direction 0.01 0.01 0.33 0.00 Duration 0.57 0.32 0.33 0.88 Table 1: Similarity between fictitious scanpaths using the 1) correlation between attention maps (r ∈ [−1, 1]) 2) Levenshtein distance and 3) the method proposed in this paper. For the two latter methods, ‘0’ indicates a high similarity. The proposed method outputs differences with respect to certain aspects of the scanpaths: Shape (difference between saccade vectors), (fixation) position, saccade length, saccade direction, and fixation duration. All differences are computed on all pairs of aligned saccades and fixations, and only the average is reported. In all examples, alignment of scanpaths is done according shape. All scanpaths except the reversed one in example 2 start in the upper left corner. 216
- 7. fixations toward the formula. The first person is rather studying since most scanpaths are of different lengths. Second, it carries all the graph, whereas the second person is rather studying the for- the problems that are associated with using AOI-based string input mula. Such a situation is presented in example 8. The attention- as described above. Another measure uses quantized saccade direc- map-based measure considers both scanpaths to be exactly similar tion and amplitude rather than stimulus space. Gbadamosi [2000] (1.00)3 . A string-editing measure may be even more biased; not as well as Zangemeister and Liman [2007] developed a version of only does it consider the two scanpaths completely similar (0), but the string representation for scanpaths, in which each saccade is it would still do so if the difference in fixation time increased even represented with a pair of (hexadecimal) numbers where the first further. The new measure sees the similarity of both scanpaths in number is for direction and the second for amplitude. Such a rep- shape, position, length, and direction (all 0). However, it detects resentation of scanpaths does not require a segmentation of space the high difference in fixation durations of both scanpaths (0.88), into AOIs; in fact, it completely abandons position information, and which may be very important in certain scenarios. instead focusses on representing the overall shape. It is much more precise than the original AOI string representation, more true to our subjective perception of scanpath shape, but still based on segmen- 6 Conclusions and Future Work tation and therefore sensitive to quantization effects. This paper presents an alternative to common measures that assess Still, the new measure proposed in this paper has at least two draw- the similarity of two scanpaths, like the Levenshtein distance or backs. First, this measure compares only two scanpaths. Some- correlating attention-maps. The new measure uses a representation times, the overall aim is to compare whole groups of participants of scanpaths as mathematical vectors. This type of representation with each other. One solution for performing the according sta- preserves important features of the real scanpath: the overall shape, tistical comparison is delivered by Feusner and Lukoff [2008]. the direction and the amplitude of its saccades, the position and the They present a method to combine many pairwise comparisons in a duration of its fixations. In a second step the measure simplifies the meaningful and computational efficient way. Second, the new mea- scanpath representation by amplitude- and direction-based cluster- sure presumes fixations and saccades to occur, however, smooth ing. The simplified scanpath representation is compared to another pursuit is not handled. Indeed, the representation of scanpaths with one with regard to their shape. This is done by calculating vector geometrical vectors allows in principle to handle smooth pursuit. In differences within a recurrence matrix. The found path of least re- such case, smooth pursuit would have to be represented as a series sistance within this matrix is the basis for further calculations. On of short vectors, which must not be clustered into one long vector. this path, the position and the duration of the fixations of both scan- paths are compared. Moreover, the length and the direction of the Despite these drawbacks, the proposed measure is capable of deal- saccades of both scanpaths are compared. As a result the measure ing with important issues concerning scanpath similarity. However, delivers five normalized values, one for each scanpath dimension. open questions for future research still remain. One question is about simplifying the scanpath representation. In this paper, thresh- This new measure was compared to the currently most common olds for amplitude- and direction-based clustering are set rather ar- measures: the Levenshtein distance and correlation of attention bitrarily. Of course, a change in these would influence the final maps. In general, the presented examples demonstrate that the output. It is difficult to decide from a theoretical point of view how new measure provides more detailed information on the type of to decide the thresholds. Thus, further research that quantifies the (dis)similarity of two scanpaths according to the five dimensions. influence of different thresholds is required. In particular, the new measure provides more valid results for spe- cific cases compared to the other measures. Since the new measure Moreover, the comparison dimensions need to be investigated in does not require to define AOIs, it does not get biased by fixations depth. Some dimensions are not important for certain tasks and thus falling into AOI border regions as it is the case for the Levenshtein should play a minor role in the overall judgement. For instance, distance. Moreover, the new measure is capable of describing the the results from the reversed order example show a relative low (dis)similarity of spatially shifted or scaled scanpaths. In contrast, similarity between the two scanpaths. However, for certain studies the other two measures simply estimate such cases as dissimilar. this might not be important. Thus, in further it should be possibile Furthermore, the new measure deals also with temporal issues. Not to match two scanpaths in the other direction for specific research only does it take into account fixation durations, but it also deals questions. On the other hand, other dimensions should sometimes successfully with shifts in time and different scanpath lengths. Note be handled more strictly. Temporal alignment is not always appro- that the difference in scanpath length is a severe problem, in partic- priate. For instance, in comparing scanpaths on dynamic material, ular for string-editing. Some measures are often not applicable at like movies. In that, temporal misalignment is a meaningful dif- all, like the Hamming distance, or “unique assignment” variants of ference between two scanpaths. Imagine a dynamic scene, where linear distance metrics [Henderson et al. 2007]. several persons are walking across a street. If one participant looks at person A and another participant looks at person B, both will pro- It must be mentioned that neither the Levenshtein distance mea- duce scanpaths that are quite similar, but temporally shifted. This sure nor the correlation of attention maps are the newest or the best temporal shift is, however, meaningful saying that both participants measures available. We nevertheless chose those two, because they look at different objects. In this case temporal alignment would lead are currently the most popular ones. Still, several measures exist to almost identical scanpaths, although they are from a semantical that have solved some of the addressed problems. One attempt to point of view diverse. Thus, the temporal shift is not a bias in this face the problem of how to take temporal shift into account, is the case. In future, the scanpath similarity measure should allow for ClustalG method [Wilson et al. 1999]. This algorithm only makes weighting of sub-measures influencing the similarity index. binary comparison between AOIs: A0 is just as far away from A1 than it is to F6, so A0 = A1 is just as false as A0 = F6. Thereby, the Irrespective of the content of the single dimensions, they need to scanpaths are temporally aligned based on their most similar part. be investigated further as a total. The measure presented so far de- Although this measure faces successfully the problem of temporal livers five separate values for five dimensions of the scanpath that shift, it still has two main disadvantages. First, it does not take dif- can be used independently to compare different aspects of scanpath ferent scanpath lengths into account, which is a severe drawback, similarity. Alternatively, the five dimensions could be added up to give one precise value for scanpath similarity. However, in this 3 It should be noted, however, that certain attention map implementations case further research is needed on how those dimensions should be take fixation duration into account. weighted. Most probably, different kinds of weighting are neces- 217
- 8. sary for different applications. Still, the five dimensions in separate world wide web. In ETRA ’02: Proceedings of the 2002 sympo- allow for the application of multilevel analyses. Another impor- sium on Eye tracking research & applications, ACM, New York, tant issue is the question, whether the dimensions are independent NY, USA, 43–49. and if not, how they relate to each other. This should be investi- gated in further research by calculating the measure on real data. J OSEPHSON , S., AND H OLMES , M. E. 2006. Clutter or con- Thereupon, a factorial analysis would deliver the dependencies of tent? how on-screen enhancements affect how tv viewers scan the dimensions. Again, those dependencies might differ between and what they learn. In ETRA ’06: Proceedings of the 2006 task and/or stimuli types. Consequently, in future actual experi- symposium on Eye tracking research & applications, ACM, New ments must be conducted where the differences in performance be- York, NY, USA, 155–162. tween the new measure and established metrics are presented based L AENG , B., AND T EODORESCU , D.-S. 2002. Eye scanpaths dur- on real data of a number of subjects on several stimuli types. ing visual imagery reenact those of perception of the same vi- sual scene. Cognitive Science: A Multidisciplinary Journal 26, References 2 (March), 207–231. L EVENSHTEIN , V. 1966. Binary codes capable of correcting dele- B OCCIGNONE , G., C AGGIANO , V., M ARCELLI , A., NAPOLE - tions, insertions and reversals. Soviet Physice – Doklady 10, TANO , P., AND D I F IORE , G. 2005. An architectural model 707–710. for combining spatial-based and object-based information for at- tentive video analysis. In CAMP ’05: Proceedings of the Seventh M ANNAN , S., RUDDOCK , K., AND W OODING , D. 1996. The International Workshop on Computer Architecture for Machine relationship between the locations of spatial features and those Perception, 116–121. of fixations made during visual examination of briefly presented images. Spatial Vision 10, 3, 165–188. B RANDT, S. A., AND S TARK , L. W. 1997. Spontaneous eye movements during visual imagery reflect the content of the vi- ¨ O UERHANI , N., VON WARTBURG , R., H UGLI , H., AND M URI , ¨ sual scene. Journal of Cognitive Neuroscience 9, 1, 27–38. R. M. 2003. Empirical validation of the saliency-based model of visual attention. Electronic Letters on Computer Vision and C ERF, M., H AREL , J., H UTH , A., E INHAEUSER , W., AND KOCH , Image Analysis 3, 1, 13–24. C. 2009. Decoding what people see from where they look: pre- dicting visual stimuli from scanpaths. Lecture Notes in Artifical R AJASHEKAR , U., C ORMACK , L., AND B OVIK , A. 2004. Point of Intelligence 5395, 15–26. gaze analysis reveals visual search strategies. In Human Vision and Electronic Imaging, vol. 5292, Human Vision and Electronic D EMPERE -M ARCO , L., H U , X.-P., E LLIS , S. M., H ANSELL , Imaging, 296–306. D. M., AND YANG , G.-Z. 2006. Analysis of visual search patterns with emd metric in normalized anatomical space. IEEE R AJASHEKAR , U., VAN DER L INDE , I., B OVIK , A., AND C OR - Trans. Medical Imaging 25, 8 (Aug.), 1011–1021. MACK , L. 2008. Gaffe: A gaze-attentive fixation finding engine. IEEE Trans. Image Processing 17, 4, 564–573. D IJKSTRA , E. 1959. A note on two problems in connexion with graphs. Numerische mathematik 1, 1, 269–271. S ANTELLA , A., AND D E C ARLO , D. 2004. Robust clustering of eye movement recordings for quantification of visual interest. In E HMKE , C., AND W ILSON , C. 2007. Identifying web usability Proceedings of the 2004 symposium on Eye tracking research & issues from eye-tracking data. In People and Computers XXI – applications, vol. 22, Citeseer, 27–34. HCI. . . but not as we know it: Proceedings of HCI 2007, BCS TATLER , B., BADDELEY, R., AND G ILCHRIST, I. 2005. Visual Press, London, P. of HCI-07, Ed. correlates of fixation selection: effects of scale and time. Vision F EUSNER , M., AND L UKOFF , B. 2008. Testing for statistically sig- Research 45, 5, 643 – 659. nificant differences between groups of scan patterns. In ETRA T ZANIDOU , E., M INOCHA , S., AND P ETRE , M. 2005. Apply- ’08: Proceedings of the 2008 symposium on Eye tracking re- ing eye tracking for usability evaluations of e-commerce sites. search applications, ACM, New York, NY, USA, 43–46. In workshop on ’Commercial Uses of Eye tracking’ held at the G BADAMOSI , J. 2000. Visual Imagery und Mikro-Sakkaden bei 19th British HCI Group Annual Conference, Napier University, Hemianopsie-Patienten. PhD thesis, Fachbereich Medizin der Edinburgh. Universit¨ t Hamburg. a U NDERWOOD , G., H UMPHREY, K., AND F OULSHAM , T. 2008. G RONER , R., WALDER , F., AND G RONER , M. 1984. Looking at HCI and Usability for Education and Work, vol. Volume faces: Local and global aspects of scanpaths. In Theoretical and 5298/2008. Springer Berlin / Heidelberg, ch. Knowledge-Based Applied Aspects of Eye Movement Research, Elsevier Science Patterns of Remembering: Eye Movement Scanpaths Reflect Publishers B.V., North-Holland, A. G. Gale and F. Johnson, Eds., Domain Experience, 125–144. 523–533. W ILSON , C., H ARVEY, A., AND T HOMPSON , J. 1999. Clustalg: H ENDERSON , J. M., B ROCKMOLE , J. R., AND M ACK , M. S. C. Software for analysis of activities and sequential events. In M. L. 2007. Visual saliency does not account for eye movements IATUR Conference Proceedings. during search in real-world scenes. In Eye Movement Research: W OODING , D. 2002. Fixation maps: quantifying eye-movement Insights into Mind and Brain, Elsevier: Oxford, R. van Gom- traces. In Eye Tracking Research & Applications, ACM Press, pel R., M. Fischer, W. Murray, and R. W. Hill, Eds., 537–562. New York, New Orleans, LA, 31–36. I TTI , L. 2005. Quantifying the contribution of low-level saliency Z ANGEMEISTER , W. H., AND L IMAN , T. 2007. Foveal versus to human eye movements in dynamic scenes. Visual Cognition parafoveal scanpaths of visual imagery in virtual hemianopic 12, 6, 1093–1123. subjects. Computers in Biology and Medicine 37, 7, 975 – 982. J OSEPHSON , S., AND H OLMES , M. E. 2002. Visual attention Vision and Movement in Man and Machines. to repeated internet images: testing the scanpath theory on the 218