SlideShare a Scribd company logo
Visual Scanpath Representation
                                                            Joseph H. Goldberg and Jonathan I. Helfman
                                                                   Applications User Experience
                                                                           Oracle USA
                                                        joe.goldberg@oracle.com; jon.helfman@oracle.com

Abstract                                                                                           attention at particular positions, and saccades, rapid eye move-
                                                                                                   ments with suppressed visual perception. Fixation positions are
Eye tracking scanpaths contain information about how people                                        specified with respect to the background stimulus image or dis-
see, but traditional tangled, overlapping scanpath representations                                 play that was visible when the samples were collected. A se-
provide little insight about scanning strategies. The present work                                 quence of consecutive fixations and saccades results in a scan-
describes and extends several compact visual scanpath represen-                                    path – a trace through time and space that may overlap itself.
tations that can provide additional insight about individual and                                   Scanpaths, along with other ocular-based metrics, may indicate
aggregate/multiple scanning strategies. Three categories of rep-                                   higher cognitive strategies and states [e.g., Marshall 2007; Hor-
resentations are introduced: (1) Scaled traces are small images of                                 nof and Halverson 2003]. A challenge for the researcher is to
scanpaths as connected saccades, allowing the comparison of                                        compare multiple scanpaths and quickly comprehend each par-
relative fixation densities and distributions of saccades. (2) Time                                ticipant’s scanning strategy in each tested condition.
expansions, substituting ordinal position for either the scan-
path’s x or y-coordinates, can uncover otherwise subtle horizon-                                   Typical visual representations of scanpaths use circles to repre-
tal or vertical reversals in visual scanning. (3) Radial plots repre-                              sent fixations and lines to represent saccades. Figure 1 displays a
sent scanpaths as a set of radial arms about an origin, with each                                  hypothetical scanpath with 5 fixations and 4 saccades. A scan-
arm representing saccade counts or lengths within a binned set                                     path may be replayed using animation, but to show a scanpath as
of absolute or relative angles. Radial plots can convey useful                                     a static image, fixation duration is typically coded by circle ra-
shape characteristics of scanpaths, and can provide a basis for                                    dius.
new metrics. Nine different prototype scanning strategies were
represented by these plots, then heuristics were developed to
classify the major strategies. The heuristics were subsequently
applied to real scanpath data, to identify strategy trends. Future
work will further automate the identification of scanning strate-
gies to provide researchers with a tool to uncover and diagnose
scanning-related challenges.

CR Categories: H.1.2. User/Machine Systems: Software Psy-
chology, H.5.2. User Interfaces: Graphical User Interfaces.                                         Figure 1. A hypothetical scanpath with 5 fixations and 4 sac-
                                                                                                     cades, showing fixation order and varying saccade lengths.
Keywords: Eye Tracking, Usability Evaluation, Scanpath,
Scanning Strategy, Visualization                                                                   Actual scanpath records are usually quite complex, and can be
                                                                                                   difficult to interpret and compare. Figure 2 shows a scanpath
                                                                                                   from a single observer trying to locate content within the lower
1       Introduction                                                                               right hand area of a web page. The scanpath visited multiple
                                                                                                   content columns as well as the search area before locating the
It has been said that proper representation of a problem is the                                    content. Frequent crossing of saccades and revisiting of loca-
single most important step towards its eventual solution. The                                      tions contribute to scanpath complexity.
problem we are investigating is the comparison of visual scan-
ning strategies between individuals, groups, and conditions, in                                    1.2     Scanpath Comparison
order to make inferences for improving the visual design of
software and digital media. This paper introduces and extends                                      Numerical techniques have been presented for summarizing and
several representations for scanpaths, then demonstrates how                                       comparing scanpaths.. The length of a scanpath can be a meas-
these representations can be used to develop heuristics and met-                                   ure of productivity, and average fixation duration may measure
rics for detecting and labeling scanning strategies.                                               of cognitive complexity [Goldberg and Kotval 1999]. The sum
                                                                                                   and distribution of the inter-fixation angles within a scanpath
1.1        Scanpaths                                                                               have been proposed as a measure of efficiency on a task, with
                                                                                                   simple, direct scanpaths signaling higher efficiency [Goldberg
Eye tracking methods typically sample gaze locations at 50-120                                     and Kotval 1998]. String editing distance computes the mini-
Hz. Samples are then reduced to fixations, periods of visual                                       mum number of editing steps required to transform one scanpath
Copyright © 2010 by the Association for Computing Machinery, Inc.
                                                                                                   into another [West, et al. 2006]. Sequence alignment techniques
Permission to make digital or hard copies of part or all of this work for personal or
classroom use is granted without fee provided that copies are not made or distributed              first attempt to align sequences as closely as possible, before
for commercial advantage and that copies bear this notice and the full citation on the             computing their disimilarity [Josephson and Holmes 2002].
first page. Copyrights for components of this work owned by others than ACM must be                Matching alignments from multiple sequences may also repre-
honored. Abstracting with credit is permitted. To copy otherwise, to republish, to post on
                                                                                                   sent the ‘averaged’ scanpath from a set of users [Hembrooke, et
servers, or to redistribute to lists, requires prior specific permission and/or a fee.
Request permissions from Permissions Dept, ACM Inc., fax +1 (212) 869-0481 or e-mail
permissions@acm.org.
ETRA 2010, Austin, TX, March 22 – 24, 2010.
© 2010 ACM 978-1-60558-994-7/10/0003 $10.00
                                                                                             203
2.1     Scaled Traces

                                                                              A scaled trace is a spatial representation of a scanpath sequence
                                                                              that shows saccadic lengths and inter-fixation angles, but not
                                                                              fixation durations. In classic studies of the influence of intent on
                                                                              eye movements, Yarbus [1967] demonstrated that eye move-
                                                                              ment scanpaths gathered over 3 minute periods differed dramati-
                                                                              cally depending on the demands of specific tasks. Our represen-
                                                                              tation of scaled traces, derived from this work, is shown in Fig-
                                                                              ure 3.




    Figure 2. A scanpath from a single observer trying to locate
      content within the lower right hand area of a web page.
                                                                                Figure 3. Representation of scanpath (left) as a trace image
al., 2006]. Statistical tests are also available for testing signifi-              (right) by excluding fixations and minimizing scale.
cant differences between sets of scanpaths. In one comparison
approach, Heminghous and Duchowski [2006] provide a method                    In addition to showing a standard trace line (Figure 4A), sequen-
to automate the process of assigning areas of interest. They first            tial fixation order can also be represented by changing graphics
cluster scanpaths using a mean shift algorithm [Santella and                  properties of the line as it is drawn from beginning to end, from
DeCarlo, 2004], then tabulate similarity coefficients within and              light to dark (Figure 4B), or from thin to thick (Figure 4D).
between participants and images. In another approach, Feusner                 Multicolored segments can also be used (Figure 4C), but it may
and Lukoff [2008] compute the distance between all pairwise                   be more difficult to convey fixation order with color than with
combinations of scanpath sequences using editing distance, then               brightness or thickness.
determine the difference between the average distances within
and between groups to be compared. Using Monte Carlo simula-
tion methods, they generate significance values for observed
differences.

Scanpaths can also be summarized and compared visually. This
paper provides a set of compact visual representations of scan-
paths that can help usability professionals identify and compare
scanning strategies in eye-tracking data. Visual representations
can provide rich summaries of scanpath metrics, such as the
distribution of inter-fixation angles and saccade lengths. Visual
representations are also useful for validating and comparing the              Figure 4. Alternate strategies for coding temporal order of sac-
expressiveness of different metrics, as well as associating ca-                                   cades in a scaled trace.
nonical visual search strategies with scanpath data.
                                                                              Though not intended to communicate detailed scanning tenden-
                                                                              cies, traces can effectively differentiate very horizontal from
2      Methods: Scanpath Visual Representations                               very vertical scanning tendencies. Figure 5 shows 12 different
                                                                              trace examples, each with 8 fixations and 7 saccades. This array
Several compact visual representations that summarize scan-                   of traces is an example of a small multiple, in which minimized
paths are described and compared below. These show scanpath                   graphics allow the viewer to compare many variables simultane-
fixation density and complexity. Some of the representations                  ously [Tufte 1983]. In this case, these scanpaths can be com-
show aggregate values associated with revisited locations and                 pared for relative aspect ratios, horizontal/vertical extent of
aren’t subject to the visual complexity problems associated with              scanning, and density of fixations. Saccade crossings in scaled
saccade crossings. The representations can be used to help vali-              traces (Figure 5) may provide a rough approximation of scan-
date associations between actual data and prototypical strategies.            path complexity, but such conclusions are much less reliable for
                                                                              longer scanpaths or when comparing scanpaths of different
Scanpaths may also be compressed or simplified, prior to con-                 lengths.
verting to any of the visual representations. Compression algo-
rithms, defined for specific studies, could merge or delete dupli-            While traces provide compact scanpath representations for rapid
cate fixations within close proximity, could remove duplicate                 comparison, they cannot accurately convey number of fixations,
sub-sequences, or could remove sub-sequences that match in                    lengths of saccades, or search area. Rather, they only provide
reverse order [Tsai 2001].                                                    relative comparisons of scanpath fixation density and complex-
                                                                              ity.




                                                                        204
To construct a radial plot, a scanpath is first broken into its indi-
                                                                             vidual saccadic segments, much like in Noton and Stark’s origi-
                                                                             nal scanpath theory [Noton and Stark 1971]. As shown in Table
                                                                             1, each of these segments can be assigned a length, an absolute
                                                                             inter-fixation angle, and a relative inter-fixation angle. Rantala
                                                                             [2008] introduced a radial plot (‘saccade star’) that presents a
                                                                             distribution of gaze statistics, such as fixation duration, arranged
                                                                             by angle. The present concept extends this representation to
                                                                             several other statistics and aggregation methods.

                                                                                  Table 1. Computation of angles in example scanpath.

                                                                                Saccade ID                  Length     Absolute    Relative Angle
 Figure 5. Array of scanpath traces, forming a small multiple.                                              (pixels)   Angle (o)         (o)
       Each trace contains 8 fixations and 7 saccades.
                                                                                     1                       130          45             --
2.2     Time Expansion
                                                                                     2                        55         290           260
Scanpaths and scanpath traces are often tangled with significant
crossing and overlap, making it hard to separate horizontal and                      3                        52          30            90
vertical tendencies in scanning. By plotting the scanpath x or y-
component against time, ‘time expanded’ representations of                           4                       206         240           220
scanning tendencies are obtained. The general concept of substi-
tuting time for the horizontal axis of a scanpath has been previ-                    5                       305         320           100
ously introduced for cases of searching programming code
[Uwano et al. 2006] and search result listings [Raiha et al., 2005,                  6                       239         110           135
Aula et al. 2005].
                                                                                     7                       138         170            40
Separation of the scanpath into vertical and horizontal time ex-
pansions can broaden this concept to scanning whole pages.
Figure 6 shows an 11-fixation scanpath trace that is separated                       8                       122         320           150
into x and y time expansion graphs. Frequent horizontal shifts
are clearly seen in the x graph, and vertical shifts are apparent in                 9                        43          75            90
the y graph. Neither of these tendencies is as easily observed in
the original scanpath trace.                                                        10                       145          30           310

The time expansion representations are similar to ‘sparklines;’
when plotted as small multiples, they are much like those used
for stock prices and temperature [Tufte 2006]. General move-                 Absolute angles are measured relative to a global coordinate
ment trends and back-tracking frequency become apparent when                 system (see Figure 7A), whereas relative angles are measured
several time expansions are compared.                                        relative to the scanpath’s current direction of saccadic move-
                                                                             ment (see Figure 7B). Scanpath metrics using absolute angles
                                                                             reveal spatial tendencies relative to element layout on a page,
                                                                             whereas metrics using relative angles highlight veering from a
                                                                             set course.




 Figure 6. Time expanded x and y representations of scanpath.
                                                                             Figure 7. Scanpath saccade angles measured counter-clockwise
2.3     Radial Plots                                                         from an absolute or global reference frame (A), and a frame that
                                                                                        is relative to the direction of motion (B).
Radial plots are another compact way to represent scanpaths
visually. Radial plots represent scanpath angles and magnitudes              The angles are then aggregated into angle ranges, or bins, of a
with bars that radiate from a center point like spokes in a wheel.           predetermined resolution, as shown in Table 2. Here, bins are of
                                                                             width 45o; smaller widths provide greater precision in the plots,



                                                                       205
at the expense of few samples per bin. Fewer samples per bin, in             Example radial plots, based upon the scanpath in Table 1, are
turn, result in less matching because there is less opportunity for          shown in Figure 8. Plots based on absolute angles are shown in
different distributions to share the same bins. On the other hand,           the left column, and those based upon relative angles are in the
bin sizes that are too large result in less precision for individual         right column. The radial histograms, using saccade segment
distributions, but provide more information about how different              counts, indicate movement in most of the binned directions, with
distributions are similar.                                                   some bias to rightward movement (absolute angles). From the
                                                                             relative angles, however, there was a small bias in the direction
Each bin is then shown as a bar in the radial plot. Bar angle cor-           of 90o movement, left-hand turns, or counter-clockwise scan-
responds to the range of angles in the bin. We typically set each            ning. The longest average saccade lengths appeared to be in
bar angle to the angle at the center of the range of angles in each          upward and downward directions, although large rightward sac-
bin, but alternate values may be chosen (e.g. the average of the             cades were also noted. Notice that only short average saccade
actual angles in the bin). The bar length for each bin may be                lengths were made at a relative 0o angle, indicating that the
computed in a number of ways; it may be proportional to:                     scanpath was indeed quite twisted. The relative angle, summed
                                                                             lengths were even more sensitive to the twisting than the relative
•      Count: The number of saccade segments in the bin (i.e.                angle, average saccade length plots.
       whose inter-fixation angle lies within the angle range of the
       bin). These plots can also be termed radial histograms.
•      Average The average length of the saccade segments in the
       bin.
•      Sum: The sum total length of the saccade segments in the
       bin.
•      Max (or Min): The maximum (or minimum) saccade seg-
       ment length in the bin.


    Table 2. Computation of saccade bin counts, with bins of 45o
             width and saccade segments from Table 1.

            Bin Angles      Number of Saccades within Bin
             (degrees)
                            Absolute      Relative Angle (o)
                            Angle (o)

          1-45                  3                 1

          46-90                 1                 2

          91-135                1                 2

          136-180               1                 1
                                                                             Figure 8. Radial plot representations of scanpath from Table 1,
          181-225               0                 1                           based either on absolute or relative angles. The number indi-
                                                                                       cates the maximum value along each axis.
          226-270               1                 1
                                                                             3     Results: Strategy Comparison using Repre-
          271-315               1                 1
                                                                                   sentations
          316-360               2                 0
                                                                             3.1     Prototype Strategies

                                                                             The proposed scanpath representations can be applied to a few
In addition to bin lengths, the mean shift indicates the geometric           prototypical scanning strategies, to assess the representations’
or spatial mean of each radial plot. It is represented as a single           diagnostic potential for identifying these prototype patterns. A
dot on the radial plot, and is computed from the mean of the x               software tool was developed to render the scanpath representa-
and y components of the graph. This represents the overall                   tions, and to provide a basis for their qualitative and quantitative
movement tendency for the scanpath. The coordinates of this dot              comparison. The tool was developed in HTML and JavaScript,
could, optionally, be computed from the weighted means of each               and runs in either Firefox or Chrome browsers. It allows cursor-
bin length, where weights are defined by the count (the number               based entry of a prototype scanpath trace, which is then repre-
of scanpaths contained) in each bin. Compared to an unweighted               sented by each of the plots. Radial plots use default bin widths
mean shift, this weighted mean shift would show greater move-                of 20o, but this value is easily modified.
ment in the direction of bins containing greater numbers of
scanpath segments.                                                           Table 3 shows nine scanning traces with their compact represen-
                                                                             tations. These should be considered as examples in a much lar-



                                                                       206
ger library of prototypical scanning strategies. Sequential traces             identical. The absolute angle plots contain data and mean
(the first three columns) are drawn with ink that transitions from             shifts that are in the direction of scanning, while the bin an-
light to dark over each temporal sequence. The last two columns                gle from the relative angle plots was 0o. Mean shift location
contain an example metric, the number of bins containing data in               relative to origin can indicate the direction of scanning.
the radial plots, that could lead to a programmatic solution for
teasing apart the scanning strategies.                                     •   Downward, followed by rightward scanning (row 3). X
                                                                               and Y-time expansions both move downward and to the
By studying these prototypical traces, we can derive a few quali-              right. Relative angle radial plots show mostly 0o data, due
tative indicators of various scanning strategies. These indicators             to the fact that most saccades are in the same direction as
can, in turn, lead to appropriate metrics to enable more auto-                 preceding saccades. Absolute angle radial plots are more
mated classification of the scanning strategies. These heuristics              balanced between rightward and downward data.
are discussed below, with references to the appropriate row(s)
                                                                           •   Square scanning (rows 4 and 5). Scanpaths that traverse a
from Table 3.
                                                                               closed loop are indicated by absolute angle radial plots con-
                                                                               taining data spanning many bins. If scanning is primarily in
•    Linear scanning (rows 1 and 2). Whether scanning
                                                                               one direction, relative angle radial plots show most data in
     downward or rightward, both time expansions are linear,
                                                                               one angular bin. The more looping that is present in a scan-
     without reversals, and count and length radial plots are
                                                                               path, the closer the absolute angle mean shifts are to the
                              Table 3. Sample prototype scanning strategies, with their compact representations.




                                                                     207
origin, compared to relative angle mean shifts. In the case             ized here, the following are likely to separate scanpath strategies
      of a square-like scanning strategy, time expansions show a              in classification analyses:
      single horizontal and vertical reversal pattern. The radial
      plots are primarily single bin (relative angle) and four bins            •      Number of time expansion reversals. Measures amount
      (absolute), whether clockwise (CW) or counter-clockwise                         of reversal movement in X or Y-components of scanpath
      (CCW). Teasing apart CW from CCW direction can be                               movement. Circles, for example, sweep 1 or 2 reversals,
      based upon the primary angle of the mean shift associated                       depending on starting and ending locations. This measure
      with average length, relative angle radial plots. Note that                     can also be further subdivided into positive and negative
      radial plots that are based on count data don’t necessarily                     directions, to provide further classification precision.
      have equal length arms. This is caused by differences in                 •      Number of bins containing data in radial plots. Com-
      cursor velocity when inputting the square scanning strat-                       puted for both absolute and relative angle plots, based on
      egy. Here, the absolute angle 90o arm, moving upward, was                       either counts or scanpath lengths. This metric measures
      shorter because it contained fewer fixation counts than the                     directness and amount of circularity versus squareness in
      other arms. This, in turn, was due to faster upward cursor                      scanpaths. Circles contain many bins, and lines only con-
      movement, compared to the other directions.                                     tain a single bin.
•     Circular scanning (rows 6 and 7). Both CW and CCW                        •      Angle and Length of mean shift in radial plots. Also
      circles showed complex, correlated movement patterns in                         computed for both absolute and relative angle plots, based
      time expansions. Both Relative angle radial plots contain                       on either counts or scanpath lengths. Measures position
      directional data primarily in one or two angular bins. Abso-                    and distance of the mean shift, relative to 0o origin. Indi-
      lute angle radial plots typically contain data in all bins, al-                 cates primary direction and magnitude of scanpath move-
      though the length of individual bins depend on cursor input                     ment.
      velocity (fixation count data) or relative differences in sac-
      cade lengths (average length data). Both absolute angle                 3.3      Validation: Dataset Analysis
      plots contain data that are generally more distributed
      among multiple bins than for relative radial plots. If the              The process of using the aforementioned heuristics to diagnose
      mean shift is located between 0o-180o, there is a tendency              scanning tendencies can be illustrated using the results from a
      towards counter-clockwise scanning, and clockwise scan-                 recent study, in which over 120 participants scanned the image
      ning otherwise.                                                         of a novel iPhone application, shown in Figure 9, in order to
                                                                              locate the ‘Status’ of the ‘CHI Services Deal’, located in the
•     Triangular scanning (row 8). Absolute angle radial plots                yellow shaded cell. This relatively simple scanning task resulted
      clearly show data in three bins, corresponding to the three             in a very rich set of varied scanning strategies.
      primary angles, whereas relative plots were primarily one
      or two bins, in correspondence with few turns. As for
      square scanning, the length of the arms in the absolute an-
      gle, fixation count plots depends heavily upon cursor input
      speed. Whereas the absolute angle plots contain 3 arms
      with data, the relative angle plots contain 2 arms for an
      equilateral triangle. These relative angles correspond to 0o,
      and 60o (leftward).
•     More complex tendencies (row 9). Although not nearly as
      complex as ‘real’ scanpath data, a ‘Figure-8’ prototype
      scanpath starts to approximate the subtle complexities of
      real data. Here, there are more reversals in the X than the Y
      time expansion plot, indicating more complex horizontal
      than vertical movement. Absolute radial plots show all an-
      gular bins are filled, much like circular scanning. Their                 Figure 9. Emulated iPhone application task image, in which
      mean shifts are close to the origin, indicating closed loop               participants had to report the status of the CHI Service Deal
      scanning. The fact that the relative angle mean shifts were                                   (shaded yellow here).
      positive and close to 0o indicates that equal movements
      were made rightward and leftward from current scanning                  3.3.1      Similar Time Expansions
      direction. In even more complex scanpaths, radial plots                 Table 4 shows a subset of participant scanpaths that contain
      based on fixation counts indicate sub-segments with many                similar time expansion representations. Due to space constraints,
      longer, saccades in a specified direction. Radial plots based           we only show the time expansion representations and the scan-
      on saccade lengths could indicate a few long saccades in a              paths; other figures will show pertinent representations where
      specified direction, within a larger set of saccades.                   there are similarities. Each X-component time expansion indi-
                                                                              cates an initial leftward movement, followed by a rightward
3.2      Potential Metrics                                                    movement. Indeed, each scanpath image validates this pattern.

Based upon the preceding analysis of prototypical strategies,                 3.3.2      Similar Radial Plots
metrics can be obtained from the scanpath representations to                  Similar scanpath sub-sequences can also be identified by radial
help identify prototype strategies within real scanpath data.                 plots. Table 5 shows absolute and relative angle radial histo-
Many metrics are possible; although these are not further util-               grams that contain saccade count data. In each of these scan-
                                                                              paths, the absolute angle plots had significant bins at 0 o, 180 o,




                                                                        208
and 270o. Corresponding histograms based on relative angle                    Table 6. Similar scanpaths, from similar radial histograms.
were dominated by data at 90 o, indicating left turns relative to
current direction. The three scanpaths each travel left, down,
right, up, and down in a similar manner.

Table 4. Similar scanpaths, identified from similar x-component
                       time expansions.




                                                                           Table 7. Similar scanpaths, from similar relative angle radial
                                                                                                   histograms.




Table 5. Similar scanpaths, identified from absolute and relative
angle radial histograms. Each scanpath was dominated by find-
   ing the left table column, scanning down to the bottom row,
 locating the ‘Status’ column header, then ending at the desired
                                cell.
                                                                          4      Discussion

                                                                          The scanpath representations introduced in this paper provide
                                                                          eye tracking and usability specialists with a new tool for identi-
                                                                          fying scanpaths and scanning strategies. We have described
                                                                          them, generated examples using prototypical scanpath shapes,
                                                                          and showed how they can be used to diagnose scanning tenden-
                                                                          cies.

                                                                          Scanpaths are fascinating records of human visual attention in
                                                                          space and time. Scanpaths capture fixation start-time, duration,
                                                                          and position. Because the samples are time-stamped, they are
                                                                          ordered and form a sequence.

                                                                          Each type of representation carries its advantages, much like
                                                                          providing multiple windows into a dataset. The scaled traces
                                                                          provide relative comparisons of scanpath fixation density and
                                                                          complexity. The time expansions help to visually unwind scan-
                                                                          paths of circular scanning or backtracking. The radial plots and
                                                                          mean shifts can show individual, aggregated, and multiple scan-
                                                                          path data. Like heatmaps, they display tendencies within and
                                                                          across groups and conditions. Unlike heatmaps, however, they
                                                                          provide directional scanning strategy information. Whether or
Other scanning tendencies were also noted. The two scanpaths              not these plots are exposed to the end user, they can serve as a
shown in Table 6 exhibited remarkably similar absolute and                basis for metrics that can describe aspects of scanning behavior.
relative histograms. The relative mean shifts were at an angle of         It is possible, for example, to count the number of reversals in
about 120 o, indicating lots of leftward turning. Each scanpath           the time expansions or the dominant bins in the radial plots.
made a partial CCW circle starting at the center of the table.            These metrics could be applied to a set of prototype strategies
                                                                          and used to train a classification analysis to help tease apart
Radial histograms with many filled angular bins and mean shifts           similarities and differences among scanpaths.
near the origin can indicate circular closed loop scanning. This
was observed in Table 7, where both scanpaths made a CCW                  The compact representations discussed here can be applied to
circle before moving rightward in the table.                              any path or sequence-related problem with a spatial reference,
                                                                          not just scanpaths. For example, routing and logistics domains
                                                                          could benefit from new ways to compare transportation com-
                                                                          plexities. Radial plots could convey the number, length, and
                                                                          direction of delivery routes. More abstract spatial analogies,




                                                                    209
such as comparing imaginary numbers plotted in polar coordi-              HORNOF, A.J. AND HALVERSON, T. 2003. Cognitive Strategies
nates to compare filter response, are also possible.                         and Eye Movements for Searching Hierarchical Computer
                                                                             Displays, Proc. CHI 2003, ACM Press, 5, 249-256.
We believe that higher-level cognitive strategies can be exposed
by analyzing individual scanpath sequences, then showing ag-              JOSEPHSON, S. AND HOLMES, M.E. 2002. Visual Attention to
gregated strategies. As a first step, we have developed visual                Repeated Internet Images: Testing the Scanpath Theory on
representations of scanpaths based on data derived from their                 the World Wide Web. Proc. ETRA 2002, ACM Press, 43-
sequences: saccade lengths, saccade angles, and time expan-                   49.
sions. We have also explored how radial plots that aggregate              MARSHALL, S.P. 2007. Identifying Cognitive State from Eye
saccades by their inter-fixation angles can be used to character-            Metrics. Aviation, Space, and Environmental Medicine,
ize actual scanpaths.                                                        78(5), B165-B185.
Future work will attempt to further understand how compact                NOTON, D. AND STARK, L.W. 1971. Scanpaths in Saccadic Eye
scanpath representations can help identify scanning strategies               Movements While Viewing and Recognizing Patterns, Vi-
that are exhibited in and across scanpaths. This information, in             sion Research, 11, 929-942.
turn, will help eye tracking and usability specialists understand
                                                                          RAIHA, K., AULA, A., MAJARANTA, P., RANTALA, H., AND KOI-
how people scan images and software user interfaces.
                                                                              VUNEN, K. 2005. Static Visualization of Temporal Eye-
                                                                              Tracking Data. Proc. INTERACT 2005, Int. Fed. Info Proc,
                                                                              946-949.
References                                                                RANTALA, H. 2008. Eye2i: Coordinated Multiple Views for
AULA, A., MAJARANTA, P., AND RAIHA, K. 2005. Eye-tracking                    Gaze Data. Proc. ETRA 2008, ACM Press, 147.
   Reveals the Personal Styles for Search Result Evaluation,              SANTELLA, A., AND DECARLO, D. 2004. Robust Clustering of
   Proc. INTERACT 2005, Int. Fed. Info Proc, 1058-1061.                       Eye Movement Recordings for Quantification of Visual In-
FEUSNER, M., AND LUKOFF, B. 2008. Testing for Statistically                   terest. Proc. ETRA 2004, ACM Press, 27-34.
    Significant Differences Between Groups of Scan Patterns.              TSAI, C. 2001. A C-program for Scanpath Compression,
    Proc. ETRA 2008, ACM Press, 43-46.                                         http://research.chtsai.org/papers/scanpath-
GOLDBERG, J.H., AND KOTVAL, X.P. 1999. Computer Interface                      compression.html.
   Evaluation using Eye Movements: Methods and Constructs.                TUFTE, E. 1983. The Visual Display of Quantitative Information.
   Int. J. Industrial Ergonomics, 14, 631-645.                                Graphics Press, Cheshire, CT., 170-175.
GOLDBERG, J.H., AND KOTVAL, X.P. 1998. Eye Movement-Based                 TUFTE, E. 2006. Beautiful Evidence. Graphics Press, Cheshire,
   Evaluation of the Computer Interface, in Kumar, J. (Ed.),                  CT.
   Advances in Occupational Ergonomics and Safety, IOS
   Press, 529-532.                                                        UWANO, H., NAKUMURA, M., MONDEN, A., AND MATSUMOTO, K.
                                                                             2006. Analyzing Individual Performance of Source Code
HEMBROOKE, H., FEUSNER, M., AND GAY, G. 2006. Averaging                      Review Using Reviewers’ Eye Movement. Proc. ETRA
   Scan Patterns and What They Can Tell Us. Proc. ETRA                       2006, ACM Press, 133-140.
   2006, ACM Press, 41.
                                                                          WEST, J.M., HAAKE, A.R., ROZANSKI, E.P., AND KARN, K.S.
HEMINGHOUS, J., AND DUCHOWSKI, A.T. 2008. iComp: A Tool                      2006. eyePatterns: Software for Identifying Patterns and
   for Scanpath Visualization and Comparison. PROC APGV                      Similarities Across Fixation Sequences. Proc. ETRA 2006,
   2008, Symposium on Applied Perception in Graphics and                     ACM Press, 149-154.
   Visualization, ACM Press, 152.
                                                                          YARBUS, A.L. 1967. Eye Movements and Vision. New York,
                                                                             Plenum Press.




                                                                    210

More Related Content

PDF
Goldberg Scanpath Clustering And Aggregation
PDF
Skovsgaard Small Target Selection With Gaze Alone
PPT
icvgip poster
PDF
Takemura Estimating 3 D Point Of Regard And Visualizing Gaze Trajectories Und...
PDF
Coutinho A Depth Compensation Method For Cross Ratio Based Eye Tracking
PDF
Dorr Space Variant Spatio Temporal Filtering Of Video For Gaze Visualization ...
PDF
26.motion and feature based person tracking
PDF
A Novel Approach to Fingerprint Identification Using Gabor Filter-Bank
Goldberg Scanpath Clustering And Aggregation
Skovsgaard Small Target Selection With Gaze Alone
icvgip poster
Takemura Estimating 3 D Point Of Regard And Visualizing Gaze Trajectories Und...
Coutinho A Depth Compensation Method For Cross Ratio Based Eye Tracking
Dorr Space Variant Spatio Temporal Filtering Of Video For Gaze Visualization ...
26.motion and feature based person tracking
A Novel Approach to Fingerprint Identification Using Gabor Filter-Bank

What's hot (15)

PDF
International Journal of Engineering Research and Development (IJERD)
PDF
Dw32759763
PDF
An enhanced face recognition technique based on Overlapped Modular PCA approa...
PDF
human face counting
PDF
Matlab image processing_2013_ieee
PPTX
Extreme Spatio-Temporal Data Analysis
PDF
F0164348
PDF
Learning Moving Cast Shadows for Foreground Detection (VS 2008)
PDF
Stellmach Advanced Gaze Visualizations For Three Dimensional Virtual Environm...
PDF
Rotation Scale Invariant Semi Blind Biometric Watermarking Technique for Colo...
PDF
Dohyoung lee icassp2012_poster
PDF
Hansen Homography Normalization For Robust Gaze Estimation In Uncalibrated Se...
PDF
Komogortsev Qualitative And Quantitative Scoring And Evaluation Of The Eye Mo...
PDF
Zhang Eye Movement As An Interaction Mechanism For Relevance Feedback In A Co...
PDF
Single Image Super Resolution using Interpolation and Discrete Wavelet Transform
International Journal of Engineering Research and Development (IJERD)
Dw32759763
An enhanced face recognition technique based on Overlapped Modular PCA approa...
human face counting
Matlab image processing_2013_ieee
Extreme Spatio-Temporal Data Analysis
F0164348
Learning Moving Cast Shadows for Foreground Detection (VS 2008)
Stellmach Advanced Gaze Visualizations For Three Dimensional Virtual Environm...
Rotation Scale Invariant Semi Blind Biometric Watermarking Technique for Colo...
Dohyoung lee icassp2012_poster
Hansen Homography Normalization For Robust Gaze Estimation In Uncalibrated Se...
Komogortsev Qualitative And Quantitative Scoring And Evaluation Of The Eye Mo...
Zhang Eye Movement As An Interaction Mechanism For Relevance Feedback In A Co...
Single Image Super Resolution using Interpolation and Discrete Wavelet Transform
Ad

Viewers also liked (20)

PDF
A1 D Togni Gifi Ise 250310
KEY
About Me Final Presentation
PPT
eMarketing value + tips & tricks
PPT
Nossa Rotina 2
PDF
הפלמח ביטוי לממלכתיות אמיתית
PPTX
PDF
Preaviso elecciones ggaa
PPT
Ebtee Presentation
PPT
E Commerce Trends In Israel
PPT
Designer-Engine LTD
PPT
Sniffer1
PPT
Greeting1
KEY
English presentation
PDF
Statby school 2553_m6_1057012007
ODP
dotNET Foundation FGSL 2015
PDF
Creating a Leak-Free Water Containment Tank
PPT
Irma Irmayani Kelas 2 A Pe
PPT
Székelyföldi nyomtatott kulturális értékek digitalizálása
DOC
קביעת גבולותיה של מדינת ישראל 1948
A1 D Togni Gifi Ise 250310
About Me Final Presentation
eMarketing value + tips & tricks
Nossa Rotina 2
הפלמח ביטוי לממלכתיות אמיתית
Preaviso elecciones ggaa
Ebtee Presentation
E Commerce Trends In Israel
Designer-Engine LTD
Sniffer1
Greeting1
English presentation
Statby school 2553_m6_1057012007
dotNET Foundation FGSL 2015
Creating a Leak-Free Water Containment Tank
Irma Irmayani Kelas 2 A Pe
Székelyföldi nyomtatott kulturális értékek digitalizálása
קביעת גבולותיה של מדינת ישראל 1948
Ad

Similar to Goldberg Visual Scanpath Representation (20)

PDF
Grindinger Group Wise Similarity And Classification Of Aggregate Scanpaths
PDF
Jarodzka A Vector Based, Multidimensional Scanpath Similarity Measure
PDF
Blignaut Visual Span And Other Parameters For The Generation Of Heatmaps
PDF
Blignaut Visual Span And Other Parameters For The Generation Of Heatmaps
PDF
Ryan Match Moving For Area Based Analysis Of Eye Movements In Natural Tasks
PDF
Duchowski Scanpath Comparison Revisited
PPTX
Animation in user interfaces
PPTX
Psych590 Presentation Interest Point Sujoy
PDF
Liu Natural Scene Statistics At Stereo Fixations
PDF
Pontillo Semanti Code Using Content Similarity And Database Driven Matching T...
PDF
Tobii Usability Brochure
PDF
Faro An Interactive Interface For Remote Administration Of Clinical Tests Bas...
PDF
Identifying Fixations and Saccades in Eye-Tracking Protocols
PPSX
Eye-based head gestures
PDF
Psychophysiology and Eyetracking in User Experience
PDF
Porta Ce Cursor A Contextual Eye Cursor For General Pointing In Windows Envir...
PPTX
Eye Tracking & Design
PDF
The Evolution Of Eye Tracking
PDF
Istance Designing Gaze Gestures For Gaming An Investigation Of Performance
Grindinger Group Wise Similarity And Classification Of Aggregate Scanpaths
Jarodzka A Vector Based, Multidimensional Scanpath Similarity Measure
Blignaut Visual Span And Other Parameters For The Generation Of Heatmaps
Blignaut Visual Span And Other Parameters For The Generation Of Heatmaps
Ryan Match Moving For Area Based Analysis Of Eye Movements In Natural Tasks
Duchowski Scanpath Comparison Revisited
Animation in user interfaces
Psych590 Presentation Interest Point Sujoy
Liu Natural Scene Statistics At Stereo Fixations
Pontillo Semanti Code Using Content Similarity And Database Driven Matching T...
Tobii Usability Brochure
Faro An Interactive Interface For Remote Administration Of Clinical Tests Bas...
Identifying Fixations and Saccades in Eye-Tracking Protocols
Eye-based head gestures
Psychophysiology and Eyetracking in User Experience
Porta Ce Cursor A Contextual Eye Cursor For General Pointing In Windows Envir...
Eye Tracking & Design
The Evolution Of Eye Tracking
Istance Designing Gaze Gestures For Gaming An Investigation Of Performance

More from Kalle (20)

PDF
Yamamoto Development Of Eye Tracking Pen Display Based On Stereo Bright Pupil...
PDF
Wastlund What You See Is Where You Go Testing A Gaze Driven Power Wheelchair ...
PDF
Vinnikov Contingency Evaluation Of Gaze Contingent Displays For Real Time Vis...
PDF
Urbina Pies With Ey Es The Limits Of Hierarchical Pie Menus In Gaze Control
PDF
Urbina Alternatives To Single Character Entry And Dwell Time Selection On Eye...
PDF
Tien Measuring Situation Awareness Of Surgeons In Laparoscopic Training
PDF
Stevenson Eye Tracking With The Adaptive Optics Scanning Laser Ophthalmoscope
PDF
San Agustin Evaluation Of A Low Cost Open Source Gaze Tracker
PDF
Rosengrant Gaze Scribing In Physics Problem Solving
PDF
Qvarfordt Understanding The Benefits Of Gaze Enhanced Visual Search
PDF
Prats Interpretation Of Geometric Shapes An Eye Movement Study
PDF
Park Quantification Of Aesthetic Viewing Using Eye Tracking Technology The In...
PDF
Palinko Estimating Cognitive Load Using Remote Eye Tracking In A Driving Simu...
PDF
Nakayama Estimation Of Viewers Response For Contextual Understanding Of Tasks...
PDF
Nagamatsu User Calibration Free Gaze Tracking With Estimation Of The Horizont...
PDF
Nagamatsu Gaze Estimation Method Based On An Aspherical Model Of The Cornea S...
PDF
Mulligan Robust Optical Eye Detection During Head Movement
PDF
Moshnyaga The Use Of Eye Tracking For Pc Energy Management
PDF
Morimoto Context Switching For Fast Key Selection In Text Entry Applications
PDF
Mollenbach Single Gaze Gestures
Yamamoto Development Of Eye Tracking Pen Display Based On Stereo Bright Pupil...
Wastlund What You See Is Where You Go Testing A Gaze Driven Power Wheelchair ...
Vinnikov Contingency Evaluation Of Gaze Contingent Displays For Real Time Vis...
Urbina Pies With Ey Es The Limits Of Hierarchical Pie Menus In Gaze Control
Urbina Alternatives To Single Character Entry And Dwell Time Selection On Eye...
Tien Measuring Situation Awareness Of Surgeons In Laparoscopic Training
Stevenson Eye Tracking With The Adaptive Optics Scanning Laser Ophthalmoscope
San Agustin Evaluation Of A Low Cost Open Source Gaze Tracker
Rosengrant Gaze Scribing In Physics Problem Solving
Qvarfordt Understanding The Benefits Of Gaze Enhanced Visual Search
Prats Interpretation Of Geometric Shapes An Eye Movement Study
Park Quantification Of Aesthetic Viewing Using Eye Tracking Technology The In...
Palinko Estimating Cognitive Load Using Remote Eye Tracking In A Driving Simu...
Nakayama Estimation Of Viewers Response For Contextual Understanding Of Tasks...
Nagamatsu User Calibration Free Gaze Tracking With Estimation Of The Horizont...
Nagamatsu Gaze Estimation Method Based On An Aspherical Model Of The Cornea S...
Mulligan Robust Optical Eye Detection During Head Movement
Moshnyaga The Use Of Eye Tracking For Pc Energy Management
Morimoto Context Switching For Fast Key Selection In Text Entry Applications
Mollenbach Single Gaze Gestures

Goldberg Visual Scanpath Representation

  • 1. Visual Scanpath Representation Joseph H. Goldberg and Jonathan I. Helfman Applications User Experience Oracle USA joe.goldberg@oracle.com; jon.helfman@oracle.com Abstract attention at particular positions, and saccades, rapid eye move- ments with suppressed visual perception. Fixation positions are Eye tracking scanpaths contain information about how people specified with respect to the background stimulus image or dis- see, but traditional tangled, overlapping scanpath representations play that was visible when the samples were collected. A se- provide little insight about scanning strategies. The present work quence of consecutive fixations and saccades results in a scan- describes and extends several compact visual scanpath represen- path – a trace through time and space that may overlap itself. tations that can provide additional insight about individual and Scanpaths, along with other ocular-based metrics, may indicate aggregate/multiple scanning strategies. Three categories of rep- higher cognitive strategies and states [e.g., Marshall 2007; Hor- resentations are introduced: (1) Scaled traces are small images of nof and Halverson 2003]. A challenge for the researcher is to scanpaths as connected saccades, allowing the comparison of compare multiple scanpaths and quickly comprehend each par- relative fixation densities and distributions of saccades. (2) Time ticipant’s scanning strategy in each tested condition. expansions, substituting ordinal position for either the scan- path’s x or y-coordinates, can uncover otherwise subtle horizon- Typical visual representations of scanpaths use circles to repre- tal or vertical reversals in visual scanning. (3) Radial plots repre- sent fixations and lines to represent saccades. Figure 1 displays a sent scanpaths as a set of radial arms about an origin, with each hypothetical scanpath with 5 fixations and 4 saccades. A scan- arm representing saccade counts or lengths within a binned set path may be replayed using animation, but to show a scanpath as of absolute or relative angles. Radial plots can convey useful a static image, fixation duration is typically coded by circle ra- shape characteristics of scanpaths, and can provide a basis for dius. new metrics. Nine different prototype scanning strategies were represented by these plots, then heuristics were developed to classify the major strategies. The heuristics were subsequently applied to real scanpath data, to identify strategy trends. Future work will further automate the identification of scanning strate- gies to provide researchers with a tool to uncover and diagnose scanning-related challenges. CR Categories: H.1.2. User/Machine Systems: Software Psy- chology, H.5.2. User Interfaces: Graphical User Interfaces. Figure 1. A hypothetical scanpath with 5 fixations and 4 sac- cades, showing fixation order and varying saccade lengths. Keywords: Eye Tracking, Usability Evaluation, Scanpath, Scanning Strategy, Visualization Actual scanpath records are usually quite complex, and can be difficult to interpret and compare. Figure 2 shows a scanpath from a single observer trying to locate content within the lower 1 Introduction right hand area of a web page. The scanpath visited multiple content columns as well as the search area before locating the It has been said that proper representation of a problem is the content. Frequent crossing of saccades and revisiting of loca- single most important step towards its eventual solution. The tions contribute to scanpath complexity. problem we are investigating is the comparison of visual scan- ning strategies between individuals, groups, and conditions, in 1.2 Scanpath Comparison order to make inferences for improving the visual design of software and digital media. This paper introduces and extends Numerical techniques have been presented for summarizing and several representations for scanpaths, then demonstrates how comparing scanpaths.. The length of a scanpath can be a meas- these representations can be used to develop heuristics and met- ure of productivity, and average fixation duration may measure rics for detecting and labeling scanning strategies. of cognitive complexity [Goldberg and Kotval 1999]. The sum and distribution of the inter-fixation angles within a scanpath 1.1 Scanpaths have been proposed as a measure of efficiency on a task, with simple, direct scanpaths signaling higher efficiency [Goldberg Eye tracking methods typically sample gaze locations at 50-120 and Kotval 1998]. String editing distance computes the mini- Hz. Samples are then reduced to fixations, periods of visual mum number of editing steps required to transform one scanpath Copyright © 2010 by the Association for Computing Machinery, Inc. into another [West, et al. 2006]. Sequence alignment techniques Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed first attempt to align sequences as closely as possible, before for commercial advantage and that copies bear this notice and the full citation on the computing their disimilarity [Josephson and Holmes 2002]. first page. Copyrights for components of this work owned by others than ACM must be Matching alignments from multiple sequences may also repre- honored. Abstracting with credit is permitted. To copy otherwise, to republish, to post on sent the ‘averaged’ scanpath from a set of users [Hembrooke, et servers, or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from Permissions Dept, ACM Inc., fax +1 (212) 869-0481 or e-mail permissions@acm.org. ETRA 2010, Austin, TX, March 22 – 24, 2010. © 2010 ACM 978-1-60558-994-7/10/0003 $10.00 203
  • 2. 2.1 Scaled Traces A scaled trace is a spatial representation of a scanpath sequence that shows saccadic lengths and inter-fixation angles, but not fixation durations. In classic studies of the influence of intent on eye movements, Yarbus [1967] demonstrated that eye move- ment scanpaths gathered over 3 minute periods differed dramati- cally depending on the demands of specific tasks. Our represen- tation of scaled traces, derived from this work, is shown in Fig- ure 3. Figure 2. A scanpath from a single observer trying to locate content within the lower right hand area of a web page. Figure 3. Representation of scanpath (left) as a trace image al., 2006]. Statistical tests are also available for testing signifi- (right) by excluding fixations and minimizing scale. cant differences between sets of scanpaths. In one comparison approach, Heminghous and Duchowski [2006] provide a method In addition to showing a standard trace line (Figure 4A), sequen- to automate the process of assigning areas of interest. They first tial fixation order can also be represented by changing graphics cluster scanpaths using a mean shift algorithm [Santella and properties of the line as it is drawn from beginning to end, from DeCarlo, 2004], then tabulate similarity coefficients within and light to dark (Figure 4B), or from thin to thick (Figure 4D). between participants and images. In another approach, Feusner Multicolored segments can also be used (Figure 4C), but it may and Lukoff [2008] compute the distance between all pairwise be more difficult to convey fixation order with color than with combinations of scanpath sequences using editing distance, then brightness or thickness. determine the difference between the average distances within and between groups to be compared. Using Monte Carlo simula- tion methods, they generate significance values for observed differences. Scanpaths can also be summarized and compared visually. This paper provides a set of compact visual representations of scan- paths that can help usability professionals identify and compare scanning strategies in eye-tracking data. Visual representations can provide rich summaries of scanpath metrics, such as the distribution of inter-fixation angles and saccade lengths. Visual representations are also useful for validating and comparing the Figure 4. Alternate strategies for coding temporal order of sac- expressiveness of different metrics, as well as associating ca- cades in a scaled trace. nonical visual search strategies with scanpath data. Though not intended to communicate detailed scanning tenden- cies, traces can effectively differentiate very horizontal from 2 Methods: Scanpath Visual Representations very vertical scanning tendencies. Figure 5 shows 12 different trace examples, each with 8 fixations and 7 saccades. This array Several compact visual representations that summarize scan- of traces is an example of a small multiple, in which minimized paths are described and compared below. These show scanpath graphics allow the viewer to compare many variables simultane- fixation density and complexity. Some of the representations ously [Tufte 1983]. In this case, these scanpaths can be com- show aggregate values associated with revisited locations and pared for relative aspect ratios, horizontal/vertical extent of aren’t subject to the visual complexity problems associated with scanning, and density of fixations. Saccade crossings in scaled saccade crossings. The representations can be used to help vali- traces (Figure 5) may provide a rough approximation of scan- date associations between actual data and prototypical strategies. path complexity, but such conclusions are much less reliable for longer scanpaths or when comparing scanpaths of different Scanpaths may also be compressed or simplified, prior to con- lengths. verting to any of the visual representations. Compression algo- rithms, defined for specific studies, could merge or delete dupli- While traces provide compact scanpath representations for rapid cate fixations within close proximity, could remove duplicate comparison, they cannot accurately convey number of fixations, sub-sequences, or could remove sub-sequences that match in lengths of saccades, or search area. Rather, they only provide reverse order [Tsai 2001]. relative comparisons of scanpath fixation density and complex- ity. 204
  • 3. To construct a radial plot, a scanpath is first broken into its indi- vidual saccadic segments, much like in Noton and Stark’s origi- nal scanpath theory [Noton and Stark 1971]. As shown in Table 1, each of these segments can be assigned a length, an absolute inter-fixation angle, and a relative inter-fixation angle. Rantala [2008] introduced a radial plot (‘saccade star’) that presents a distribution of gaze statistics, such as fixation duration, arranged by angle. The present concept extends this representation to several other statistics and aggregation methods. Table 1. Computation of angles in example scanpath. Saccade ID Length Absolute Relative Angle Figure 5. Array of scanpath traces, forming a small multiple. (pixels) Angle (o) (o) Each trace contains 8 fixations and 7 saccades. 1 130 45 -- 2.2 Time Expansion 2 55 290 260 Scanpaths and scanpath traces are often tangled with significant crossing and overlap, making it hard to separate horizontal and 3 52 30 90 vertical tendencies in scanning. By plotting the scanpath x or y- component against time, ‘time expanded’ representations of 4 206 240 220 scanning tendencies are obtained. The general concept of substi- tuting time for the horizontal axis of a scanpath has been previ- 5 305 320 100 ously introduced for cases of searching programming code [Uwano et al. 2006] and search result listings [Raiha et al., 2005, 6 239 110 135 Aula et al. 2005]. 7 138 170 40 Separation of the scanpath into vertical and horizontal time ex- pansions can broaden this concept to scanning whole pages. Figure 6 shows an 11-fixation scanpath trace that is separated 8 122 320 150 into x and y time expansion graphs. Frequent horizontal shifts are clearly seen in the x graph, and vertical shifts are apparent in 9 43 75 90 the y graph. Neither of these tendencies is as easily observed in the original scanpath trace. 10 145 30 310 The time expansion representations are similar to ‘sparklines;’ when plotted as small multiples, they are much like those used for stock prices and temperature [Tufte 2006]. General move- Absolute angles are measured relative to a global coordinate ment trends and back-tracking frequency become apparent when system (see Figure 7A), whereas relative angles are measured several time expansions are compared. relative to the scanpath’s current direction of saccadic move- ment (see Figure 7B). Scanpath metrics using absolute angles reveal spatial tendencies relative to element layout on a page, whereas metrics using relative angles highlight veering from a set course. Figure 6. Time expanded x and y representations of scanpath. Figure 7. Scanpath saccade angles measured counter-clockwise 2.3 Radial Plots from an absolute or global reference frame (A), and a frame that is relative to the direction of motion (B). Radial plots are another compact way to represent scanpaths visually. Radial plots represent scanpath angles and magnitudes The angles are then aggregated into angle ranges, or bins, of a with bars that radiate from a center point like spokes in a wheel. predetermined resolution, as shown in Table 2. Here, bins are of width 45o; smaller widths provide greater precision in the plots, 205
  • 4. at the expense of few samples per bin. Fewer samples per bin, in Example radial plots, based upon the scanpath in Table 1, are turn, result in less matching because there is less opportunity for shown in Figure 8. Plots based on absolute angles are shown in different distributions to share the same bins. On the other hand, the left column, and those based upon relative angles are in the bin sizes that are too large result in less precision for individual right column. The radial histograms, using saccade segment distributions, but provide more information about how different counts, indicate movement in most of the binned directions, with distributions are similar. some bias to rightward movement (absolute angles). From the relative angles, however, there was a small bias in the direction Each bin is then shown as a bar in the radial plot. Bar angle cor- of 90o movement, left-hand turns, or counter-clockwise scan- responds to the range of angles in the bin. We typically set each ning. The longest average saccade lengths appeared to be in bar angle to the angle at the center of the range of angles in each upward and downward directions, although large rightward sac- bin, but alternate values may be chosen (e.g. the average of the cades were also noted. Notice that only short average saccade actual angles in the bin). The bar length for each bin may be lengths were made at a relative 0o angle, indicating that the computed in a number of ways; it may be proportional to: scanpath was indeed quite twisted. The relative angle, summed lengths were even more sensitive to the twisting than the relative • Count: The number of saccade segments in the bin (i.e. angle, average saccade length plots. whose inter-fixation angle lies within the angle range of the bin). These plots can also be termed radial histograms. • Average The average length of the saccade segments in the bin. • Sum: The sum total length of the saccade segments in the bin. • Max (or Min): The maximum (or minimum) saccade seg- ment length in the bin. Table 2. Computation of saccade bin counts, with bins of 45o width and saccade segments from Table 1. Bin Angles Number of Saccades within Bin (degrees) Absolute Relative Angle (o) Angle (o) 1-45 3 1 46-90 1 2 91-135 1 2 136-180 1 1 Figure 8. Radial plot representations of scanpath from Table 1, 181-225 0 1 based either on absolute or relative angles. The number indi- cates the maximum value along each axis. 226-270 1 1 3 Results: Strategy Comparison using Repre- 271-315 1 1 sentations 316-360 2 0 3.1 Prototype Strategies The proposed scanpath representations can be applied to a few In addition to bin lengths, the mean shift indicates the geometric prototypical scanning strategies, to assess the representations’ or spatial mean of each radial plot. It is represented as a single diagnostic potential for identifying these prototype patterns. A dot on the radial plot, and is computed from the mean of the x software tool was developed to render the scanpath representa- and y components of the graph. This represents the overall tions, and to provide a basis for their qualitative and quantitative movement tendency for the scanpath. The coordinates of this dot comparison. The tool was developed in HTML and JavaScript, could, optionally, be computed from the weighted means of each and runs in either Firefox or Chrome browsers. It allows cursor- bin length, where weights are defined by the count (the number based entry of a prototype scanpath trace, which is then repre- of scanpaths contained) in each bin. Compared to an unweighted sented by each of the plots. Radial plots use default bin widths mean shift, this weighted mean shift would show greater move- of 20o, but this value is easily modified. ment in the direction of bins containing greater numbers of scanpath segments. Table 3 shows nine scanning traces with their compact represen- tations. These should be considered as examples in a much lar- 206
  • 5. ger library of prototypical scanning strategies. Sequential traces identical. The absolute angle plots contain data and mean (the first three columns) are drawn with ink that transitions from shifts that are in the direction of scanning, while the bin an- light to dark over each temporal sequence. The last two columns gle from the relative angle plots was 0o. Mean shift location contain an example metric, the number of bins containing data in relative to origin can indicate the direction of scanning. the radial plots, that could lead to a programmatic solution for teasing apart the scanning strategies. • Downward, followed by rightward scanning (row 3). X and Y-time expansions both move downward and to the By studying these prototypical traces, we can derive a few quali- right. Relative angle radial plots show mostly 0o data, due tative indicators of various scanning strategies. These indicators to the fact that most saccades are in the same direction as can, in turn, lead to appropriate metrics to enable more auto- preceding saccades. Absolute angle radial plots are more mated classification of the scanning strategies. These heuristics balanced between rightward and downward data. are discussed below, with references to the appropriate row(s) • Square scanning (rows 4 and 5). Scanpaths that traverse a from Table 3. closed loop are indicated by absolute angle radial plots con- taining data spanning many bins. If scanning is primarily in • Linear scanning (rows 1 and 2). Whether scanning one direction, relative angle radial plots show most data in downward or rightward, both time expansions are linear, one angular bin. The more looping that is present in a scan- without reversals, and count and length radial plots are path, the closer the absolute angle mean shifts are to the Table 3. Sample prototype scanning strategies, with their compact representations. 207
  • 6. origin, compared to relative angle mean shifts. In the case ized here, the following are likely to separate scanpath strategies of a square-like scanning strategy, time expansions show a in classification analyses: single horizontal and vertical reversal pattern. The radial plots are primarily single bin (relative angle) and four bins • Number of time expansion reversals. Measures amount (absolute), whether clockwise (CW) or counter-clockwise of reversal movement in X or Y-components of scanpath (CCW). Teasing apart CW from CCW direction can be movement. Circles, for example, sweep 1 or 2 reversals, based upon the primary angle of the mean shift associated depending on starting and ending locations. This measure with average length, relative angle radial plots. Note that can also be further subdivided into positive and negative radial plots that are based on count data don’t necessarily directions, to provide further classification precision. have equal length arms. This is caused by differences in • Number of bins containing data in radial plots. Com- cursor velocity when inputting the square scanning strat- puted for both absolute and relative angle plots, based on egy. Here, the absolute angle 90o arm, moving upward, was either counts or scanpath lengths. This metric measures shorter because it contained fewer fixation counts than the directness and amount of circularity versus squareness in other arms. This, in turn, was due to faster upward cursor scanpaths. Circles contain many bins, and lines only con- movement, compared to the other directions. tain a single bin. • Circular scanning (rows 6 and 7). Both CW and CCW • Angle and Length of mean shift in radial plots. Also circles showed complex, correlated movement patterns in computed for both absolute and relative angle plots, based time expansions. Both Relative angle radial plots contain on either counts or scanpath lengths. Measures position directional data primarily in one or two angular bins. Abso- and distance of the mean shift, relative to 0o origin. Indi- lute angle radial plots typically contain data in all bins, al- cates primary direction and magnitude of scanpath move- though the length of individual bins depend on cursor input ment. velocity (fixation count data) or relative differences in sac- cade lengths (average length data). Both absolute angle 3.3 Validation: Dataset Analysis plots contain data that are generally more distributed among multiple bins than for relative radial plots. If the The process of using the aforementioned heuristics to diagnose mean shift is located between 0o-180o, there is a tendency scanning tendencies can be illustrated using the results from a towards counter-clockwise scanning, and clockwise scan- recent study, in which over 120 participants scanned the image ning otherwise. of a novel iPhone application, shown in Figure 9, in order to locate the ‘Status’ of the ‘CHI Services Deal’, located in the • Triangular scanning (row 8). Absolute angle radial plots yellow shaded cell. This relatively simple scanning task resulted clearly show data in three bins, corresponding to the three in a very rich set of varied scanning strategies. primary angles, whereas relative plots were primarily one or two bins, in correspondence with few turns. As for square scanning, the length of the arms in the absolute an- gle, fixation count plots depends heavily upon cursor input speed. Whereas the absolute angle plots contain 3 arms with data, the relative angle plots contain 2 arms for an equilateral triangle. These relative angles correspond to 0o, and 60o (leftward). • More complex tendencies (row 9). Although not nearly as complex as ‘real’ scanpath data, a ‘Figure-8’ prototype scanpath starts to approximate the subtle complexities of real data. Here, there are more reversals in the X than the Y time expansion plot, indicating more complex horizontal than vertical movement. Absolute radial plots show all an- gular bins are filled, much like circular scanning. Their Figure 9. Emulated iPhone application task image, in which mean shifts are close to the origin, indicating closed loop participants had to report the status of the CHI Service Deal scanning. The fact that the relative angle mean shifts were (shaded yellow here). positive and close to 0o indicates that equal movements were made rightward and leftward from current scanning 3.3.1 Similar Time Expansions direction. In even more complex scanpaths, radial plots Table 4 shows a subset of participant scanpaths that contain based on fixation counts indicate sub-segments with many similar time expansion representations. Due to space constraints, longer, saccades in a specified direction. Radial plots based we only show the time expansion representations and the scan- on saccade lengths could indicate a few long saccades in a paths; other figures will show pertinent representations where specified direction, within a larger set of saccades. there are similarities. Each X-component time expansion indi- cates an initial leftward movement, followed by a rightward 3.2 Potential Metrics movement. Indeed, each scanpath image validates this pattern. Based upon the preceding analysis of prototypical strategies, 3.3.2 Similar Radial Plots metrics can be obtained from the scanpath representations to Similar scanpath sub-sequences can also be identified by radial help identify prototype strategies within real scanpath data. plots. Table 5 shows absolute and relative angle radial histo- Many metrics are possible; although these are not further util- grams that contain saccade count data. In each of these scan- paths, the absolute angle plots had significant bins at 0 o, 180 o, 208
  • 7. and 270o. Corresponding histograms based on relative angle Table 6. Similar scanpaths, from similar radial histograms. were dominated by data at 90 o, indicating left turns relative to current direction. The three scanpaths each travel left, down, right, up, and down in a similar manner. Table 4. Similar scanpaths, identified from similar x-component time expansions. Table 7. Similar scanpaths, from similar relative angle radial histograms. Table 5. Similar scanpaths, identified from absolute and relative angle radial histograms. Each scanpath was dominated by find- ing the left table column, scanning down to the bottom row, locating the ‘Status’ column header, then ending at the desired cell. 4 Discussion The scanpath representations introduced in this paper provide eye tracking and usability specialists with a new tool for identi- fying scanpaths and scanning strategies. We have described them, generated examples using prototypical scanpath shapes, and showed how they can be used to diagnose scanning tenden- cies. Scanpaths are fascinating records of human visual attention in space and time. Scanpaths capture fixation start-time, duration, and position. Because the samples are time-stamped, they are ordered and form a sequence. Each type of representation carries its advantages, much like providing multiple windows into a dataset. The scaled traces provide relative comparisons of scanpath fixation density and complexity. The time expansions help to visually unwind scan- paths of circular scanning or backtracking. The radial plots and mean shifts can show individual, aggregated, and multiple scan- path data. Like heatmaps, they display tendencies within and across groups and conditions. Unlike heatmaps, however, they provide directional scanning strategy information. Whether or Other scanning tendencies were also noted. The two scanpaths not these plots are exposed to the end user, they can serve as a shown in Table 6 exhibited remarkably similar absolute and basis for metrics that can describe aspects of scanning behavior. relative histograms. The relative mean shifts were at an angle of It is possible, for example, to count the number of reversals in about 120 o, indicating lots of leftward turning. Each scanpath the time expansions or the dominant bins in the radial plots. made a partial CCW circle starting at the center of the table. These metrics could be applied to a set of prototype strategies and used to train a classification analysis to help tease apart Radial histograms with many filled angular bins and mean shifts similarities and differences among scanpaths. near the origin can indicate circular closed loop scanning. This was observed in Table 7, where both scanpaths made a CCW The compact representations discussed here can be applied to circle before moving rightward in the table. any path or sequence-related problem with a spatial reference, not just scanpaths. For example, routing and logistics domains could benefit from new ways to compare transportation com- plexities. Radial plots could convey the number, length, and direction of delivery routes. More abstract spatial analogies, 209
  • 8. such as comparing imaginary numbers plotted in polar coordi- HORNOF, A.J. AND HALVERSON, T. 2003. Cognitive Strategies nates to compare filter response, are also possible. and Eye Movements for Searching Hierarchical Computer Displays, Proc. CHI 2003, ACM Press, 5, 249-256. We believe that higher-level cognitive strategies can be exposed by analyzing individual scanpath sequences, then showing ag- JOSEPHSON, S. AND HOLMES, M.E. 2002. Visual Attention to gregated strategies. As a first step, we have developed visual Repeated Internet Images: Testing the Scanpath Theory on representations of scanpaths based on data derived from their the World Wide Web. Proc. ETRA 2002, ACM Press, 43- sequences: saccade lengths, saccade angles, and time expan- 49. sions. We have also explored how radial plots that aggregate MARSHALL, S.P. 2007. Identifying Cognitive State from Eye saccades by their inter-fixation angles can be used to character- Metrics. Aviation, Space, and Environmental Medicine, ize actual scanpaths. 78(5), B165-B185. Future work will attempt to further understand how compact NOTON, D. AND STARK, L.W. 1971. Scanpaths in Saccadic Eye scanpath representations can help identify scanning strategies Movements While Viewing and Recognizing Patterns, Vi- that are exhibited in and across scanpaths. This information, in sion Research, 11, 929-942. turn, will help eye tracking and usability specialists understand RAIHA, K., AULA, A., MAJARANTA, P., RANTALA, H., AND KOI- how people scan images and software user interfaces. VUNEN, K. 2005. Static Visualization of Temporal Eye- Tracking Data. Proc. INTERACT 2005, Int. Fed. Info Proc, 946-949. References RANTALA, H. 2008. Eye2i: Coordinated Multiple Views for AULA, A., MAJARANTA, P., AND RAIHA, K. 2005. Eye-tracking Gaze Data. Proc. ETRA 2008, ACM Press, 147. Reveals the Personal Styles for Search Result Evaluation, SANTELLA, A., AND DECARLO, D. 2004. Robust Clustering of Proc. INTERACT 2005, Int. Fed. Info Proc, 1058-1061. Eye Movement Recordings for Quantification of Visual In- FEUSNER, M., AND LUKOFF, B. 2008. Testing for Statistically terest. Proc. ETRA 2004, ACM Press, 27-34. Significant Differences Between Groups of Scan Patterns. TSAI, C. 2001. A C-program for Scanpath Compression, Proc. ETRA 2008, ACM Press, 43-46. http://research.chtsai.org/papers/scanpath- GOLDBERG, J.H., AND KOTVAL, X.P. 1999. Computer Interface compression.html. Evaluation using Eye Movements: Methods and Constructs. TUFTE, E. 1983. The Visual Display of Quantitative Information. Int. J. Industrial Ergonomics, 14, 631-645. Graphics Press, Cheshire, CT., 170-175. GOLDBERG, J.H., AND KOTVAL, X.P. 1998. Eye Movement-Based TUFTE, E. 2006. Beautiful Evidence. Graphics Press, Cheshire, Evaluation of the Computer Interface, in Kumar, J. (Ed.), CT. Advances in Occupational Ergonomics and Safety, IOS Press, 529-532. UWANO, H., NAKUMURA, M., MONDEN, A., AND MATSUMOTO, K. 2006. Analyzing Individual Performance of Source Code HEMBROOKE, H., FEUSNER, M., AND GAY, G. 2006. Averaging Review Using Reviewers’ Eye Movement. Proc. ETRA Scan Patterns and What They Can Tell Us. Proc. ETRA 2006, ACM Press, 133-140. 2006, ACM Press, 41. WEST, J.M., HAAKE, A.R., ROZANSKI, E.P., AND KARN, K.S. HEMINGHOUS, J., AND DUCHOWSKI, A.T. 2008. iComp: A Tool 2006. eyePatterns: Software for Identifying Patterns and for Scanpath Visualization and Comparison. PROC APGV Similarities Across Fixation Sequences. Proc. ETRA 2006, 2008, Symposium on Applied Perception in Graphics and ACM Press, 149-154. Visualization, ACM Press, 152. YARBUS, A.L. 1967. Eye Movements and Vision. New York, Plenum Press. 210