Object recognition with cortex like mechanisms pami-07

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 29, NO. 3, MARCH 2007 411

Robust Object Recognition with
Cortex-Like Mechanisms
Thomas Serre, Lior Wolf, Stanley Bileschi, Maximilian Riesenhuber, and
Tomaso Poggio, Member, IEEE

Abstract—We introduce a new general framework for the recognition of complex visual scenes, which is motivated by biology: We
describe a hierarchical system that closely follows the organization of visual cortex and builds an increasingly complex and invariant
feature representation by alternating between a template matching and a maximum pooling operation. We demonstrate the strength of
the approach on a range of recognition tasks: From invariant single object recognition in clutter to multiclass categorization problems
and complex scene understanding tasks that rely on the recognition of both shape-based as well as texture-based objects. Given the
biological constraints that the system had to satisfy, the approach performs surprisingly well: It has the capability of learning from only a
few training examples and competes with state-of-the-art systems. We also discuss the existence of a universal, redundant dictionary
of features that could handle the recognition of most object categories. In addition to its relevance for computer vision, the success of
this approach suggests a plausibility proof for a class of feedforward models of object recognition in cortex.

Index Terms—Object recognition, model, visual cortex, scene understanding, neural network.

Ç

1 INTRODUCTION

U NDERSTANDING how visual cortex recognizes objects is a
critical question for neuroscience. Because humans and
primates outperform the best machine vision systems with
object recognition in cortex have not yet been extended to
deal with real-world image databases [11], [12], [13], [14].
We present a system that is based on a quantitative theory
respect to almost any measure, building a system that of the ventral stream of visual cortex [14], [15]. A key element
emulates object recognition in cortex has always been an in the approach is a new set of scale and position-tolerant
attractive but elusive goal. For the most part, the use of feature detectors, which agree quantitatively with the tuning
visual neuroscience in computer vision has been limited to properties of cells along the ventral stream of visual cortex.
early vision for deriving stereo algorithms (e.g., [1]) and to These features are adaptive to the training set, though we also
justify the use of DoG (derivative-of-Gaussian) filters and show that a universal feature set, learned from a set of natural
more recently of Gabor filters [2], [3]. No real attention has images unrelated to any categorization task, likewise achieves
been given to biologically plausible features of higher good performance. To exemplify the strength of this feature-
complexity. While mainstream computer vision has always based representation, we demonstrate classification results
been inspired and challenged by human vision, it seems to with simple linear classifiers. We show that the approach
never have advanced past the very first stages of processing performs well on the recognition of isolated objects in clutter
in the simple (and binocular) cells in V 1 and V 2. Although for both binary and multiclass classification problems on
some of the systems inspired—to various degrees—by publicly available data sets. Our approach also demonstrates
neuroscience [4], [5], [6], [7], [8], [9], [10] have been tested good classification results on a challenging (street) scene
on at least some natural images, neurobiological models of understanding application that requires the recognition of
both shape-based as well as texture-based objects.
Both the source code of our system and the StreetScenes
. T. Serre is with the Massachusetts Institute of Technology, the Center for data set used in our experiments are readily available [16].
Biological and Computational Learning, McGovern Institute for Brain
Research and Brain & Cognitive Sciences Department, 77 Massachusetts 1.1 Related Work
Avenue, Bldg 46-5155B, Cambridge, MA 02139. E-mail: serre@mit.edu. Hierarchical architectures have been shown to outperform
. L. Wolf and S. Bileschi are with the Massachusetts Institute of Technology,
single-template (flat) object recognition systems on a variety
the Center for Biological and Computational Learning, McGovern Institute
for Brain Research and Brain & Cognitive Sciences Department, of object recognition tasks (e.g., face detection [17] and car
77 Massachusetts Avenue, Bldg 46-5155C, Cambridge, MA 02139. detection [18]). In particular, constellation models [19], [20],
E-mail: {liorwolf, bileschi}@mit.edu. [21] have been shown to be able to learn to recognize many
. M. Riesenhuber is with Georgetown University Medical Center, Research objects (one at a time) using an unsegmented training set
Building Room WP-12 3970, Reservoir Rd., NW, Washington, DC 20007.
E-mail: mr287@georgetown.edu. from just a few examples [20], [21]. Multilayered convolu-
. T. Poggio is with the Massachusetts Institute of Technology, the Center for tional networks were shown to perform extremely well in
Biological and Computational Learning, McGovern Institute for Brain the domain of digit recognition [4], [5] and, more recently,
Research and Brain & Cognitive Sciences Department, 77 Massachusetts generic object recognition [10] and face identification [22].
Avenue, Bldg 46-5177B, Cambridge, MA 02139. E-mail: tp@ai.mit.edu.
The simplest and one of the most popular appearance-
Manuscript received 8 Sept. 2005; revised 14 May 2006; accepted 27 June based feature descriptors corresponds to a small gray value
2006; published online 15 Jan. 2007.
Recommended for acceptance by M. Srinivasan.
patch [23] of an image, also called component [17], [24], part
For information on obtaining reprints of this article, please send e-mail to: [19], [25], or fragment [26]. Such patch-based descriptors are,
tpami@computer.org, and reference IEEECS Log Number TPAMI-0488-0905. however, limited in their ability to capture variations in the
0162-8828/07/$25.00 ß 2007 IEEE Published by the IEEE Computer Society

412 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 29, NO. 3, MARCH 2007

object appearance: They are very selective for a target shape units similar to the C2 units in Fig. 1 predicted recent read-
but lack invariance with respect to object transformations. out experiments in monkey IT cortex [33], showing very
At the other extreme, histogram-based descriptors [27], similar selectivity and invariance for the same set of stimuli.
[28], [29] have been shown to be very robust with respect to The model in its initial version [14] used a very simple
object transformations. Perhaps the most popular features static dictionary of handcrafted features. It was suggested
are the SIFT features [27], which excel in the redetection of a that features from intermediate and higher layers in the
previously seen object under new image transformations model should instead be learned from visual experience.
and have been shown to outperform other descriptors [30]. Here, we extend the model by showing how to learn a
However, as we confirmed experimentally (see Sec- vocabulary of visual features from images and applying it
tion 3.1.2), with such a degree of invariance, it is very to the recognition of real-world object-categories. Prelimin-
unlikely that these features could perform well on a generic ary results previously appeared in several conference
object recognition task. proceedings [34], [35], [36].
The new appearance-based feature descriptors described
here exhibit a balanced trade-off between invariance and
selectivity. They are more flexible than image patches and
2 DETAILED IMPLEMENTATION
more selective than local histogram-based descriptors. S1 units: Our system is summarized in Fig. 1. A gray-value
Though they are not strictly invariant to rotation, invariance input image is first analyzed by a multidimensional array of
to rotation could, in principle, be introduced via the training simple S1 units which correspond to the classical simple cells
set (e.g., by introducing rotated versions of the original input). of Hubel and Wiesel found in the primary visual cortex (V1)
[11]. S1 units take the form of Gabor functions [2], which have
1.2 The Standard Model of Visual Cortex been shown to provide a good model of cortical simple cell
Our system follows a recent theory of the feedforward receptive fields [3] and are described by the following
path of object recognition in cortex that accounts for the equation:
first 100-200 milliseconds of processing in the ventral
stream of primate visual cortex [14], [15]. The model itself ðx2 þ
2 y2 Þ
o o 2
F ðx; yÞ ¼ exp À Â cos xo ; s:t: ð1Þ
attempts to summarize—in a quantitative way—a core of 22
well-accepted facts about the ventral stream in the visual
cortex (see [15] for a review): xo ¼ x cos þ y sin and yo ¼ Àx sin þ y cos : ð2Þ
1.Visual processing is hierarchical, aiming to build All filter parameters, i.e., the aspect ratio,
¼ 0:3, the
invariance to position and scale first and then to orientation , the effective width , the wavelength as well
viewpoint and other transformations. as the filter sizes s were adjusted so that the tuning
2. Along the hierarchy, the receptive fields of the neurons properties of the corresponding S1 units match the bulk of
(i.e., the part of the visual field that could potentially V1 parafoveal simple cells based on data from two groups
elicit a response from the neuron) as well as the [37], [38], [39], [40]. This was done by sampling the
complexity of their optimal stimuli (i.e., the set of parameter space, applying the corresponding filters to
stimuli that elicit a response of the neuron) increases. stimuli commonly used to probe cortical cells (i.e., gratings,
3. The initial processing of information is feedforward bars, and edges) and selecting the parameter values that
(for immediate recognition tasks, i.e., when the image capture the tuning properties of the bulk of V1 simple cells
presentation is rapid and there is no time for eye (see Table 1 and [41] for details). We arranged the S1 filters
movements or shifts of attention). to form a pyramid of scales, spanning a range of sizes from
4. Plasticity and learning probably occurs at all stages 7 Â 7 to 37 Â 37 pixels in steps of two pixels. To keep the
and certainly at the level of inferotemporal (IT) number of units tractable, we considered four orientations
cortex and prefrontal cortex (PFC), the top-most (0 , 45 , 90 , and 135 ), thus leading to 64 different S1
layers of the hierarchy. receptive field types total (16 scales Â 4 orientations).
C1 units: The next, C1 , stage corresponds to cortical
In its simplest form, the model consists of four layers of
complex cells which show some tolerance to shift and size:
computational units, where simple S units alternate with
Complex cells tend to have larger receptive fields (twice as
complex C units. The S units combine their inputs with a large as simple cells), respond to oriented bars or edges
bell-shaped tuning function to increase selectivity. The anywhere within their receptive fields (tolerance to posi-
C units pool their inputs through a maximum (MAX) tion), and tend to be more broadly tuned than simple cells
operation, thereby increasing invariance.1 Evidence for the (tolerance to size) [11]. C1 units pool over retinotopically
two key operations as well as biophysically plausible organized afferent S1 units from the previous layer with the
circuits can be found in [15]. The model is qualitatively same orientation and from the same scale band (see Table 1).
and quantitatively consistent with (and, in some cases, Each scale band contains two adjacent filter sizes (there are
actually predicts) several properties of cells along the eight scale bands for a total of 16 S1 filter sizes). For
ventral stream of visual cortex (see [15] for an overview). instance, scale band 1 contains S1 filters with sizes 7 Â 7 and
For instance, the model predicts, at the C1 and C2 levels (see 9 Â 9. The scale band index of the S1 units also determines
Fig. 1), respectively, the max-like behavior of a subclass of the size of the S1 neighborhood NS Â NS over which the
complex cells in V1 [31] and cells in V4 [32]. Read-out from C1 units pool. Again, this process is done for each of the
four orientations and each scale band independently.
1. In this paper, we used a Gaussian function but, as discussed in [15], a
bell-shaped tuning function could also be approximated via a normalized This pooling increases the tolerance to 2D transformations
dot-product. from layer S1 to C1 . The corresponding pooling operation is a

SERRE ET AL.: ROBUST OBJECT RECOGNITION WITH CORTEX-LIKE MECHANISMS 413

Fig. 1. System overview: A gray-value input image is first analyzed by an array of S1 units at four different orientations and 16 scales. At the next
C1 layer, the image is subsampled through a local MAX ðMÞ pooling operation over a neighborhood of S1 units in both space and scale, but with the
same preferred orientation. In the next stage, S2 units are essentially RBF units, each having a different preferred stimulus. Note that S2 units are tiled
across all positions and scales. A MAX pooling operation is performed over S2 units with the same selectivity to yield the C2 unit responses.

MAX operation. That is, the response r of a complex unit filter of size 7 Â 7 and the one obtained using a filter of size
corresponds to the response of the strongest of its m afferents 9 Â 9 (see Table 1). The maps have the same dimensionality
ðx1 ; . . . ; xm Þ from the previous S1 layer such that: but they are the outputs of different filters. The C1 unit
responses are computed by subsampling these maps using a
r ¼ max xj : ð3Þ cell grid of size NS Â NS ¼ 8 Â 8. From each grid cell, one
j¼1...m
single measurement is obtained by taking the maximum of all
Consider, for instance, the first band: S ¼ 1. For each 64 elements. As a last stage, we take a max over the two scales
orientation, it contains two S1 maps: The one obtained using a from within the same spatial neighborhood, by recording


TABLE 1
images. These prototypes are extracted at the level of the C1
Summary of the S1 and C1 SMFs Parameters layer across all four orientations, i.e., a patch Po of size n Â n
contains n Â n Â 4 elements. In the following, we extracted
patches of four different sizes ðn ¼ 4; 8; 12; 16Þ. An important
question for both neuroscience and computer vision regards
the choice of the unlabeled target set from which to learn—in
an unsupervised way—this vocabulary of visual features. In
the following, features are learned from the positive training
set for each object independently, but, in Section 3.1.2, we
show how a universal dictionary of features can be learned
from a random set of natural images and shared between
multiple object classes.
The Classification Stage: At runtime, each image is propa-
gated through the architecture described in Fig. 1. The C1 and
C2 standard model features (SMFs) are then extracted and
further passed to a simple linear classifier (we experimented
with both SVM and boosting).

3 EMPIRICAL EVALUATION
We evaluate the performance of the SMFs in several object
only the maximum value from the two maps. Note that
detection tasks. In Section 3.1, we show results for detection in
C1 responses are not computed at every possible locations
clutter (sometimes referred to as weakly supervised) for
and that C1 units only overlap by an amount ÁS . This makes
which the target object in both the training and test sets
the computations at the next stage more efficient. Again,
appears at variable scales and positions within an unseg-
parameters (see Table 1) governing this pooling operation
mented image, such as in the CalTech101 object database [21].
were adjusted such that the tuning of the C1 units match the
For such applications, because 1) the size of the image to be
tuning of complex cells as measured experimentally (see [41]
classified may vary and 2) because of the large variations in
for details).
appearance, we use the scale and position-invariant C2 SMFs
S2 units: In the S2 layer, units pool over afferent C1 units
(the number N of which is independent of the image size and
from a local spatial neighborhood across all four orienta-
only depends on the number of prototypes learned during
tions. S2 units behave as radial basis function (RBF) units.2
training) that we pass to a linear classifier trained to perform a
Each S2 unit response depends in a Gaussian-like way on
simple object present/absent recognition task.
the Euclidean distance between a new input and a stored
In Section 3.2, we evaluate the performance of the SMFs in
prototype. That is, for an image patch X from the previous
conjunction with a windowing approach. That is, we extract a
C1 layer at a particular scale S, the response r of the
large number of fixed-size image windows from an input
corresponding S2 unit is given by:
image at various scales and positions, which each have to be
classified for a target object to be present or absent. In this task,
r ¼ exp À

kX À Pi k2 ; ð4Þ the target object in both the training and test images exhibits a
limited variability to scale and position (lighting and within-
where

defines the sharpness of the TUNING and Pi is one
class appearance variability remain) which is accounted for
of the N features (center of the RBF units) learned during
by the scanning process. For this task, the presence of clutter
training (see below). At runtime, S2 maps are computed
within each image window to be classified is also limited.
across all positions for each of the eight scale bands. One
Because the size of the image windows is fixed, both C1 and
such multiple scale map is computed for each one of the
C2 SMFs can be used for classification. We show that, for such
ðN $ 1; 000Þ prototypes Pi .
an application, due to the limited variability of the target
C2 units: Our final set of shift- and scale-invariant C2
object in position and scale and the absence of clutter, C1 SMFs
responses is computed by taking a global maximum ((3))
appear quite competitive.
over all scales and positions for each S2 type over the entire
In Section 3.3, we show results using the SMFs for the
S2 lattice, i.e., the S2 measures the match between a stored
recognition of texture-based objects like trees and roads. For
prototype Pi and the input image at every position and
this application, the performance of the SMFs is evaluated at
scale; we only keep the value of the best match and discard
every pixel locations from images containing the target
the rest. The result is a vector of N C2 values, where N
object which is appropriate for detecting amorphous objects
corresponds to the number of prototypes extracted during
in a scene, where drawing a closely cropped bounding box
the learning stage.
is often impossible. For this task, the C2 SMFs outperform
The learning stage: The learning process corresponds to
the C1 SMFs.
selecting a set of N prototypes Pi (or features) for the S2 units.
This is done using a simple sampling process such that, 3.1 Object Recognition in Clutter
during training, a large pool of prototypes of various sizes Because of their invariance to scale and position, the
and at random positions are extracted from a target set of C2 SMFs can be used for weakly supervised learning tasks
for which a labeled training set is available but for which the
2. This is consistent with well-known response properties of neurons in
primate inferotemporal cortex and seems to be the key property for learning training set is not normalized or segmented. That is, the
to generalize in the visual and motor systems [42]. target object is presented in clutter and may undergo large


TABLE 2
Results Obtained with 1,000 C2 Features Combined with
SVM or GentleBoost (boost) Classifiers and Comparison
with Existing Systems (Benchmark)

Fig. 2. Sample images from the MIT-CBCL multiview car [18] and face
[17] data sets.

changes in position and scales. Importantly, the number of images for training and 427 positive and 5,000 negative
C2 features depends only on the number of patches extracted images for testing. The car data set contains 4,000 positive and
during training and is independent of the size of the input 1,600 negative training examples and 1,700 test examples
image. Here, to perform different categorization tasks, the (both positive and negative). Although the benchmark algo-
C2 responses computed over a new input image are simply rithms were trained on the full sets and the results reported
passed to a linear classifier (linear SVM or boosting).3 accordingly, our system only used a subset of the training sets
Below, we compare the performance of the scale and (500 examples of each class only).
translation-invariant C2 features when used as inputs to These two MIT-CBCL data sets are challenging: The face
simple linear classifiers with other benchmark systems for patterns used for testing are a subset of the CMU PIE database
the recognition of objects in clutter (i.e., both training and [44], which contains a large variety of faces under extreme
testing are performed on unsegmented images). We illumination conditions (see [17]). The test nonface patterns
consider three data sets, denoted CalTech5, CalTech101, were selected by a low-resolution LDA classifier as the most
and MIT-CBCL, to evaluate our system performance. similar to faces (the LDA classifier was trained on an
independent 19 Â 19 low-resolution training set). The car
3.1.1 Image Data Sets database includes a wide variety of vehicles, including SUVs,
CalTech5: We consider five of the databases,4 i.e., the frontal- trucks, buses, etc., under wide pose and lighting variations.
face, motorcycle, rear-car, and airplane data sets from [20], Random image patterns at various scales that were not labeled
as well as the leaf data set from [19]. On these data sets, we as vehicles were extracted and used as a negative test set.
used the same fixed splits as in the corresponding studies
3.1.2 Results
whenever applicable and otherwise generated random
splits. All images were rescaled to be 140 pixels in height Comparison with benchmark systems: Table 2 summarizes the
(width was rescaled accordingly so that the image aspect performance of the C2 SMFs compared with other published
ratio was preserved) and converted to gray scale. results from benchmark systems: the constellation models by
CalTech101: It contains 101 object classes plus a back- Perona et al. [19], [20], the hierarchical SVM-based face-
ground class (see [21] for details). All results reported were detection system by Heisele et al. [17] and a standard system
generated with 10 random splits. In the binary experiments, [18] that uses Ullman et al.’s fragments [26] and gentleBoost
we used 50 negative training examples and a variable number as in [45]. The performance measure reported is the accuracy
of positive training examples (1, 3, 15, 30, and 40). For testing, at the equilibrium point, i.e., the accuracy point such that the
false positive rate equals the miss rate. Results obtained with
we selected 50 negative examples and 50 positive examples
the C2 SMFs are superior to previous approaches [17], [18] on
from the remaining images (or as many left if less than 50 were
the MIT-CBCL data sets and comparable to the best systems
available). In the multiclass experiment, we used 15 or
[46], [47] on the CalTech5 data sets.5
30 training images per class. This includes the background
Comparison with SIFT features: We also compared the C2
class and the “faces” and “faces-easy” as three of the classes.
SMFs to a system based on Lowe’s SIFT features [27]. To
We used as many as 50 testing examples per class, less if there
perform this comparison at the feature level and ensure a fair
were not enough examples left after training. If less than
comparison between the two systems, we neglected all
50 examples were used, the results were normalized to reflect
position information recovered by Lowe’s algorithm. It was
equal contributions for each class. We report the mean and
recently suggested in [47] that structural information does not
standard deviation of the performance across all classes. All
seem to help improve recognition performance. We selected
images were rescaled to be 140 pixels in height (width was
1,000 random reference key-points from the training set.
rescaled accordingly so that the image aspect ratio was
Given a new image, we measured the minimum distance
preserved) and converted to gray scale.
between all its key-points and the 1,000 reference key-points,
MIT-CBCL: This includes a near-frontal ðÆ30 Þ face data
thus obtaining a feature vector of size 1,000.6
set [17] and a multiview car data set from [18] (see Fig. 2). The
face data set contains about 6,900 positive and 13,700 negative 5. Experimental procedures may vary from one group to another (e.g.,
splits used, preprocessing, scale normalization, etc.). Comparisons should
3. More biologically plausible classifiers are described in [43]. Such therefore be taken cautiously.
classifiers are likely to correspond to the task-specific circuits in the cortex 6. Lowe recommends using the ratio of the distances between the nearest
from IT to PFC (see [15], [43]). and the second closest key-point as a similarity measure. We found instead
4. Available at http://www.robots.ox.ac.uk/vgg/data3.html. that the minimum distance leads to better performance than the ratio.


Fig. 3. Comparison between the SIFT and the C2 features on the CalTech5 for (a) different numbers of features and on the (b) CalTech101 for a
different number of training examples.

Fig. 4. Performance obtained with gentleBoost and different numbers of C2 features on the (a) CalTech5 and on sample categories from the
(b) CalTech101 for a different number of training examples.

Fig. 3 shows a comparison between the performance of the 1,000-5,000 features) is much larger than the number used
SIFT and the C2 SMFs (both with gentleBoost; similar results by current systems (on the order of 10-100 for [17], [26], [45]
were obtained with a linear SVM). Fig. 3a shows a comparison and 4-8 for constellation approaches [19], [20], [21]). This
on the CalTech5 database for different numbers of features may come from the fact that we only sample the space of
(obtained by selecting a random number of them from the features and do not perform any clustering step like other
1,000 available) and Fig. 3b on the CalTech101 database for approaches (including an earlier version of this system
different number of training examples. In both cases, the [34]). We found clustering to be sensitive to the choice of
C2 features outperform the SIFT features significantly. SIFT parameters and initializations, leading to poorer results.
features excel in the redetection of a transformed version of a We also studied the influence of the number of training
previously seen example, but may lack selectivity for a more examples on the performance of the system on the
general categorization task at the basic level. CalTech101 database. For each object category, we generated
Number of features and training examples: To investigate different positive training sets of size 1, 3, 6, 15, and 30 as in
the contribution of the number of features on performance, [21] (see Section 3.1.1). As shown in Fig. 4b, the system
we first created a set of 10,000 C2 SMFs and then randomly achieves error rates comparable to [21] on a few training
selected subsets of various sizes. The results reported are examples (less than 15), but its performance still improves
averaged over 10 independent runs. As Fig. 4a shows, with more examples (where the system by Fei-Fei et al.
while the performance of the system can be improved with seems to be reaching a plateau, see [21]). Results with an
more features (e.g., the whole set of 10,000 features), SVM (not shown) are similar, although the performance
reasonable performance can already be obtained with 50- tended to be higher on very few training examples (as SVM
100 features. Features needed to reach the plateau (about seems to avoid overfitting even for one example).


Fig. 5. Histogram of the C2 features performance for all 101-object categories with (a) linear SVM and (b) gentleBoost for different numbers of positive
training examples.

However, since SVM does not select the relevant features, random natural images (downloaded from the Web). While
its performance tends to be lower than gentleBoost as the the object-specific set performs significantly better with
number of training examples increases. Fig. 5 shows the enough training examples, the universal set appears to be
performance of the gentleBoost and SVM classifiers used with competitive for smaller training sets.
the C2 SMFs on all categories and for various numbers of Indeed the universal feature set is less prone to overfitting
training examples (each result is an average of 10 different with few training examples (both the learning of the features
random splits). Each plot is a single histogram of all and the training of the final classifier are performed on the
101 scores, obtained using a fixed number of training same set with the object-specific set). In addition, contrary to the
examples, e.g., with 40 examples, the gentleBoost-based object-specific set, the size of the universal set is constant
system gets around 95 percent ROC area for 42 percent of the regardless of the number of training examples (10,000). As a
object categories. result, with small training data sets, fewer features can be used
Toward a universal dictionary of features: We here describe with the object-specific set (we found that extracting more
experiments that suggest that it is possible to perform robust than 200 features per training image had very little effect on
object recognition with C2 SMFs learned from a separate set of performance). This may constitute a relevant and intriguing
randomly selected natural images. In Fig. 6, we compare the result on its own. Our results also suggest that it should be
performance of two sets of features on the CalTech101 possible for biological organisms to acquire a basic vocabulary
database: 1) a standard set of object-specific features that were of features early in development while refining it with more specific
learned from a training set of images from the target object features later on. The latter point is consistent with reports of
category (200 features per training image) and 2) a universal plasticity in inferotemporal cortex from adult monkey (the
set of 10,000 features learned independently from a set of complexity and sizes of the largest C2 features are consistent
with the receptive fields of posterior IT neurons).
Multiclass results on the CalTech101: Finally, we report
results on multiclass classification on the CalTech101 data-
base. To conduct this experiment, we use the universal
dictionary of 1,000 features similar to the one described
earlier. This offers a significant gain in speed in a multiclass
setting compared to the standard object-specific set. The
classifier is a multiclass linear SVM that applied the all-pairs
method and is trained on 102 labels (101 categories plus the
background category). The performance of the system
reaches above 44 Æ 1.14 percent correct classification rate
when using 15 training examples per class averaged over
10 repetitions (see Section 3.1.1). Using only five training
images per class, the performance degrades to $ 30 percent.
By considering gestalt-like features (e.g., good-continuity
detectors, circularity detectors, and symmetry detectors)
within the same framework in addition to the C2 SMFs,
Wolf et al. obtained 51.2 percent Æ 1.2 percent correct [48],
[49] and recently incorporated some changes with Sharat
Chikkerur to get 55.0 percent Æ 0.9 percent (all these results
Fig. 6. Object-specific versus universal C2 features. are for 15 training images). At press time, some of the best


TABLE 3
image. This window of support is equal to a rectangle three
The StreetScenes Database times the size of pi and centered in the image at the same
relative location from which pi was originally extracted. The
advantage of the patch-based features over the single-
template approach is that local patches can be highly selective
while maintaining a degree of position invariance. The
system was implemented with N ¼ 1; 024 features and with
patches of size 12 Â 12 in images of size 128 Â 128. The third
systems include the system in [50] (% 44 percent correct) benchmark system is a Part-based system as described in [25].
and in [51] (45 percent correct). Briefly, both object parts and a geometric model are learned
via image patch clustering. The detection stage is performed
3.2 Object Recognition without Clutter by redetecting these parts and allowing them to vote for
3.2.1 The StreetScenes Database objects-at-poses in a generalized Hough transform frame-
In order to test the SMFs on a challenging real-world object work. Finally, we compare to an implementation of the
detection problem, we have built training and test data from Histogram of Gradients (HoG) feature of [52], which has
the StreetScenes scene-understanding data set. This database shown excellent performance on these types of objects. All
consists of more than 3,000 labeled images of the streets benchmark systems were trained and tested on the same data
around Boston and Cambridge. Sample images and their sets as the SMFs-based system. They all use gentleBoost
hand labelings are illustrated in Fig. 10; some statistics of the except [25].
content of the data set are given in Table 3. The accurate
detection of many of these object categories is made difficult 3.2.4 Results
by the wide internal variability in their appearance. For The ROC results of this experiment are illustrated in Fig. 7.
example, the object class cars includes examples of many For the two (C1 and C2 ) SMFs-based systems, the Grayscale
diverse models, at many poses, and in various types of as well as the Local Patch Correlation system, the classifier is
occlusion and lighting, trees appear very different in gentleBoost, but we found very similar results with both a
summer and winter, and the class of buildings includes linear and a polynomial-kernel SVM. Overall, for all three
skyscrapers as well as suburban houses. Capturing this wide object categories tested, the SMFs-based system performs
variability while maintaining high accuracy is part of the best on cars and bicycles and second behind HoG on
challenge of the scene-understanding problem. The data- pedestrians (the HoG system was parameter-tuned in [52]
base is available online at [16]. to achieve maximal performance on this one class). Finally,
for this recognition task, i.e., with a windowing framework,
3.2.2 Training the SMFs-Based Systems the C1 SMFs seem to be superior to the C2 SMFs. Indeed, the
Using data extracted from our StreetScenes database, we C1 SMFs are adept at representing the object boundaries of
trained object detectors for the classes car, pedestrian, and these shape-based objects, which have strong interexample
bicycle. This data was extracted by cropping out labeled correspondence.
examples of these object classes. Negative examples were 3.3 Object Recognition of Texture-Based Objects
extracted similarly by finding locations and scales which
Here, we demonstrate the utility of the SMFs in a texture-
matched the positive data, but did not overlap the labeled
based object recognition task. Performance is measured by
positives. Each example, positive and negative, was resized
considering each pixel, rather than each instance of an
to 128 Â 128 pixels and converted to gray scale. This image
object, to be a separate example. We consider four texture-
was then converted into C1 space using the method of
based objects: buildings, trees, roads, and skies.
Section 2. For a 128 Â 128 gray-scale image and our
parameter values, this resulted in a feature vector of
3.3.1 Training the SMFs-Based Systems
13,362 features that provided the input to the C1 -based
classifier. The C2 representation was built as in Section 3.1 In building a database of labeled texture examples, we were
for the recognition in clutter. Classifiers for these objects careful to avoid errors due to overlap and loose polygonal
labeling in the StreetScenes database. Because of object
were trained using gentleBoost For these experiments, all
occlusions, some pixels in the database are labeled as one
labeled positive examples and 10 times as many negative
object, i.e., building, but their actual appearance is due to
examples were extracted. The systems evaluation was
another object, i.e., tree. We addressed this by removing
performed using randomized training (1/3) and testing
pixels with either multiple labels or no labe, from the test.
(2/3) splits.. Additionally, training samples were never drawn from
within 15 pixels of any object’s border. The same training
3.2.3 Benchmark Systems
and test locations were used for both the SMFs-based and
For comparison, we also implemented four other benchmark the benchmark systems.
systems. Our most simple baseline detector is a single- To build the C1 SMFs-based system, C1 maps were
template Grayscale system: Each image is normalized in size computed for each image and, for each sample point, feature
and histogram equalized before the gray-values are passed to vector elements were collected by sampling the resulting C1
a linear classifier (gentleBoost). Another baseline detector, maps at a set of relative locations and scale-bands. A C2 SMF-
Local Patch Correlation, is built using patch-based features based system was also built as in Section 3.1 except for the
similar to [45]. Each feature fi is associated with a particular maximum over position at the S2 level that was taken over a
image patch pi , extracted randomly from the training set. local neighborhood instead of the whole image. This local
Each feature fi is calculated in a test image as the maximum area corresponded to a 60 Â 60 pixel window in the original
normalized cross correlation of pi within a subwindow of the 960 Â 1; 280 pixel image.


Fig. 7. ROC curves illustrating the performance of the standard-model object-detectors compared to four baseline systems (see Section 3.2). Note
that, in this test, the amount of clutter is limited by the windowing process, creating better interexample correspondence and thereby allowing the
direct application of the C1 SMFs.

3.3.2 Benchmark Systems Finally, the Histogram of edges (HoE) system was built
We implemented four benchmark texture classification by simply using the same type of histogram framework, but
systems. The Blobworld (BW) system was constructed as over the local 36-dimensional directional filter responses
described in [53]. Briefly, the Blobworld feature, originally (using the filters of T 1) rather than the texton identity. Here,
as well, learning was done using the gentleBoost algorithm
designed for image segmentation, is a six-dimensional vector
(again a linear SVM produced very similar results). The
at each pixel location; three dimensions encode color in the
within-class variability of the texture-objects in this test is
Lab color space and three dimensions encode texture using
considerably larger than that of the texture classes usually
the local spectrum of gradient responses. We did not include
used to test texture-detection systems, making this task
the color information for a fair comparison between all the
somewhat different. This may explain the relatively poor
various texture detection methods.
performance of some of these systems on certain objects.
The systems labeled T 1 and T 2 are based on [29]. In these
systems, the test image is first processed with a number of 3.3.3 Results
predefined filters. T 1 uses 36 oriented edge-filters arranged
in five degrees increments from 0 degrees to 180 degrees. T 2 As shown in Fig. 8, the SMFs-based texture system seems to
follows [29] exactly by using 36 Gabor filters at six consistently outperform the benchmarks (BW, T 1, T 2, and
orientations, three scales, and two phases. For both systems HoE). C2 compared to C1 SMFs may be better suited to this
independently, a large number of random samples of the task because of their increased invariance properties and
36-dimensional edge response images were taken and complexity.
subsequently clustered using k-means to find 100 cluster
3.4 Toward a Full System for Scene Understanding
centroids (i.e., the textons). The texton image was then
calculated by finding the index of the nearest texton to the The SMFs-based object detection systems described pre-
filter response vector at each pixel in the response images. A viously were combined into a complete system for scene
100-dimensional texton feature vector was then built by understanding. The objects to be detected are divided into
calculating the local 10 Â 10 histogram of nearest texton two distinct categories, texture-based objects and shape-based
indexes. objects, which are handled using different recognition


Fig. 8. Performance (ROC curves) of five texture classification algorithms for the detection of buildings, trees, skies, and roads. This texture
classification task requires reliable recognition of texture classes with wide intraclass variability. This difficult test may in part explain the inferior
performance of the benchmark algorithms, which have been previously used to detect object boundaries and classify materials, but not for object
recognition.

Fig. 9. Data flow diagram of the scene-understanding system (see text for details).

strategies. Fig. 9 illustrates the architecture of the data flow 3.4.1 Shape-Based Object Detection in StreetScenes
diagram, specifically highlighting the two pathways for the Shape-based objects are those objects for which there exists
detection of the texture-based and shape-based objects. a strong part-to-part correspondence between examples,


Fig. 10. Top row: Sample StreetScenes examples. Middle row: True hand-labeling; color overlay indicates texture-based objects and bounding
rectangles indicate shape-based objects. Note that pixels may have multiple labels due to overlapping objects or no label at all (indicated in white).
Bottom row: Results obtained with a system trained on examples like (but not including) those in the second row.

including pedestrians, cars, and bicycles. In order to detect responses by segmenting the input image and averaging
shape-based objects, a standard windowing technique is the responses of the detectors over each segment. As a result,
used. This contrasts with the approach presented in uncertain responses at the object borders are compensated for
Section 3.1, wherein objects in clutter are detected using by the more numerous responses within the object bound-
scale and translation-invariant C2 SMFs, rather than testing aries. This was accomplished using the segmentation soft-
for object presence at each position and scale indepen- ware Edison [54]. Sample results of our texture recognition
dently. While the C2 approach is appropriate for fast system can be seen in the bottom row of Fig. 10.
decisions of object presence or absence, it would be
impractical for this scene-understanding application as the
locations of individual objects would be lost. 4 DISCUSSION
In conjunction with this windowing approach, we use 4.1 A Computer Vision Perspective on the Model
the C1 SMFs. Since the window crops away much of the The computer vision system described in this work was
clutter, leaving the potential object nearly centered, the constructed from a neuroscience model of the primate visual
additional invariance from the C2 features is not necessary. cortex which is a rather unusual approach. The model itself is
It is important to note that the good performance of the C1 based on a consensus among neuroscientists and on fitting
SMFs is dependent upon training data with accurate available experimental data. Still, one may wonder about the
descriptions of the position and scale of the target object. relationships between the SMFs and other computer vision
Performance metrics for both C1 and C2 SMFs were shown algorithms: Because of the hierarchical and nonlinear nature
in Section 3.2, as well as those for a number of benchmark of the architecture described in Fig. 1, there is little hope in
systems. In the final system, the classifiers output is finding a simple general cost function that the system would
thresholded and a standard local neighborhood suppres- minimize. These types of functions are seldom available for
sion technique is used in which the maximum detection is hierarchical systems which are not probabilistic in nature or
recorded and the response within a neighborhood in scale explicitly set out to minimize an energy function. Instead, we
space is suppressed. In Fig. 10, we present some sample next study each layer of the system separately.
results obtained with the system. The first layer ðS1 Þ consists of applying Gabor filters to the
input image, which mimics the processing by simple cells in
3.4.2 Pixel-Wise Detection of Texture-Based Objects the primary visual cortex (V1). Gabor filters have been
Texture-based objects are those objects for which, unlike around in computer vision for decades, starting with
shape-based objects, there is no obvious visible interobject Gabor’s demonstration [2] that these elementary functions
part-wise correspondence. These objects are better de- minimize the uncertainty of their product and Daugman’s
scribed by their texture rather than the geometric structure extension [55] to 2D. They are also very similar to DoG filters
of reliably detectable parts. For the StreetScenes database, used since the 1960s to model receptive fields in the retina
these currently include buildings, roads, trees, and skies. and primary visual cortex and to perform edge detection in
Using the models trained in Section 3.3 and applying them computer vision (see [56], [57]). Bovik et al. [58] used Gabor
to each pixel within the image, one obtains a detection filters for texture segmentation and Sanger [59] for the
confidence map of the size of the original image for each computation of disparity in stereovision. In biometrics, it has
object. This map is used to judge which pixel belongs to been used for face recognition (e.g., [60]), iris recognition as
which texture-object category. Simply taking the value with well as fingerprint recognition. Olshausen and Fields
maximum response strength results in unsatisfactory results, demonstrated that optimizing a sparse coding scheme over
as it was found that, when the receptive field of a unit a set of natural images produces a set of edge filters similar
overlaps a texture-boundary, the response becomes unreli- to Gabor filters [61]. Hence, it was expected that the output
able. This was addressed by smoothing the anomalous of Gabor filters on natural images would be sparse. This


Fig. 11. Max scale space images of Lena (top row) and of the Gabor filtered version of Lena (bottom row). While the gray-value image gets distorted,
the information in the sparse edge image is enhanced.

Fig. 12. Gaussian scale space images of Lena (top row) and of the Gabor filtered version of Lena (bottom row). While the gray-value image degrades
gracefully, revealing structures at different scales, the sparse edge image fades away.

result comes from the fact that Gabor filters, as edge general it is not an appropriate scale space— among other
detecting filters, are activated only near image edges. Any problems, when applying it to an image, new local extrema
further analysis step done on top of Gabor filters should take are being created. This can be seen in the top row of Fig. 11,
this sparseness property into account. where applying the max scale space to the Lena image
The next layer (C1 ) does something which is unorthodox creates block-like structures, which are new level sets, and
for computer vision systems—it maximizes the output of where the corners are new local maxima.
the filters locally. Each C1 unit computes the maximum over However, our application of the local maximum operator
a small pool of S1 outputs. While many systems maximize is on the Gabor filtered image, which is a sparse representa-
the output of a detector over the entire image, local tion of the original image. For such an input, the Gaussian
maximization has only been done recently. For part-based scale space results in a diluted image (see bottom row of
object detection [17], [26], [45], local detectors of each part Fig. 12). The max scale space, on the other hand, is successful
are learned independently, and are then applied to local in keeping the sparse inputs through the consecutive
regions where the parts are expected to appear. applications of the max filter. Put differently, for the analysis
Our work seems novel in that general purpose filters are of gray-level images, it is important not to create new
being maximized over uniformly distributed local regions structures while moving to coarser scales: In this, a Gaussian
in the image. In order to understand this stage, we can scale space is appropriate and a local maximum type of
invoke some scale space terminology (see [62] for an analysis is not. For the analysis of sparse coding, it is
overview). Scale space theory was mostly concerned at first important to conserve the local maxima, which is precisely
with the Gaussian scale space. This scale space has many what the maximum operator does (the Gaussian scale space
desirable properties such as separability, linearity, shift on the other hand flattens the input).
invariance, isotropy, homogeneity, and causality. The last The next two levels in our system involve the combina-
property is an important one: Causality means that no new tion of C1 outputs using a template matching approach.
level sets are created by going into coarser scales. A related Prototype templates (patches) are extracted from the training
property is to demand the noncreation of local extrema in images and the best match with these serves as an image
coarser scales. In our application, a local maximization representation. The first template-based stage (S2 ) measures
(instead of Gaussian blurring) is used to go from a fine to a the “correlation” (Euclidean distance) of the C1 maps with
coarser scale in order to make the C1 layer invariant to small many small crops obtained from such maps.
local translations. As a pseudoscale space, local maximiza- The correlation is measured for the four orientations
tion has some desirable properties: It is separable (one can together, thus making our algorithm sensitive to large
apply it over the rows and then over the columns), it is shift rotations of the image. Small rotations can be approximated
invariant, and it is homogeneous (it can be applied in the by small translations, which are handled by the maximiza-
same way to each scale; applying it repeatedly corresponds tion at the C1 level. Note that this stage is done at multiple
to moving into coarser and coarser scales). However, in scales such that a given template taken from a C1 map at a


certain scale during the prototype templates collection whereas biology is unlikely to be able to use it—at least in
stage of training is matched across all C1 maps when the cortical stream dedicated to shape processing and object
constructing the C2 feature vector. The last stage of our recognition). The system described in this paper may be the
system (C2 ) is a standard maximization over the entire first counterexample to this situation: It is based on a model
image (in principle and more biologically, this would be of object recognition in cortex [14], [15], it respects the
over an area of the size of the fovea but not the whole properties of cortical processing (including the absence of
visual field, see [15]). This is equivalent to scanning over all geometrical information) while showing performance at least
locations and scales for the maximum correlation with each comparable to the best computer vision systems.
C1 template selected in training. It has been suggested that “immediate recognition” during
scene categorization tasks may rely on partial processing by
4.2 What SMFs to Use for Which Tasks? the visual system based on a rapid and parallel detection of
To summarize our main results: In Section 3.1, we have shown disjunctive sets of unbound features of the target category
an application to the C2 SMFs to the semisupervised [64], [65]. Interestingly a recent psychophysical experiment
recognition of objects in clutter. For such tasks, the training [66] suggested that spatial information about the objects
images are unsegmented: The target object is embedded in location may be absent during “immediate recognition.” That
clutter and undergo changes in scale and position. Addition-
is, even though human observers correctly detect a target
ally, because the training images come in different sizes, only a
object within a frame embedded in a rapid sequence of
global representation based on a fixed-length scale and
images, they are, however, not able to recover even its
position-invariant feature vector such as the C2 SMFs is
approximate location [66]. Such an observation is in good
suitable.
As described in Section 3.2 for the recognition of shape- agreement with the experiment described in Section 3.1 in
based objects in conjunction with a scanning approach (the which the recognition of objects in clutter is based on a bag of
images to be classified are segmented and normalized), a translation and scale-invariant C2 features computed over the
more “holistic” representation based on C1 SMFs which are entire image for which spatial information is lost. Indeed, we
adept at detecting object boundaries tend to be superior. For recently showed that an extension of the model described in
such tasks, the variations in scale and position are limited this paper accounts for the level and the pattern of
and clutter is almost completely absent. As a result, the performance of human observers [43] on a rapid animal
scale and position-invariance of the C2 SMFs does not bring versus nonanimal categorization task [67]. This may be the
any extra computational benefit. first time that a neurobiological model, faithful to the
Finally, in Section 3.3, we showed that the C2 SMFs excel physiology and the anatomy of the visual cortex, provides a
at the recognition of texture-based objects which lack a realistic alternative to engineered artificial vision systems.
geometric structure of reliably detectable parts in compar-
ison to the C1 SMFs as well as other benchmark systems. 4.4 Open Questions, Limitations, and Possible
Improvements
4.3 Object Recognition in Cortex: Remarks 4.4.1 Have We Reached the Limit of What a/this
Our system belongs to a family of feedforward models of Feedforward Architecture Can Achieve in Terms
object recognition in the cortex that have been shown to be
able to duplicate the tuning properties of neurons in several
of Performance?
visual cortical areas [14]. In particular, Riesenhuber and There seem to be at least three directions that could be
Poggio showed that such a class of models accounts followed to further improve the performance of the
quantitatively for the tuning properties of view-tuned units architecture described here: First, very recent experiments
in IT cortex which respond to images of the learned object [43] suggests that the addition of extra layers (e.g., S3 , C3 ,
more strongly than to distractor objects, despite significant S4 , etc.), in agreement with the anatomy and physiology of
changes in position and size [63]. Model performance was the visual cortex, may provide a significant gain in
so far only reported for simple artificial stimuli such as performance. Additionally, we also found that loosening
paperclips on a uniform background [14], with no real- the hierarchy described in Fig. 1 may also provide some
world image degradations such as change in illumination, significant computational benefits. As already suggested by
clutter, etc. The success of our extension of the original the results of our experimental simulations in Section 3, not
model on a variety of large-scale real-world’s object all tasks are equal. Depending on the amount of clutter and
recognition databases provides a compelling plausibility 2D transformations involved, it is sometimes beneficial to
proof for this class of feed-forward models. use the fine information from low-level SMFs and some
A long-time goal for computer vision has been to build a other times to use more invariant high-level SMFs. We
system that achieves human-level recognition performance. found that passing different types of SMFs to the final
Until now, biology had not suggested a good solution. In fact, classifier and letting the classifier choose for the optimal
the superiority of human performance over the best artificial features may further improve performance (for instance,
recognition systems has continuously lacked a satisfactory passing both C1 and C2 SMFs) [43], [48].
explanation. The computer vision approaches had also Second, the sampling procedure we used here to learn
diverged from biology: For instance, some of the best existing features is very simple. It is likely that not all features are
computer vision systems use geometrical information about useful for recognition. Applying a standard feature selection
objects’ constitutive parts (the constellation approaches [19], technique may give further improvement in performance.
[20], [21] rely on a probabilistic shape model; in [17], the Indeed, a very recent study showed that selecting the subset
position of the facial components is passed to a combination of the C2 SMFs that are highly weighted by the SVM classifier
classifier (along with their associated detection values) provide a substantial increase in performance [68].


Third, for all the tests reported here, we did not tune a from a few examples and could compete with generative
single parameter to get optimal performance. Instead, models that use prior category information [21].
model parameters were set to match what is known about Second, we have described a new approach to scene
the primate visual system. Further improvements could understanding with an application to a StreetScenes database
likely be obtained by tuning some of the model parameters involving different types of rigid objects as well as texture-
[69] (see Table 1)—perhaps through learning. based objects. We found that the Standard Model Features
(SMFs) constitute a flexible framework that can be used in
4.4.2 Beyond Feedforward Architectures conjunction with standard computer vision techniques, i.e.,
As a feedforward model of the ventral stream pathway, the image scanning for the detection and localization of several
architecture of Fig. 1 cannot account for all aspects of our target objects at multiple scales and image segmentation for
everyday vision which involve eye movements and top- the recognition of nonrigid texture-based objects.
down effects, which are mediated by higher brain centers
and the extensive anatomical back-projections found
throughout the visual cortex and are not implemented in ACKNOWLEDGMENTS
the present feedforward model. While our system exhibits The authors would like to thank Sharat Chikkerur and
competitive performance compared to other benchmark Timothee Masquelier for useful comments on this manu-
systems, it remains limited compared to biological visual script. This report describes research done at the Center for
systems: The model seems to be able to account for the level Biological Computational Learning, which is in the
of performance of human observers on a rapid categoriza- McGovern Institute for Brain Research at MIT, the Depart-
tion task [67] when the stimulus presentation times are ment of Brain Cognitive Sciences, and the Computer
short and back-projections are inactive [43]. Yet the Sciences Artificial Intelligence Laboratory (CSAIL). This
performance of the model remains far behind the perfor- research was sponsored by grants from: the US Defense
mance of human observers for long presentation times. Advanced Research Projects Agency (B. Yoon), US Office of
It is important to point out that this recognition with a Naval Research (DARPA), US National Science Foundation-
glimpse only constitutes the initial processing step in natural National Institutes of Health (CRCNS). Additional support
vision. In particular, the model—in its present form—does was provided by: Daimler-Chrysler AG, Eastman Kodak
not capture Gestalt-like properties such as continuity and Company, Honda Research Institute USA, Inc., Komatsu
parallelism or figure-ground segmentation, which probably Ltd., Oxygen, Siemens Corporate Research, Inc., Sony,
involves lateral and feedback connections, yet to be inserted Sumitomo Metal Industries, Toyota Motor Corporation, and
in the model. A feedforward system (like the one we the Eugene McDermott Foundation.
presented here) could, in principle, be used as the front-end
of a visual system as part of a prediction-verification loop
[70]. The feedforward path would provide an initial REFERENCES
hypothesis about what object is presented in the visual [1] D. Marr and T. Poggio, “A Computational Theory of Human
field, yet to be verified through feedback loops. Stereo Vision,” Proc. Royal Soc. London B, vol. 204, pp. 301-328,
1979.
4.4.3 Future Work [2] D. Gabor, “Theory of Communication,” J. IEE, vol. 93, pp. 429-459,
1946.
Perhaps the major limitation of our system in a real-world [3] J.P. Jones and L.A. Palmer, “An Evaluation of the Two-Dimen-
applications setting remains its processing speed (limited by sional Gabor Filter Model of Simple Receptive Fields in Cat Striate
the S1 and C1 stages)—typically, tens of seconds, depending Cortex,” J. Neurophysiology, vol. 58, pp. 1233-1258, 1987.
[4] K. Fukushima, “Neocognitron: A Self Organizing Neural Network
on the size of the input image—which is too slow for a real- Model for a Mechanism of Pattern Recognition Unaffected by Shift
time application. Another important question, yet to be in Position,” Biological Cybernetics, vol. 36, pp. 193-201, 1980.
addressed, is whether the recognition results obtained with [5] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, “Gradient-Based
Learning Applied to Document Recognition,” Proc. IEEE, vol. 86,
bags of C2 features could be extended to other tasks, such as
no. 11, pp. 2278-2324, Nov. 1998.
face and gesture recognition or the analysis of video. [6] B.W. Mel, “SEEMORE: Combining Color, Shape, and Texture
Histogramming in a Neurally Inspired Approach to Visual Object
4.5 Conclusion Recognition,” Neural Computation, vol. 9, pp. 777-804, 1997.
In this paper, we have described a new framework for robust [7] S. Thorpe, “Ultra-Rapid Scene Categorisation with a Wave of
Spikes,” Proc. Biologically Motivated Computer Vision, pp. 1-15, 2002.
object recognition, which we have applied to two different [8] H. Wersing and E. Koerner, “Learning Optimized Features for
recognition scenarios: First, we have shown an application to Hierarchical Models of Invariant Recognition,” Neural Computa-
the problem of semisupervised recognition of objects in tion, vol. 15, no. 7, pp. 1559-1588, 2003.
clutter that does not involve image scanning. The system first [9] Y. Amit and M. Mascaro, “An Integrated Network for Invariant
Visual Detection and Recognition,” Vision Research, vol. 43, no. 19,
computes a set of scale and translation-invariant C2 features pp. 2073-2088, 2003.
from a training set of images, which is then passed to a [10] Y. LeCun, F.J. Huang, and L. Bottou, “Learning Methods for
standard classifier on the vector of features obtained from the Generic Object Recognition with Invariance to Pose and Lighting,”
input image. The system was tested on several object Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2004.
[11] D.H. Hubel and T.N. Wiesel, “Receptive Fields, Binocular
databases and shown to outperform several more complex Interaction and Functional Architecture in the Cat’s Visual
benchmark systems (e.g., the systems in [19], [20], [21] involve Cortex,” J. Physiology, vol. 160, pp. 106-154, 1962.
the estimation of probability distributions; [17] uses a [12] D. Perrett and M. Oram, “Neurophysiology of Shape Processing,”
Image and Vision Computing, vol. 11, pp. 317-333, 1993.
hierarchy of SVMs and requires accurate correspondences
[13] G. Wallis and E. Rolls, “A Model of Invariant Object Recognition
between positive training images, i.e., 3D head models). in the Visual System,” Progress in Neurobiology, vol. 51, pp. 167-
Interestingly, the approach was shown to be able to learn 194, 1997.

Object recognition with cortex like mechanisms pami-07

More Related Content

What's hot (18)

Similar to Object recognition with cortex like mechanisms pami-07 (20)

Recently uploaded (20)

Object recognition with cortex like mechanisms pami-07