cs551_intro.pdf

Introduction to Pattern Recognition
Selim Aksoy
Department of Computer Engineering
Bilkent University
saksoy@cs.bilkent.edu.tr
CS 551, Fall 2019
CS 551, Fall 2019 c 2019, Selim Aksoy (Bilkent University) 1 / 38

Human Perception
I Humans have developed highly sophisticated skills for
sensing their environment and taking actions according to
what they observe, e.g.,
I recognizing a face,
I understanding spoken words,
I reading handwriting,
I distinguishing fresh food from its smell.
I We would like to give similar capabilities to machines.

What is Pattern Recognition?
I A pattern is an entity, vaguely defined, that could be given a
name, e.g.,
I fingerprint image,
I handwritten word,
I human face,
I speech signal,
I DNA sequence,
I . . .
I Pattern recognition is the study of how machines can
I observe the environment,
I learn to distinguish patterns of interest,
I make sound and reasonable decisions about the categories
of the patterns.

Human and Machine Perception
I We are often influenced by the knowledge of how patterns
are modeled and recognized in nature when we develop
pattern recognition algorithms.
I Research on machine perception also helps us gain deeper
understanding and appreciation for pattern recognition
systems in nature.
I Yet, we also apply many techniques that are purely
numerical and do not have any correspondence in natural
systems.

Pattern Recognition Applications
Figure 1: English handwriting recognition.

Figure 2: Chinese handwriting recognition.

Figure 3: Biometric recognition.

Figure 4: Fingerprint recognition.

Figure 5: Autonomous navigation.

Figure 6: Cancer detection and grading using microscopic tissue data. (left)
A whole slide image with 75568 × 74896 pixels. (right) A region of interest with
7440 × 8260 pixels.

Figure 7: Land cover classification using satellite data.

Figure 8: Building and building group recognition using satellite data.

Figure 9: License plate recognition: US license plates.

Figure 10: Clustering of microarray data.

An Example
I Problem: Sorting incoming
fish on a conveyor belt
according to species.
I Assume that we have only
two kinds of fish:
I sea bass,
I salmon.
Figure 11: Picture taken from a
camera.

An Example: Decision Process
I What kind of information can distinguish one species from
the other?
I length, width, weight, number and shape of fins, tail shape,
etc.
I What can cause problems during sensing?
I lighting conditions, position of fish on the conveyor belt,
camera noise, etc.
I What are the steps in the process?
I capture image → isolate fish → take measurements → make
decision

An Example: Selecting Features
I Assume a fisherman told us that a sea bass is generally
longer than a salmon.
I We can use length as a feature and decide between sea
bass and salmon according to a threshold on length.
I How can we choose this threshold?

Figure 12: Histograms of the length feature for two types of fish in training
samples. How can we choose the threshold l∗
to make a reliable decision?

I Even though sea bass is longer than salmon on the
average, there are many examples of fish where this
observation does not hold.
I Try another feature: average lightness of the fish scales.

Figure 13: Histograms of the lightness feature for two types of fish in training
samples. It looks easier to choose the threshold x∗
but we still cannot make a
perfect decision.

An Example: Cost of Error
I We should also consider costs of different errors we make
in our decisions.
I For example, if the fish packing company knows that:
I Customers who buy salmon will object vigorously if they see
sea bass in their cans.
I Customers who buy sea bass will not be unhappy if they
occasionally see some expensive salmon in their cans.
I How does this knowledge affect our decision?

An Example: Multiple Features
I Assume we also observed that sea bass are typically wider
than salmon.
I We can use two features in our decision:
I lightness: x1
I width: x2
I Each fish image is now represented as a point (feature
vector)
x =
x1
x2
!
in a two-dimensional feature space.

Figure 14: Scatter plot of lightness and width features for training samples.
We can draw a decision boundary to divide the feature space into two
regions. Does it look better than using only lightness?

I Does adding more features always improve the results?
I Avoid unreliable features.
I Be careful about correlations with existing features.
I Be careful about measurement costs.
I Be careful about noise in the measurements.
I Is there some curse for working in very high dimensions?

An Example: Decision Boundaries
I Can we do better with another decision rule?
I More complex models result in more complex boundaries.
Figure 15: We may distinguish training samples perfectly but how can
we predict how well we can generalize to unknown samples?

An Example: Decision Boundaries
I How can we manage the tradeoff between complexity of
decision rules and their performance to unknown samples?
Figure 16: Different criteria lead to different decision boundaries.

More on Complexity

0 1
−1
0
1
Figure 17: Regression example: plot of 10 sample points for the input
variable x along with the corresponding target variable t. Green curve is the
true function that generated the data.

More on Complexity

0 1
−1
0
1
(a) 0’th order polynomial

0 1
−1
0
1
(b) 1’st order polynomial

0 1
−1
0
1
(c) 3’rd order polynomial

0 1
−1
0
1
(d) 9’th order polynomial
Figure 18: Polynomial curve fitting: plots of polynomials having various
orders, shown as red curves, fitted to the set of 10 sample points.

More on Complexity

0 1
−1
0
1
(a) 15 sample points

0 1
−1
0
1
(b) 100 sample points
Figure 19: Polynomial curve fitting: plots of 9’th order polynomials fitted to
15 and 100 sample points.

Pattern Recognition Systems
Physical environment
Data acquisition/sensing
Pre−processing
Feature extraction
Features
Classification
Post−processing
Decision
Model learning/estimation
Features
Feature extraction/selection
Pre−processing
Training data
Model
Figure 20: Object/process diagram of a pattern recognition system.

I Data acquisition and sensing:
I Measurements of physical variables.
I Important issues: bandwidth, resolution, sensitivity,
distortion, SNR, latency, etc.
I Pre-processing:
I Removal of noise in data.
I Isolation of patterns of interest from the background.
I Feature extraction:
I Finding a new representation in terms of features.

I Model learning and estimation:
I Learning a mapping between features and pattern groups
and categories.
I Classification:
I Using features and learned models to assign a pattern to a
category.
I Post-processing:
I Evaluation of confidence in decisions.
I Exploitation of context to improve performance.
I Combination of experts.

The Design Cycle
model
Train
classifier
Evaluate
classifier
Collect
data features
Select Select
Figure 21: The design cycle.
I Data collection:
I Collecting training and testing data.
I How can we know when we have adequately large and
representative set of samples?

The Design Cycle
I Feature selection:
I Domain dependence and prior information.
I Computational cost and feasibility.
I Discriminative features.
I Similar values for similar patterns.
I Different values for different patterns.
I Invariant features with respect to translation, rotation and
scale.
I Robust features with respect to occlusion, distortion,
deformation, and variations in environment.

The Design Cycle
I Model selection:
I Domain dependence and prior information.
I Definition of design criteria.
I Parametric vs. non-parametric models.
I Handling of missing features.
I Computational complexity.
I Types of models: templates, decision-theoretic or statistical,
syntactic or structural, neural, and hybrid.
I How can we know how close we are to the true model
underlying the patterns?

The Design Cycle
I Training:
I How can we learn the rule from data?
I Supervised learning: a teacher provides a category label or
cost for each pattern in the training set.
I Unsupervised learning: the system forms clusters or natural
groupings of the input patterns.
I Reinforcement learning: no desired category is given but the
teacher provides feedback to the system such as the
decision is right or wrong.

The Design Cycle
I Evaluation:
I How can we estimate the performance with training
samples?
I How can we predict the performance with future data?
I Problems of overfitting and generalization.

Summary
I Pattern recognition techniques find applications in many
areas: machine learning, statistics, mathematics, computer
science, biology, etc.
I There are many sub-problems in the design process.
I Many of these problems can indeed be solved.
I More complex learning, searching and optimization
algorithms are developed with advances in computer
technology.
I There remain many fascinating unsolved problems.

cs551_intro.pdf

More Related Content

Similar to cs551_intro.pdf (20)

Recently uploaded (20)

cs551_intro.pdf