Free Ebooks Download ! Edhole.com
Download as PPT, PDF1 like427 views
This document provides an introduction to pattern recognition. It defines pattern recognition as the assignment of physical objects or events to prespecified categories. It discusses the basic components of a pattern recognition system including sensors, feature extraction, classifiers, and learning algorithms. Several examples of pattern recognition applications are given such as optical character recognition, biometrics, and medical diagnosis. Common approaches to pattern recognition like statistical, structural, and neural networks are overviewed. Key concepts discussed include feature vectors, hidden states, empirical risk minimization, overfitting, and unsupervised learning algorithms.
Free Ebooks Download ! Edhole.com
- 1. Introduction to Pattern Recognition Vojtěch Franc xfrancv@cmp.felk.cvut.cz Center for Machine Perception Czech Technical University in Prague
- 2. What is pattern recognition? A pattern is an object, process or event that can be given a name. A pattern class (or category) is a set of patterns sharing common attributes and usually originating from the same source. During recognition (or classification) given objects are assigned to prescribed classes. A classifier is a machine which performs classification. “The assignment of a physical object or event to one of several prespecified categeries” -- Duda & Hart
- 3. Examples of applications • Optical Character Recognition (OCR) • Biometrics • Diagnostic systems • Military applications • Handwritten: sorting letters by postal code, input device for PDA‘s. • Printed texts: reading machines for blind people, digitalization of text documents. • Face recognition, verification, retrieval. • Finger prints recognition. • Speech recognition. • Medical diagnosis: X-Ray, EKG analysis. • Machine diagnostics, waster detection. • Automated Target Recognition (ATR). • Image segmentation and analysis (recognition from aerial or satelite photographs).
- 4. Approaches Statistical PR: based on underlying statistical model of patterns and pattern classes. Structural (or syntactic) PR: pattern classes represented by means of formal structures as grammars, automata, strings, etc. Neural networks: classifier is represented as a network of cells modeling neurons of the human brain (connectionist approach).
- 5. Basic concepts y x= nx x x 2 1 Feature vector - A vector of observations (measurements). - is a point in feature space . Hidden state - Cannot be directly measured. - Patterns with equal hidden state belong to the same class. X∈x x X Y∈y Task - To design a classifer (decision rule) which decides about a hidden state based on an onbservation. YX →:q Pattern
- 6. Example x= 2 1 x x height weight Task: jockey-hoopster recognition. The set of hidden state is The feature space is },{ JH=Y 2 ℜ=X Training examples )},(,),,{( 11 ll yy xx 1x 2x Jy = Hy =Linear classifier: <+⋅ ≥+⋅ = 0)( 0)( )q( bifJ bifH xw xw x 0)( =+⋅ bxw
- 7. Components of PR system Sensors and preprocessing Feature extraction Classifier Class assignment • Sensors and preprocessing. • A feature extraction aims to create discriminative features good for classification. • A classifier. • A teacher provides information about hidden state -- supervised learning. • A learning algorithm sets PR from training examples. Learning algorithmTeacher Pattern
- 8. Feature extraction Task: to extract features which are good for classification. Good features: • Objects from the same class have similar feature values. • Objects from different classes have different values. “Good” features “Bad” features
- 9. Feature extraction methods km m m 2 1 nx x x 2 11φ 2φ nφ km m m m 3 2 1 nx x x 2 1 Feature extraction Feature selection Problem can be expressed as optimization of parameters of featrure extractor . Supervised methods: objective function is a criterion of separability (discriminability) of labeled examples, e.g., linear discriminat analysis (LDA). Unsupervised methods: lower dimesional representation which preserves important characteristics of input data is sought for, e.g., principal component analysis (PCA). φ(θ)
- 10. Classifier A classifier partitions feature space X into class-labeled regions such that ||21 YXXXX ∪∪∪= }0{||21 =∩∩∩ YXXX and 1X 3X 2X 1X 1X 2X 3X The classification consists of determining to which region a feature vector x belongs to. Borders between decision boundaries are called decision regions.
- 11. Representation of classifier A classifier is typically represented as a set of discriminant functions ||,,1,:)(f YX =ℜ→ ii x The classifier assigns a feature vector x to the i-the class if )(f)(f xx ji > ij ≠∀ )(f1 x )(f2 x )(f || xY maxx y Feature vector Discriminant function Class identifier
- 12. Bayesian decision making • The Bayesian decision making is a fundamental statistical approach which allows to design the optimal classifier if complete statistical model is known. Definition : Obsevations Hidden states Decisions A loss function A decision rule A joint probabilityD DX →:q )p( y,x X Y RDYW →×: Task: to design decision rule q which minimizes Bayesian risk ∑∑∈ ∈ = Yy Xx yy )),W(q(),p(R(q) xx
- 13. Example of Bayesian task Task: minimization of classification error. A set of decisions D is the same as set of hidden states Y. 0/1 - loss function used ≠ = = yif yif y )q(1 )q(0 )),W(q( x x x The Bayesian risk R(q) corresponds to probability of misclassification. The solution of Bayesian task is )p( )p()|p( maxarg)|(maxargR(q)minargq * q * x x x yy ypy yy ==⇒=
- 14. Limitations of Bayesian approach • The statistical model p(x,y) is mostly not known therefore learning must be employed to estimate p(x,y) from training examples {(x1,y1),…,(x,y)} -- plug-in Bayes. • Non-Bayesian methods offers further task formulations: • A partial statistical model is avaliable only: • p(y) is not known or does not exist. • p(x|y,θ) is influenced by a non-random intervetion θ. • The loss function is not defined. • Examples: Neyman-Pearson‘s task, Minimax task, etc.
- 15. Discriminative approaches Given a class of classification rules q(x;θ) parametrized by θ∈Ξ the task is to find the “best” parameter θ* based on a set of training examples {(x1,y1),…,(x,y)} -- supervised learning. The task of learning: recognition which classification rule is to be used. The way how to perform the learning is determined by a selected inductive principle.
- 16. Empirical risk minimization principle The true expected risk R(q) is approximated by empirical risk ∑= = 1 emp )),;W(q( 1 ));(q(R i ii yx θxθ with respect to a given labeled training set {(x1,y1),…,(x,y)}. The learning based on the empirical minimization principle is defined as ));(q(Rminarg emp * θxθ θ = Examples of algorithms: Perceptron, Back-propagation, etc.
- 17. Overfitting and underfitting Problem: how rich class of classifications q(x;θ) to use. underfitting overfittinggood fit Problem of generalization: a small emprical risk Remp does not imply small true expected risk R.
- 18. Structural risk minimization principle An upper bound on the expected risk of a classification rule q∈Q ) 1 log,, 1 (R(q)RR(q) σ hstremp +≤ where is number of training examples, h is VC-dimension of class of functions Q and 1-σ is confidence of the upper bound. SRM principle: from a given nested function classes Q1,Q2,…,Qm, such that mhhh ≤≤≤ 21 select a rule q* which minimizes the upper bound on the expected risk. Statistical learning theory -- Vapnik & Chervonenkis.
- 19. Unsupervised learning Input: training examples {x1,…,x} without information about the hidden state. Clustering: goal is to find clusters of data sharing similar properties. Classifier Learning algorithm θ },,{ 1 xx },,{ 1 yy Classifier ΘY)(X: →× L YΘX →×:q Learning algorithm (supervised) A broad class of unsupervised learning algorithms:
- 20. Example of unsupervised learning algorithm k-Means clustering: Classifier ||||minarg)q( ,,1 i ki y mxx −== = Goal is to minimize ∑= − 1 2 )q( |||| i i ixmx ∑∈ = ij j i i II , || 1 xm })q(:{ ij ji == xI Learning algorithm 1m 2m 3m },,{ 1 xx },,{ 1 kmmθ = },,{ 1 yy
- 21. References Books Duda, Heart: Pattern Classification and Scene Analysis. J. Wiley & Sons, New York, 1982. (2nd edition 2000). Fukunaga: Introduction to Statistical Pattern Recognition. Academic Press, 1990. Bishop: Neural Networks for Pattern Recognition. Claredon Press, Oxford, 1997. Schlesinger, Hlaváč: Ten lectures on statistical and structural pattern recognition. Kluwer Academic Publisher, 2002. Journals Journal of Pattern Recognition Society. IEEE transactions on Neural Networks. Pattern Recognition and Machine Learning.