K Nearest Neighbors
Download as PPTX, PDF30 likes58,026 views
The document discusses the K-nearest neighbors (KNN) algorithm, a simple machine learning algorithm used for classification problems. KNN works by finding the K training examples that are closest in distance to a new data point, and assigning the most common class among those K examples as the prediction for the new data point. The document covers how KNN calculates distances between data points, how to choose the K value, techniques for handling different data types, and the strengths and weaknesses of the KNN algorithm.
![Distance Between Neighbors
• Calculate the distance between new example
(E) and all examples in the training set.
• Euclidean distance between two examples.
– X = [x1,x2,x3,..,xn]
– Y = [y1,y2,y3,...,yn]
– The Euclidean distance between X and Y is defined
as:
11
n
i
ii yxYXD
1
2
)(),(](https://image.slidesharecdn.com/knn-160904075605/85/K-Nearest-Neighbors-11-320.jpg)
![3-KNN: Example(1)
Distance from John
sqrt [(35-37)2+(35-50)2 +(3-
2)2]=15.16
sqrt [(22-37)2+(50-50)2 +(2-
2)2]=15
sqrt [(63-37)2+(200-50)2 +(1-
2)2]=152.23
sqrt [(59-37)2+(170-50)2 +(1-
2)2]=122
sqrt [(25-37)2+(40-50)2 +(4-
2)2]=15.74
13
Customer Age Income No.
credit
cards
Class
George 35 35K 3 No
Rachel 22 50K 2 Yes
Steve 63 200K 1 No
Tom 59 170K 1 No
Anne 25 40K 4 Yes
John 37 50K 2 ? YES](https://image.slidesharecdn.com/knn-160904075605/85/K-Nearest-Neighbors-13-320.jpg)
K Nearest Neighbors
- 1. Tilani Gunawardena Algorithms: K Nearest Neighbors 1
- 2. Algorithms: K Nearest Neighbors 2
- 3. Simple Analogy.. • Tell me about your friends(who your neighbors are) and I will tell you who you are. 3
- 4. Instance-based Learning Its very similar to a Desktop!! 4
- 5. KNN – Different names • K-Nearest Neighbors • Memory-Based Reasoning • Example-Based Reasoning • Instance-Based Learning • Lazy Learning 5
- 6. What is KNN? • A powerful classification algorithm used in pattern recognition. • K nearest neighbors stores all available cases and classifies new cases based on a similarity measure(e.g distance function) • One of the top data mining algorithms used today. • A non-parametric lazy learning algorithm (An Instance- based Learning method). 6
- 7. KNN: Classification Approach • An object (a new instance) is classified by a majority votes for its neighbor classes. • The object is assigned to the most common class amongst its K nearest neighbors.(measured by a distant function ) 7
- 8. 8
- 9. Distance Measure Training Records Test Record Compute Distance Choose k of the “nearest” records 9
- 10. Distance measure for Continuous Variables 10
- 11. Distance Between Neighbors • Calculate the distance between new example (E) and all examples in the training set. • Euclidean distance between two examples. – X = [x1,x2,x3,..,xn] – Y = [y1,y2,y3,...,yn] – The Euclidean distance between X and Y is defined as: 11 n i ii yxYXD 1 2 )(),(
- 12. K-Nearest Neighbor Algorithm • All the instances correspond to points in an n-dimensional feature space. • Each instance is represented with a set of numerical attributes. • Each of the training data consists of a set of vectors and a class label associated with each vector. • Classification is done by comparing feature vectors of different K nearest points. • Select the K-nearest examples to E in the training set. • Assign E to the most common class among its K-nearest neighbors. 12
- 13. 3-KNN: Example(1) Distance from John sqrt [(35-37)2+(35-50)2 +(3- 2)2]=15.16 sqrt [(22-37)2+(50-50)2 +(2- 2)2]=15 sqrt [(63-37)2+(200-50)2 +(1- 2)2]=152.23 sqrt [(59-37)2+(170-50)2 +(1- 2)2]=122 sqrt [(25-37)2+(40-50)2 +(4- 2)2]=15.74 13 Customer Age Income No. credit cards Class George 35 35K 3 No Rachel 22 50K 2 Yes Steve 63 200K 1 No Tom 59 170K 1 No Anne 25 40K 4 Yes John 37 50K 2 ? YES
- 14. How to choose K? • If K is too small it is sensitive to noise points. • Larger K works well. But too large K may include majority points from other classes. • Rule of thumb is K < sqrt(n), n is number of examples. 14 X
- 15. 15
- 16. X X X (a) 1-nearest neighbor (b) 2-nearest neighbor (c) 3-nearest neighbor K-nearest neighbors of a record x are data points that have the k smallest distance to x 16
- 17. KNN Feature Weighting • Scale each feature by its importance for classification • Can use our prior knowledge about which features are more important • Can learn the weights wk using cross‐validation (to be covered later) 17
- 18. Feature Normalization • Distance between neighbors could be dominated by some attributes with relatively large numbers. e.g., income of customers in our previous example. • Arises when two features are in different scales. • Important to normalize those features. – Mapping values to numbers between 0 – 1. 18
- 19. Nominal/Categorical Data • Distance works naturally with numerical attributes. • Binary value categorical data attributes can be regarded as 1 or 0. 19
- 20. KNN Classification $0 $50,000 $100,000 $150,000 $200,000 $250,000 0 10 20 30 40 50 60 70 Non-Default Default Age Loan$ 20
- 21. KNN Classification – Distance Age Loan Default Distance 25 $40,000 N 102000 35 $60,000 N 82000 45 $80,000 N 62000 20 $20,000 N 122000 35 $120,000 N 22000 52 $18,000 N 124000 23 $95,000 Y 47000 40 $62,000 Y 80000 60 $100,000 Y 42000 48 $220,000 Y 78000 33 $150,000 Y 8000 48 $142,000 ? 2 21 2 21 )()( yyxxD 21
- 22. KNN Classification – Standardized Distance Age Loan Default Distance 0.125 0.11 N 0.7652 0.375 0.21 N 0.5200 0.625 0.31 N 0.3160 0 0.01 N 0.9245 0.375 0.50 N 0.3428 0.8 0.00 N 0.6220 0.075 0.38 Y 0.6669 0.5 0.22 Y 0.4437 1 0.41 Y 0.3650 0.7 1.00 Y 0.3861 0.325 0.65 Y 0.3771 0.7 0.61 ? MinMax MinX Xs 22
- 23. Strengths of KNN • Very simple and intuitive. • Can be applied to the data from any distribution. • Good classification if the number of samples is large enough. 23 Weaknesses of KNN • Takes more time to classify a new example. • need to calculate and compare distance from new example to all other examples. • Choosing k may be tricky. • Need large number of samples for accuracy.
Editor's Notes
- #4: Based on similarity function “tell me who your neighbors are, and I’ll tell you who you are” Simplest Algorithms
- #7: KNN stattistical estimation Pattern recognition in the beginning of 1970
- #8: Algorithm
- #9: Weight and hight of people and we assume Red : female Blue: Male So we have new measurement of hight and weight … K=1,female ,we assigned class =female K=5 3 female, 2 males K=odd K=6 we have ties
- #10: Although there are other possible choices, most instance-based learners use Euclid- ean distance.
- #11: Although there are other possible choices, most instance-based learners use Euclid- ean distance. Other distance metrics may be more appropriate in special circumstances. If we have too many points Sum the squared differences and we get the square roots In Manhanttan distances : get absoulte values Min-ko-waski distance= Different distance measure .. We use the common one
- #12: N-feature size
- #13: This single number K is given prior to the testing phase and this number decides how many neighbors influence the classification.
- #15: If the sample set is finite.. Historically the optomal K for mpst datasets has been between 3-10 . That produces much better results than 1NN
- #19: Some configurations over data before applying KNN Different attributes are often measured on different scales, Numerical: Here the difference between two values is just the numerical difference between them, and it is this difference that is squared and added to yield the distance functio Nominal: the difference between two values that are not the same is often taken to be 1, whereas if the values are the same the difference is 0. No scaling is required in this case because only the values 0 and 1 are used.
- #20: For attributes with more than two values, binary indicator variables are used as an implicit solution. Ex: In a customers’ Bank field with 3 values, Bank1 – 100 Bank2 – 010 Bank3 - 001
- #23: That’s price to pay
- #24: Robust: strong and healthy; vigorou