Data mining assignment 6
- 1. Data mining Assignment week 6 BARRY KOLLEE 10349863
- 2. Assignment 6 Exercise 1: Lazy Learning How is lazy learning different from the other machine learning approaches we have covered so far? In the previous lectures and exercises we’ve been working on how to predict various types of data. The approach we've used so far is to split data into a test set and training set. With these sets we were able to determine several solutions for handling the data which we eventually process. The distinction between the Lazy learning system and the previous systems is that a Lazy system stores it's training data and uses it directly on the data which we process further on. In the previous lectures the training data was necessary in the first place. Thereafter we’ve determined several conditions which gave us the opportunity to predict new values/instances. In conclusion we can state that we’ve determined a formula to handle new data and after words the training data were we base our formula’s on is not necessary anymore. With a lazy learning system we always refresh our training data to be able to predict the best. A disadvantage of this ‘lazy learning system’ is that it takes a lot of storage to handle these systems. 2
- 3. Assignment 6 Exercise 2: k Nearest Neighbor Classification 2.1 How does a form of overfitting affect kNN classification, and what can be done to overcome this? The kNN classifier is used to classify new instances for an existing dataset. To classify new instances we search for the nearest point/neighbor of this instance. For using this kNN algorithm a certain variable is taken into account. This variable represents the number of nearest points which we need to take into account for classifying new instances. In case we’re only looking for one (or a view) nearest point(s) it’s easily possible to create overfitting. To describe the overfitting of the kNN algorithm I’ve used these three examples which are listed below. Example 1: In this example we try to look for a plus or a minus which is closes to the red dot. We use a k (number of points which we take into account) of 1. Because the red dot is closest to the plus we ‘generalize’ the red dot to a plus value. Example 2: In this example we try to look for a plus or a minus which is closes to the red dot. We use a k (number of points which we take into account) of 1. Because the red dot is closest to the min we ‘generalize’ the red dot to a minus value. Example 3: In this example we try to look for a plus or a minus which is closes to the red dot. We use a k (number of points which we take into account) of 3. We now check for the three nearest neighbors so now it will be harder to predict the value of the red dot. If we draw a circular line as it is listed in the screenshot you see that the same dot from example 1 has now a plus value in stead of a minus because there are two plusses and one minus in the closest region of the red dot. What you can see within example 2 and 3 is that the red dot was classified as a minus. But when we increased the ‘k’ value it became clear that this red dot it’s value has been swapped now. In conclusion we can state that we needed to increase the k value to increase the sensitivity and correctness of our classifier. If we take a lower ‘k’ value then the sensitivity of our classifier goes upwards and generalizing/overfitting is more likely to occur. 3
- 4. Assignment 6 2.2 Given the following data: How does the kNN algorithm classify instances 7 and 8, with k = 1 and k = 3? You can use simple majority voting and the Manhattan distance. I’ve chosen to use the manhattan distance measure method to classify instance 7 and 8. First we compare instance 7 with all other attributes. We note all distances. Instance 1 0.25 – 0.25 = 0.00 0.55 – 0.25 = 0.30 Distance = 0.30 Instance 2 0.25 – 0.25 = 0.00 0.55 – 0.75 = -0.20 Distance = |-0.20| = 0.20 Instance 3 0.25 – 0.50 = -0.25 0.55 – 0.25 = 0.30 Distance = |-0.25| + 0.30 = 0.55 Instance 4 0.25 – 0.50 = -0.25 0.55 – 0.75 = -0.20 Distance = |-0.25| + |-0.20| = 0.45 Instance 5 0.25 – 0.75 = -0.50 0.55 – 0.50 = 0.05 Distance = |-0.50| + 0.05 = 0.55 Instance 6 0.25 – 0.75 = -0.50 0.55 – 1.00 = -0.45 Distance = |-0.50| + |-0.45| = 0.95 1. If we use a k-value of 1 we check for just one instance which is closest to instance 7. This is done by checking the instance which has the shortest distance. This instance is instance 2. So we classify instance 7 with a plus(+). 2. Now we check for a k value of 3. So for the three nearest points (with the shortest distance to instance 7). These instances are 1,2 and 4. We classify instance 7 with a plus(+) because 2/3 of the nearest neighbors are classified as a plus. 4
- 5. Assignment 6 Now we compare instance 8 with all other attributes. We note all distances. Instance 1 0.75 – 0.25 = 0.50 0.80 – 0.25 = 0.55 Distance = 1.05 Instance 2 0.75 – 0.25 = 0.50 0.80 – 0.75 = 0.05 Distance = 0.55 Instance 3 0.75 – 0.50 = 0.25 0.80 – 0.25 = 0.55 Distance = 0.80 Instance 4 0.75 – 0.50 = 0.25 0.80 – 0.75 = 0.05 Distance = 0.30 Instance 5 0.75 – 0.75 = 0.00 0.80 – 0.50 = 0.30 Distance = 0.30 Instance 6 0.75 – 0.75 = 0.00 0.80 – 1.00 = -0.2 Distance = |-0.20| = 0.20 1. If we use a k-value of 1 we check for just one instance which is closest to instance 8. This is done by checking the instance which has the shortest distance. This instance is instance 6. So we classify instance 8 with a plus(+). 2. Now we check for a k-value of 3. So the three points with the shortest distance to instance 8. These instances are 4,5 and 6. We classify instance 8 with a minus(-) because 2/3 of the nearest neighbors are classified as a minus. 5
- 6. Assignment 6 2.3 Given the dataset from question 2.3, how are instances 7 and 8 classified when using the prototype classifier, and what are the coordinates of the prototypes (i.e., x1 and x2 values?) For classifying both instance 7 and 8 with the prototype classifier we need to define a ‘super’-plus and a ‘super’-minus. These two values will represent the average value for all plus and minus classified instances. These are represented as a new ‘average’-instance. First we calculate a ‘super’-plus: Super-plus x1 value = ((0.25 + 0.25 + 0.75) / 3) Super-plus x1 value = 0.46 Super-plus x2 value = ((0.25 + 0.75 + 1.00) / 3) Super-plus x2 value = 0.67 Now we calculate a ‘super’-minus: Super-minus x1 value = ((0.50 + 0.50 + 0.75) / 3) Super-minus x1 value = 0.58 Super-minus x2 value = ((0.25 + 0.75 + 0.50) / 3) Super-minus x2 value = 0.50 Classifying instance 7 Now we use the manhattan distance to compute the total difference of the super-plus compared to instance 7: Distance x1 = Super-plus x1 value – instance_7 x1 value = Distance x1 = 0.46 – 0.25 = 0.21 Distance x2 = Super-plus x2 value – instance_7 x2 value Distance x2 = 0.67 – 0.55 = 0.17 Distance = 0.21 + 0.17 = 0.38 And finally we compute the total difference between the super minus and instance 7: Distance x1 = Super-minus x1 value – instance_7 x1 value = Distance x1 = 0.58 – 0.25 = 0.33 Distance x2 = Super- minus x2 value – instance_7 x2 value Distance x2 = 0.50 – 0.55 = 0.05 Distance = 0.3 + 0.55 = 0.85 We see that the smallest distance is between the super plus and instance 7. So we classify instance 7 as a plus(+). 6
- 7. Assignment 6 Classifying instance 8 Now we use the manhattan distance to compute the total difference of the super-plus compared to instance 8: Distance x1 = Super-plus x1 value – instance_8 x1 value = Distance x1 = 0.46 – 0.75 = -0.29 Distance x2 = Super-plus x2 value – instance_8 x2 value Distance x2 = 0.67 – 0.80 = -0.13 Distance = |-0.29| + |-0.13| = 0.42 And finally we compute the total difference between the super minus and instance 8: Distance x1 = Super-minus x1 value – instance_8 x1 value = Distance x1 = 0.58 – 0.75 = -0.17 Distance x2 = Super- minus x2 value – instance_8 x2 value Distance x2 = 0.50 – 0.80 = -0.30 Distance = |-0.17| + |-0.30| = 0.47 We see that the smallest distance is between the super plus and instance 8. So we classify instance 8 as a plus(+). 7