Artificial Intelligence
- 1. Artificial Intelligence & Computer Vision Lab School of Computer Science and Engineering Seoul National University Machine Learning Instance-based Learning
- 2. Overview Introduction k -Nearest Neighbor Learning Locally Weighted Regression Radial Basis Functions Case-based Reasoning Remarks on Lazy and Eager Learning Summary
- 3. Introduction Approaches in previous chapters Training examples New instances Classified Target function …
- 4. Introduction (cont.) Instance-based learning Lazy : processing is postponed Until queries are encountered Stores all training examples When a new query is encountered, examples related to the query instance are retrieved and processed Training examples
- 5. Introduction (cont.) Can construct a different approx. for target function for each query Local approximation Can use more complex and symbolic representation for instances (Case-based reasoning) Cost of classifying new instances is high Indexing is a significant issue
- 6. k -Nearest Neighbor Learning Inductive bias The classification of a new instance is most similar to the one of other instances that are nearby in Euclidean distance. Assumption All instances are correspond to points in the n-dimensional space R n Distance from x i to x j is given by Target function Discrete or real-valued
- 7. Illustrative example 5-nearest neighbor 2-dim. data Target class : boolean ( + or -) k -Nearest Neighbor Learning (cont.) + and - : location and target value of training instances x q : query instance
- 8. Discrete-valued target function x 1 ~ x k : nearest k instances V : set of target values (v) Real-valued target function k -Nearest Neighbor Learning (cont.)
- 9. k-NN never forms explicit general hypothesis Implicit Voronoi diagram 1-nearest neighbor k -Nearest Neighbor Learning (cont.) query point nearest neighbor
- 10. 3 target classes Red, Green, Blue 2-dim. data k -Nearest Neighbor Learning (cont.)
- 11. Large k Less sensitive to noise (particularly class noise) Better probability estimates Small k Captures fine structure of problem space better Cost less k -Nearest Neighbor Learning (cont.)
- 12. Tradeoff between small and large k Want to use large k , but more emphasis on nearer neighbors Weight nearer neighbors more heavily Use all training examples instead of just k (Stepard’s method) k -Nearest Neighbor Learning (cont.) Discrete-valued Real-valued Distance-weighted NN
- 13. Consider ‘all’ attributes Decision tree : Subset of attributes considered What if just some of attributes are relevant to target value? ⇒ Curse of Dimensionality Solutions to Curse of Dimensionality 1. Weight each attribute differently 2. Eliminate the least relevant attributes Cross validation To determine scaling factors for each attributes Leave-one-out cross-validation (for method 2 above) k -Nearest Neighbor Learning (cont.) Remarks
- 14. Indexing is important Significant computation is required at query time Because of ‘lazy’ kd -tree (Bentley 75, Friedman et al. 1977) k -Nearest Neighbor Learning (cont.)
- 15. Locally Weighted Regression Approximation to f over a local region surrounding x q Produce “piecewise approximation” to f k-NN : local approximation to f for each query point x q Global regression : global approximation to f Approximated function f Used to get the estimated target value f ( x q ) Different local approximation for each distinct query Various forms Constant, linear function, quadratic function, … ^ ^
- 16. “ Locally Weighted Regression” “ locally” The function is approximated based only on data near the query point “ weighted” Contribution of each training example is weighted by its distance from the query point “ regression” approximating a real-valued function Locally Weighted Regression (cont.)
- 17. Approximated linear function Should choose weights that minimize the sum of squared error Can apply gradient descent rule Locally Weighted Regression (cont.)
- 18. Global approximation -> local approximation For just the k nearest neighbors Apply weight for all instances Combine above two Using gradient descent rule Locally Weighted Regression (cont.)
- 19. A broad range of method for approximating the target function Constant, linear, quadratic function More complex functions are not common Fitting is costly Simple forms suffice for small subregion of instance space Locally Weighted Regression (cont.)
- 20. An approach to function approximation related to distance-weighted regression and also to artificial neural networks. Approximated function Linear combination of radial kernel functions f(x) is a global approximation for f(x) K u (d(x u ,x)) is localized to region nearby x u Radial Basis Functions ^
- 21. K(d(x, y)) : kernel function Decreases as distance d(x, y) increases E.g. Gaussian function Radial Basis Functions (cont.) Using Gaussian radial basis functions Using sigmoidal radial basis functions
- 22. RBF Networks Two-layered network 1 st layer : computes K u 2 nd layer : computes weighted linear sum of values from 1 st layer Uses (typically) Gaussian kernel function Radial Basis Functions (cont.)
- 23. Training RBFN : 2 phases 1 st phase Number k of hidden units is determined For each hidden unit u , choose x u and σ u 2 nd phase For each u , weights w u is trained Efficiently trained (kernel functions are already determined) Radial Basis Functions (cont.)
- 24. Choosing number k of hidden units (kernel functions) 1. allocate a kernel function for each training example ( k = number of training examples) each training example < x i , f(x i ) > can influence the value of the approximated function only in the neighborhood of x i . costly 2. Choose ( k < number of training examples) much more efficient than above center x u should be determined uniformly centered throughout instance space randomly selecting a subset of training examples, thereby sampling the underlying distribution of instances identify clusters of instances, then add a kernel function centered at each cluster ( EM algorithm applied) Radial Basis Functions (cont.)
- 25. Key advantage of RBFN Can be trained much more efficiently than feedfoward networks trained with Backpropagation because input layer and output layer of an RBFN are trained separately RBFN provides a global approximation to the target function, represented by a linear combination of many local kernel functions. The value for any given kernel function is non-negligible only when the input falls into the region defined by its particular center and width. Thus, the network can be viewed as a smooth linear combination of many local approximations to the target function. Radial Basis Functions (cont.)
- 26. Case-Based Reasoning Problem solving paradigm which utilizes specific knowledge experienced from concrete problem situations or cases. by remembering a previous similar situation and by reusing information and knowledge of that situation based on human information processing (HIP) model in some problem areas
- 27. Use much complex representation for instances Can be applied to problems such as Conceptual design of mechanical device Reasoning about new legal cases on prev. rulings Solving planning and scheduling problems by reusing and combining portions of previous solutions to similar problems Case-Based Reasoning (cont.)
- 28. CADET (Sycara et al. 1992): Conceptual design of simple mechanical devices Each training example < qualitative function, mechanical structure> New query : desired function Target value : mechanical structure for this function Process If an exact match is found, then this case can be returned If no exact match occurs, find cases that match various subgraphs of the desired function. By retrieving multiple cases that match different subgraphs, the entire design can be pieced together. In general, the process of producing a final solution from multiple retrieved cases can be very complex. Case-Based Reasoning (cont.)
- 29. CADET example Design of water faucet Case-Based Reasoning (cont.)
- 30. Instances or cases may be represented by rich symbolic descriptions, such as function graphs used in CADET. This may require a similarity metric such as the size of the largest shared subgraph between two function graphs. Multiple retrieved cases may be combined to form the solution to the new problem, which relies on knowledge-based reasoning rather than statistical methods as in k-Nearest Neighbor approach. Tight coupling exists between case retrieval, knowledge-based reasoning, and search-intensive problem solving. Case-Based Reasoning (cont.)
- 31. Current research issue is to develop improved methods for indexing cases: Syntactic similarity measures, such as subgraph isomorphism between function graphs , provides only as approximate indication of the relevance of a particular case to a particular problem. When the CBR system attempts to reuse the retrieved cases, it may uncover difficulties that were not captured by this syntactic similarity measures: For example, in CADET the multiple retrieved design fragments may turn out be incompatible with one another, making it impossible to combine them into a consistent final design. When this occurs in general, the CBR system may backtrack and search for additional cases, adapting the existing cases, or resort to other problem-solving methods. In particular, if a case is retrieved based on the similarity metric but found to be irrelevant based on further analysis, then the similarity metric should be refined to reject this case for similar subsequent queries. Case-Based Reasoning (cont.)
- 32. Remarks on Lazy and Eager Learning Lazy Learning Method Generalization is delayed until each query is encountered Can consider the query when deciding how to generalize k-Nearest Neighbor, Locally Weighted Regression, Case-Based Reasoning, … Eager Learning Method Generalization beyond entire training set Radial Basis Function Networks, C4.5, Backpropagation, …
- 33. Computation time Lazy methods : less for training, but more for querying Eager methods : more for training, but less for querying Generalization accuracy Given the same hypothesis space H , Eager method provides global single approximation hypothesis Lazy method provides many different local approximation hypothesis Radial basis function networks Eager but Use multiple local approximation But not the same as lazy… Uses pre-determined center, not query instance Remarks on Lazy and Eager Learning (cont.)