![Homogenous Ensembles Use a single, arbitrary learning algorithm but manipulate training data to make it learn multiple models. Data1 Data2 … Data m Learner1 = Learner2 = … = Learner m Different methods for changing training data: Bagging : Learns a committee of classifiers each trained on a different sample of the training data [Breiman ′ 96] Boosting : Learns a series of classifiers each one focusing on the errors made by the previous one [Freund & Schapire ′ 96] D ECORATE : Learns a series of classifiers by adding artificial training data to encourage diversity [Melville and Mooney ’03]](https://image.slidesharecdn.com/ppt-slides2379/85/PPT-SLIDES-9-320.jpg)
PPT SLIDES
- 1. Maximizing the Utility of Small Training Sets in Machine Learning Raymond J. Mooney Department of Computer Sciences University of Texas at Austin
- 2. Computational Linguistics and Machine Learning Manually encoding the large amount of knowledge needed for natural-language processing (NLP), e.g. grammars, lexicons, syntactic, semantic, and pragmatic preferences, etc., is difficult and time consuming. Machine learning techniques can automatically acquire such knowledge by discovering patterns in appropriately annotated corpora. Machine learning techniques (a.k.a. empirical methods , statistical NLP , corpus-based methods ) have been more effective at building accurate and robust NLP systems than previous “rationalist” methods based on human knowledge engineering. Therefore, machine learning approaches have come to dominate computational linguistics, causing a “scientific revolution” in the field.
- 3. Demand for Annotated Corpora Learning methods typically require large amounts of supervised training data in order to produce accurate results. Large annotated corpora have been constructed for popular languages such as English. Syntax: Treebanks Word Sense: SENSEVAL data Semantic Roles: FrameNet and PropBank Building large, clean, well-balanced, annotated corpora requires significant infrastructure and many hours of dedicated effort by expert linguists. Constructing similar large corpora for less-studied languages is frequently not practical.
- 4. Treebanks English Penn Treebank : Standard corpus for testing syntactic parsing consists of 1.2 M words of text from the Wall Street Journal (WSJ). Typical to train on about 40,000 parsed sentences and test on an additional standard disjoint test set of 2,416 sentences. Chinese Penn Treebank : 100K words from the Xinhua news service. Annotated corpora exist for several other languages, see the Wikipedia article “Treebank”
- 5. Learning from Small Training Sets Various machine learning methods have been developed for improving generalization performance when training data is limited. The value of such methods is evaluated using learning curves that plot accuracy vs. training-set size.
- 6. Methods for Improving Results on Small Training Sets Ensembles : Diverse committees of alternative hypotheses. Active Learning : Selecting the most informative examples for annotation and training. Transfer Learning : Exploiting and adapting knowledge for related problems. Unsupervised Learning : Learning from unannotated data. Semi-Supervised Learning : Learning from a combination of annotated and unannotated data.
- 7. Learning Ensembles Learn multiple alternative definitions of a concept using different training data or different learning algorithms. Combine decisions of multiple definitions, e.g. using weighted voting. Training Data Data1 Data m Data2 Learner1 Learner2 Learner m Model1 Model2 Model m Model Combiner Final Model
- 8. Value of Ensembles When combing multiple independent and diverse decisions each of which is at least more accurate than random guessing, random errors cancel each other out, correct decisions are reinforced. Human ensembles are demonstrably better How many jelly beans in the jar?: Individual estimates vs. group average. Who Wants to be a Millionaire: Expert friend vs. audience vote. Ensembles are particularly useful when training data is limited and therefore the variance across training samples and learning methods is more pronounced.
- 9. Homogenous Ensembles Use a single, arbitrary learning algorithm but manipulate training data to make it learn multiple models. Data1 Data2 … Data m Learner1 = Learner2 = … = Learner m Different methods for changing training data: Bagging : Learns a committee of classifiers each trained on a different sample of the training data [Breiman ′ 96] Boosting : Learns a series of classifiers each one focusing on the errors made by the previous one [Freund & Schapire ′ 96] D ECORATE : Learns a series of classifiers by adding artificial training data to encourage diversity [Melville and Mooney ’03]
- 10. D ECORATE (Melville & Mooney, 2003) Change training data by adding new artificial training examples that encourage diversity in the resulting ensemble. Improves accuracy when the training set is small, and therefore resampling and reweighting the training set has limited ability to generate diverse alternative hypotheses.
- 11. Base Learner Overview of D ECORATE Training Examples Artificial Examples Current Ensemble - - + + + C1 + + - + -
- 12. C1 Base Learner Overview of D ECORATE Training Examples Artificial Examples Current Ensemble - - + + + - - + - + C2 + - - - +
- 13. C1 C2 Base Learner Overview of D ECORATE Training Examples Artificial Examples Current Ensemble - + + + - - - + + + C3
- 14. Experimental Methodology Compared D ECORATE with Bagging, AdaBoost and J48 J48 is a Java implementation of the C4.5 decision tree learner. We use J48 as the base learner for the ensemble methods. An ensemble size of 15 was used. 10x10-fold cross-validation were run on 15 UCI datasets Learning curves were generated To test performance on varying amounts of training data. Selected different percentages of total available data as points on the learning curve. We chose 10 points ranging from 1-100%.
- 15. Learning Curve for Labor Contract Prediction Decorate achieves higher accuracies throughout the learning curve Small dataset (57 examples) – hence Decorate has an advantage
- 16. Learning Curve for Cancer Diagnosis Typically, performance of methods will converge given enough data. Mostly, Decorate achieves higher accuracy with fewer examples. Here it produces an accuracy > 92% with just 6 examples.
- 17. Active Learning Most randomly-chosen examples are not particularly informative since they illustrate common phenomena that have probably already been learned. In active learning , the system is responsible for selecting good training examples and asking a teacher (oracle) to provide a label. In sample selection , the system picks good examples to query by picking them from a provided pool of unlabeled examples. In query generation , the system must generate the description of an example for which to request a label. Goal is to minimize the number of queries required to learn an accurate concept description.
- 18. Ensembles and Active Learning Ensembles can be used to actively select good new training examples. Select the unlabeled example that causes the most disagreement amongst the members of the ensemble. Applicable to any ensemble method: QueryByBagging QueryByBoosting ActiveD ECORATE
- 19. D ECORATE Active-D ECORATE Training Examples Unlabeled Examples Current Ensemble C1 C2 C3 C4 Utility = 0.1 - - + + - + + + +
- 20. D ECORATE Active-D ECORATE Training Examples Unlabeled Examples Current Ensemble C1 C2 C3 C4 Utility = 0.1 0.9 - - + + - + + - - 0.3 0.2 0.5 + Acquire Label
- 21. Experimental Methodology Compared Active-Decorate with QBag, QBoost and Decorate (using random sampling) Used ensembles of size 15 Used J48 as the base learner 2x10-fold cross-validations were run on 15 UCI datasets In each fold, learning curves were generated The set of available examples treated as unlabeled pool At each iteration, the active learner selected sample of examples to be labeled and added to training set For passive learner, Decorate, examples were selected randomly At the end of the learning curve, all systems see the same training examples. The curves evaluate the how well an active learner orders the set of examples in terms of utility
- 22. Learning Curve for Soybean Disease Diagnosis ≈ 60% savings in supervision
- 23. Learning Curve for Spoken Vowel Recognition ≈ 50% savings in supervision
- 24. Transfer Learning a.k.a. Adaptation, Learning to Learn, Lifelong Learning Use learning on a previous related problem (the source) to improve learning on the current problem (the target) . Various approaches: Use model learned from source as a statistical prior for the target. Hierarchical Bayesian Models and Shrinkage Theory revision: Adapt learned source model to the target. Multitask Learning: Learn one model for multiple related tasks.
- 25. Using Source as a Prior Use a statistical model trained on the source to provide priors for estimating the parameters for the target. Requires the target and the source to have the same set of features. Equivalent to “corpus mixing” in which data from the source is mixed with data from the target prior to training. Usually weight the target data more heavily.
- 26. Corpus Mixing Target Training Examples Learner Classifier Source Training Examples - - + + + - - + + + - - + - - + - - +
- 27. Corpus Mixing Results (Roark and Bacchiani, 2003) Test transfer learning for statistical syntactic treebank parsing from one English corpus to another. Source training data is 21,818 sentences from the Brown corpus. Target data is from Wall Street Journal. Training set size varied. Test set of 2,245 sentences Target data weighted 5 times as much as source data. 85.40% 84.90% 10,000 sentences 84.35% 82.60% 4,000 sentences 83.05% 80.50% 2,000 sentences Transfer F-Measure Baseline F-Measure Target Domain Training Size
- 28. Transferring from One Language to Another Many transfer methods require the same features in the target and source. Since in computational linguistics, the features are typically words, this prevents transfer across languages. However, if a word-aligned parallel bilingual corpus is available, annotation can be “projected” from a source to a target language. Statistical word alignment tools like GIZA++ can be used to align words in a parallel bilingual corpus. Once annotation has been projected across a parallel corpus from a source to target language, the resulting data can be used to train an analyzer in the target domain.
- 29. Projecting a POS Tagger (Yarowsky & Ngai, 2001) English : a significant producer for crude oil French : un producteur important de petrole brut Word alignment DT JJ NN IN JJ NN DT NN JJ IN NN JJ Projected POS Tags English POS Tagger French POS Tagger POS Tag Learner
- 30. POS Tagging Transfer Results (Yarowsky & Ngai, 2001) Evaluate on English-French Canadian Hansards parallel corpus (2 million words). Core : 94% Full : 91% Core : 96% Full : 93% Trained on Projected Data Core : 98% Full : 97% Core : 97% Full : 96% Directly Trained on 100K French Words N/A Core : 76% Full : 69% Project from English Novel French Aligned French Model
- 31. Unsupervised Learning Unannotated text is typically much easier to obtain than annotated text. However, purely unsupervised learning typically does not result in the desired analyses. Early results on unsupervised induction of probabilistic context grammars was very disappointing (Lari & Young, 1990). They tend to find structure in data that reflects a complex combination of semantic and syntactic regularities. This lead to the focus on developing supervised treebanks. Recent unsupervised learning methods using appropriately constrained probabilistic dependency models have successfully induced grammatical structure from unannotated text (Klein and Manning, 2002; 2004)
- 32. Semi-Supervised Learning Use a combination of unlabeled and labeled data to improve accuracy. Typically labeled set is small and unlabeled set is much larger since it is easier to obtain. Methods for semi-supervised learning: Self-labeling and semi-supervised EM Ghaharami & Jordan, 1994; Nigam et al ., 2000 Co-training Blum & Mitchell, 1998 Transductive Support Vector Machines (SVM’s) Vapnik, 1998; Joachims, 1999 Hidden Markov Random Field (HMRF) Basu, Bilenko, & Mooney, 2004
- 33. Self-Labeling - + Unlabeled Examples + + - Training Examples - - + + + Learner Classifier
- 34. Self-Labeling Classifier retrained on automatically labeled data is frequently more accurate Training Examples - - + + + Learner Classifier - + + + -
- 35. Semi-Supervised EM Unlabeled Examples Training Examples - - + + + Prob. Learner Prob. Classifier + + + + +
- 36. Semi-Supervised EM Training Examples - - + + + Prob. Learner + + + + + Prob. Classifier
- 37. Semi-Supervised EM Training Examples - - + + + Prob. Learner + + + + + Prob. Classifier
- 38. Semi-Supervised EM Unlabeled Examples Training Examples - - + + + Prob. Learner Prob. Classifier + + + + +
- 39. Semi-Supervised EM Continue retraining iterations until probabilistic labels on unlabeled data converge. Training Examples - - + + + Prob. Learner + + + + + Prob. Classifier
- 40. Semi-Supervised EM Results Experiments on assigning messages from 20 Usenet newsgroups their proper newsgroup label. With very few labeled examples (2 examples per class), semi-supervised EM significantly improved predictive accuracy: 27% with 40 labeled messages only. 43% with 40 labeled + 10,000 unlabeled messages. With more labeled examples, semi-supervision can actually decrease accuracy, but refinements to standard EM can help prevent this. Must weight labeled data appropriately more than unlabeled data. For semi-supervised EM to work, the “natural clustering of data” must be consistent with the desired categories Failed when applied to English POS tagging (Merialdo, 1994)
- 41. Semi-Supervised EM Example Assume “Catholic” is present in both of the labeled documents for soc.religion.christian, but “Baptist” occurs in none of the labeled data for this class. From labeled data, we learn that “Catholic” is highly indicative of the “Christian” category. When labeling unsupervised data, we label several documents with “Catholic” and “Baptist” correctly with the “Christian” category. When retraining, we learn that “Baptist” is also indicative of a “Christian” document. Final learned model is able to correctly assign documents containing only “Baptist” to “Christian”.
- 42. Semi-Supervised Clustering Uses limited supervision to aid unsupervised clustering of data. Does not assume the user has a predetermined set of known classes in mind. Supervision is typically given in the form of pairwise constraints : Must-link : These two instances should be in the same class. Cannot-link : These two instances should be in different classes.
- 43. Semi-Supervised Clustering with Pairwise Constraints Linguist Prof Student # Publications Programming Ability Computer Scientist 2-way clustering
- 44. Semi-supervised Clustering with Pairwise Constraints Linguist Prof Student # Publications Programming Ability Computer Scientist 2-way clustering Cannot-link Must-link
- 45. Semi-Supervised Clustering with Hidden Markov Random Fields (HMRFs) HMRFs provide a well-founded probabilistic model for clustering data (Basu, Bilenko, & Mooney, 2004) that considers both: Similarity between instances in a cluster. Consistency with supervisory pairwise constraints. Variant of the k -means clustering algorithm was developed for inferring the most likely class assignments in an HMRF model. Active-learning algorithm was also developed for selecting informative pairwise supervision queries (Basu, Banerjee, & Mooney, 2004). Should these two examples be put in the same class?
- 46. Active Semi-Supervised Clustering on Classifying Messages from 3 Newsgroups talk.politics.misc vs. talk.politics.guns, vs. talk.politics.mideast ≈ 80% savings in supervision!
- 47. Conclusions Typically, machine learning and data mining methods are seen as requiring large amounts of (annotated) training data. However, a variety of techniques have been developed for improving the accuracy of models learned from small training sets. Ensembles Active Learning Transfer Learning Unsupervised Learning Semi-Supervised Learning These techniques (and others) may help develop robust computational-linguistics tools from the limited data available for less studied languages.
Editor's Notes
- #16: So far we have looked at learning curve statistics summarized over many datasets. But we can also look at learning curves for individual datasets. I’ll present a couple of datasets that clearly demonstrate the improvements that Decorate can bring. We have plotted here the accuracies of the Decorate, bagging and boosting given varying amounts of training data from the Labor dataset. We note that Decorate achieves higher accuracies throughout the learning curve. This is primarily because Labor is quite a small dataset, with approximately 60 examples. So bagging and boosting are limited in the amount of diversity they can produce – as discussed earlier.
- #17: More typically the performance of the 3 ensemble methods will converge given enough data. But in most cases Decorate achieves higher accuracy given fewer examples. In the breast cancer dataset, shown here, Decorate produces an accuracy greater than 92% with just 6 examples. (Boosting & Bagging produce almost no improvement over the base learner’s accuracy of 75%)