Gradient Boosted trees
Download as PPTX, PDF0 likes3,075 views
The document provides an overview of data mining, focusing on the significance and applications of gradient boosted trees as a machine learning technique. It explains the basic models used in data mining, such as predictive, descriptive, and prescriptive models, and contrasts ensemble methods like bagging and boosting for improving predictive performance. Additionally, it delves into the mechanics of gradient boosted trees, including their pros and cons, as well as details on techniques such as AdaBoost and general gradient boosting methods.
Gradient Boosted trees
- 1. Gradient boosted trees Nihar Ranjan
- 2. Data Mining Data Mining : It is a process of extracting patterns from data. They should be: Valid: holding on to new data with some certainity Novel: being non-obvious to the system. Useful: should be possible to act on the item Understandable: Humans should be able to interpret the pattern. Also known as Knowledge Discovery in Databases (KDD).
- 3. Data Mining might mean: Statistics Visualizatiom Artificial Intelligence Database Technology Machine Learning Neural Networks Information Retreival Knowledge-based systems Knowledge acquisition Pattern Recognition High performance computing And so on….
- 4. What's needed? Suitable data Computing power Data mining software Someone who knows both the nature of data and the software tools. Reason, theory or hunch
- 5. Typical applications of Data Mining and KDD Data Mining and KDD have widespread applications. Some examples include: Marketing Healthcare Financial services And so on….
- 6. Some basic techniques Predictive model: It basically describes what will happen in the future,rather predicts by analyzing the given current data. It uses statistical analysis, machine learning algorithms and other forecast techniques to predict what might happen in the future.It is not accurate as it is essentially just a prediction into the future using the data and the given stastistical/Machine Learning techniques. Eg- Performance Analysis. Descriptive model: It basically gives a vision into the past and tells what exactly happened in the past. It involves Data Aggregation and Data Mining.It is accurate as it describes exactly what happened in the past. Eg- Sentiment Analysis. Prescriptive model: This is realtively new field in Data Science.It is a step above predictive and descriptive model. It basically provides a viable solution to the problem in hand and the impact of considering a solution on future trend.It is still an evolving technique. Eg- Google self driving car.
- 7. Some basic techniques Predictive Regression Classification Collaborative Filtering Descriptive Clustering Association rules and variants Deviation detection
- 8. Key data mining tasks Classification: mapping data into predefined groups or classes. Regression: mapping data item to a real valued prediction variable. Clustering: Grouping similar data together into clusters.
- 9. Key learning tasks in Machine Learning Supervised learning: A set of well-labled data is given with defined inputs and outputs variables (training data ) and the algorithms learn to predict the output from the input data. Unsupervised learning: Data given is not labelled ie. only input variables are given with no corresponding output variables. The algorithms find patterns and draw inferences from the given data. This is "pure Data Mining". Semi-supervised: Some data is labeled but most of it is unlabeled and a mixture of supervised and unsupervised techniques can be used.
- 10. Some basic Data Mining Methods Decision Trees Neural Networks Cluster/Nearest Neighbour Genetic Algorithms/Evolutionary Computing Bayesien Networks Statistics Hybrids
- 11. Gradient boosted trees We are interested in Gradient boosted trees. We would use Rapidminer (possibly Python?)
- 12. Gradient boosted trees Decision Trees We will discuss a bit about decision trees first. A decision tree is a tree where each node represents a feature(attribute), each link(branch) represents a decision(rule) and each leaf represents an outcome(categorical or continues value). A decision tree takes a set of input features and splits input data recursively based on those features. The processes are repeated until some stop condition is met. Ex- Depth of tree, no more information gain possible etc.
- 13. Gradient boosted trees Decision Trees have been there for a long time and have also known to suffer from bias and variance. We have a large bias with simple trees and large variance with complex trees. Ensemble methods combine several decision trees to produce better predictive performance rather than utilizing a single decision tree. The main principle behind the ensemble model is that a group of weak learners come together to form a strong learner. A few ensemble methods : Bagging, Boosting We will see each of them.
- 14. Gradient boosted trees Bagging It's used when our goal is to reduce the variance of the decision tree. Here the idea is to take a súbset of data from training sample chosen randomly with replacement. Now, each collection of subset data is used to train their decision trees. Thus we end up with ensemble of different models and their average is much more robust than a single decision tree,which is much more robust in Predictive Analysis. Random Forest is an extension of Bagging.
- 15. Gradient boosted trees Random Forest It is basically a collection or ensemble of model of numerous decision trees. A collection of trees is generally called forest. It is also a bagging technique with a key difference, it takes a subset of features at each split , and prune the trees with a stopping criteria for node splits. The tree is grown to the largest. The above steps are repeated and the prediction is given based on the aggregation of predictions from n number of trees. Used for both classification and regression. It handles higher dimensionality data and missing values well and maintains accuracy, but doesnt give precise values for the regression model as the final prediction is based on the mean predictions from subset trees.
- 16. Gradient boosted trees Boosting Boosting refers to a family of learners which convert weak learners to strong learners. It learns sequentially from the errors from a prior random sample(in our case, a tree). The weak learners are trained sequentially each trying to correct its predecessor. The early learners fit simple models to the data and then analyze the data for errors. All the weak learners with their higher accuracy of error (only slighty less than guessing,0.5) are combined in some way to get a strong classifier,with a higher accuracy. When an input is misclassified by a hypothesis, its weight is increased so that next hypothesis is more likely to classify it correctly. By combining the whole set at the end, the weak learners are converted into better performing model.
- 17. Gradient boosted trees Types of boosting AdaBoost: short for Adaptive boosting. Start from a weak classifier and learn to linearly combine them so that the error is reduced. The result is strong classifier built by boosting of weak classifiers. We train an algorithm, say Decision tree on a model, whose all features have been given equal weights. A model is built on a subset of data and predictions are made on the whole dataset,and errors are calculated by the predictions and actual values.
- 18. Gradient boosted trees Adaboost While creating the next model, higher weights are given to the data points which were predicted incorrectly ie. misclassified. Weights can be determined using the error value, ie. Higher the error, more is the weight associated to the observation. This process is repeated until the error function does not change, or the maximum limit of the estimators is reached. Its used for both classfication and regression problem,mostly decision stamps are used with Adaboost, but any machine learning algorithm, if it accepts weight on training data set can be used a base learner. One of the applications of Adaboost is face recognition systems.
- 19. Gradient boosted trees Types of Boosting Gradient Boosting We will cover this in detail now. There are other implementations of Gradient boosting like XGBoost and Light GB.
- 20. Gradient boosted trees Gradient Boost It’s also a machine learning technique which produces which produces a prediction model in the form of an ensemble of weak prediction models, typically decision trees. Thus, they may be referred as Gradient boosted trees. Like other boosting methods, it builds a model in a sequential or stage-wise fashion.
- 21. Gradient boosted trees We shall now see some maths behind it. The objective of any supervised learning algorithm is to define a loss function and minimize it. We have mean square error defined as: We want our loss function(MSE) in our predictions be minimum using gradient descent and updating our predictions based on a learning rate.
- 22. Gradient boosted trees We will see what is learning rate. Learning rates are the hypermeters which controls how much we are adjusting the weights of our network with respect to the loss gradient. The learning rate affects how quickly our model can converge to a local minima (aka. arrive at the best accuracy). The relationship is given by the formula: new_weight = existing_weight — learning_rate * gradient In gradient boosted trees, we use the following learning rate: We basically update the predictions such that the sum of our residuals is close to zero(or minimum) and the predicted values are sufficiently close to the actual values. Learning rates are so tuned so as to prevent the overfitting which the gradient boosted trees are prone to.
- 23. Gradient boosted trees In Gradient boosted trees, models are sequentially trained, and each model minimizes the loss function (y = ax + b + e, e needs special attention as it is an error term) of the whole system using Gradient descent method, as explained earlier. The learning procedure consecutively fits new models to provide a more accurate estimate of response variable. The principle idea behind this algorithm is to create new base learners, which can be maximally corelated with negative gradient of the loss function, associated with the whole ensemble. Pros of Gradient boosted trees: Fast, easy to tune, not sensitive to scale (features can be a mix of continuous and categorical data), good performance, lots of software available(well supported and tested) Cons: Sensitive to overfitting and noise (should always cross validate)
- 24. Thanks!