Decision trees & random forests
2 likes1,502 views
The document discusses decision trees and random forests machine learning algorithms. It explains that decision trees are simple algorithms that can be used for classification and regression problems. Random forests improve on decision trees by constructing multiple trees on randomly sampled data and averaging their predictions, which helps prevent overfitting issues. The document provides examples of how decision trees are constructed and how random forests make predictions through voting or averaging tree predictions.
Decision trees & random forests
- 1. DECISION TREES & RANDOM FORESTS Max Pagels, Data Science Specialist max.pagels@sc5.io 12.6.2016
- 2. A TREE IN THE REAL WORLD
- 3. A TREE IN COMPUTER SCIENCE
- 4. A TREE IN COMPUTER SCIENCE Root Leaf Edge Node
- 5. DECISION TREES A decision tree is a learning algorithm that constructs a set of decisions based on training data. Decision trees are popular because: • They are naturally non-linear, so you can use them to solve complex problems • They are easy to visualise • How they work is easily explained • They can be used for regression (predict a number) and classification (predict a class) A decision tree algorithm is an explicit version of “ten questions”.
- 6. A TOY EXAMPLE Weekend? Evening? Food? Date? Yes Yes Yes Yes No Yes No No No No Yes No Yes No Yes Yes Yes No No No Yes Yes No No
- 7. BUILDING A DECISION TREE Basic approach (starting with a root node): Loop over all leaf nodes: 1. Select the best attribute A 2. Assign A as the decision attribute for the node we are currently traversing 3. For each value of A, create a descendant node (leaf node) 4. Sort training examples to leaf nodes 5. If stopping criterion hit, stop; else continue Weekend? Evening? Food? Date? Yes Yes Yes Yes No Yes No No No No Yes No Yes No Yes Yes Yes No No No Yes Yes No No
- 8. BUILDING A DECISION TREE Basic approach (starting with a root node): Loop over all leaf nodes: 1. Select the best attribute A 2. Assign A as the decision attribute for the node we are currently traversing 3. For each value of A, create a descendant node (leaf node) 4. Sort training examples to leaf nodes 5. If stopping criterion hit, stop; else continue Weekend? Evening? Food? Date? Yes Yes Yes Yes No Yes No No No No Yes No Yes No Yes Yes Yes No No No Yes Yes No No
- 9. BUILDING A DECISION TREE Basic approach (starting with a root node): Loop over all leaf nodes: 1. Select the best attribute A 2. Assign A as the decision attribute for the node we are currently traversing 3. For each value of A, create a descendant node (leaf node) 4. Sort training examples to leaf nodes 5. If stopping criterion hit, stop; else continue Weekend? Evening? Food? Date? Yes Yes Yes Yes No Yes No No No No Yes No Yes No Yes Yes Yes No No No Yes Yes No No
- 10. DECISION TREES IN SCIKIT sklearn.tree.DecisionTreeClassifier( criterion=‘gini', splitter=‘best', max_depth=None, min_samples_split=2, min_samples_leaf=1, min_weight_fraction_leaf=0.0, max_features=None, random_state=None, max_leaf_nodes=None, min_impurity_split=1e-07, class_weight=None, presort=False )
- 11. BUILDING A DECISION TREE Weekend? Evening? Food? Date? Yes Yes Yes Yes No Yes No No No No Yes No Yes No Yes Yes Yes No No No Yes Yes No No
- 12. BUILDING A DECISION TREE Weekend? Evening? Food? Date? Yes Yes Yes Yes No Yes No No No No Yes No Yes No Yes Yes Yes No No No Yes Yes No No W
- 13. BUILDING A DECISION TREE Weekend? Evening? Food? Date? Yes Yes Yes Yes No Yes No No No No Yes No Yes No Yes Yes Yes No No No Yes Yes No No W Yes No
- 14. BUILDING A DECISION TREE Weekend? Evening? Food? Date? Yes Yes Yes Yes No Yes No No No No Yes No Yes No Yes Yes Yes No No No Yes Yes No No W F Yes No
- 15. BUILDING A DECISION TREE Weekend? Evening? Food? Date? Yes Yes Yes Yes No Yes No No No No Yes No Yes No Yes Yes Yes No No No Yes Yes No No W F Yes No Yes No
- 16. BUILDING A DECISION TREE Weekend? Evening? Food? Date? Yes Yes Yes Yes No Yes No No No No Yes No Yes No Yes Yes Yes No No No Yes Yes No No W F Yes No Yes No Date No Date
- 17. BUILDING A DECISION TREE Weekend? Evening? Food? Date? Yes Yes Yes Yes No Yes No No No No Yes No Yes No Yes Yes Yes No No No Yes Yes No No W EF Yes No Yes No Date No Date
- 18. BUILDING A DECISION TREE Weekend? Evening? Food? Date? Yes Yes Yes Yes No Yes No No No No Yes No Yes No Yes Yes Yes No No No Yes Yes No No W EF Yes No Yes No Date No Date Yes No
- 19. BUILDING A DECISION TREE Weekend? Evening? Food? Date? Yes Yes Yes Yes No Yes No No No No Yes No Yes No Yes Yes Yes No No No Yes Yes No No W EF Yes No Yes No Date No Date Yes No Date No Date
- 20. ID3 Weekend? Evening? Food? Date? Yes Yes Yes Yes No Yes No No No No Yes No Yes No Yes Yes Yes No No No Yes Yes No No W EF Yes No Yes No Date No Date Yes No Date No Date
- 21. The ID3 algorithm, and many other decision tree algorithms, are prone to overfitting: trees become too deep and start to capture noise in the training data. Overfitting means a trained algorithm will fail to generalise well to new examples. One way of combatting overfitting is to use an ensemble method. ID3
- 22. RANDOM FORESTS
- 23. A FOREST IN THE REAL WORLD
- 24. A FOREST IN COMPUTER SCIENCE
- 25. A FOREST IN COMPUTER SCIENCE
- 26. RANDOM FORESTS A random forest is an ensemble method based on decision trees. The basic idea is deceptively simple: 1. Construct N decision trees • Randomly sample a subset of the training data (with replacement) • Construct/train a decision tree using the decision tree algorithm and the sampled subset of data 2. Predict by asking all trees in the forest for their opinion • For regression problems, take the mean (average) of all trees’ predictions • For classification problems, take the mode of all trees’ predictions (i.e. vote)
- 28. REGRESSION 2.1 1.8 1.9 µ({2.1, 1.8, 1.9}) = 1.933…
- 29. SUMMARY Decision trees are easy-to-understand learning algorithms that can be used for regression and classification, even for non-linear problems. Random forests are ensemble learning algorithms that help prevent overfitting by creating many decision trees and averaging their predictions. If you are just getting started with machine learning, decision trees are an excellent starting point.