Decision trees
- 2. Overview What is a Decision Tree Sample Decision Trees How to Construct a Decision Tree Decision Tree Algorithm ID3 Heuristic Entropy Decision Tree Advantages And Limitations Summary
- 3. What is a Decision Tree? An inductive learning task Use particular facts to make more generalized conclusions A predictive model based on a branching series of Boolean tests These smaller Boolean tests are less complex than a one-stage classifier Let’s look at a sample decision tree…
- 4. Predictive Time Commuting Leave At Stall? Accident? 10 AM 9 AM 8 AM Long Long Short Medium Long No Yes No Yes If we leave at 10 AM and there are no cars stalled on the road, what will our commute time be?
- 5. Inductive Learning In this decision tree, we made a series of Boolean decisions and followed the corresponding branch Did we leave at 10 AM? Did a car stall on the road? Is there an accident on the road? By answering each of these yes/no questions, we then came to a conclusion on how long our commute might take
- 6. Decision Trees as Rules We did not have represent this tree graphically We could have represented as a set of rules. However, this may be much harder to read…
- 7. Decision Tree as a Rule Set if hour == 8am commute time = long else if hour == 9am if accident == yes commute time = long else commute time = medium else if hour == 10am if stall == yes commute time = long else commute time = short Notice that all attributes to not have to be used in each path of the decision. As we will see, all attributes may not even appear in the tree.
- 8. How to Create a Decision Tree We first make a list of attributes that we can measure These attributes (for now) must be discrete We then choose a target attribute that we want to predict Then create an experience table that lists what we have seen in the past
- 9. Sample Experience Table Example Attributes Target Hour Weather Accident Stall Commute D1 8 AM Sunny No No Long D2 8 AM Cloudy No Yes Long D3 10 AM Sunny No No Short D4 9 AM Rainy Yes No Long D5 9 AM Sunny Yes Yes Long D6 10 AM Sunny No No Short D7 10 AM Cloudy No No Short D8 9 AM Rainy No No Medium D9 9 AM Sunny Yes No Long D10 10 AM Cloudy Yes Yes Long D11 10 AM Rainy No No Short D12 8 AM Cloudy Yes No Long D13 9 AM Sunny No No Medium
- 10. Decision Tree Algorithms The basic idea behind any decision tree algorithm is as follows: Choose the best attribute(s) to split the remaining instances and make that attribute a decision node Repeat this process for recursively for each child Stop when: ○ All the instances have the same target attribute value ○ There are no more attributes ○ There are no more instances
- 11. Identifying the Best Attributes Refer back to our original decision tree Leave At Stall? Accident? 10 AM 9 AM 8 AM Long Long Short Medium No Yes No Yes Long How did we know to split on leave at and then on stall and accident and not weather?
- 12. ID3 Heuristic To determine the best attribute, we look at the ID3 heuristic ID3 splits attributes based on their entropy. Entropy is the measure of disinformation…
- 13. Entropy Entropy is minimized when all values of the target attribute are the same. If we know that commute time will always be short, then entropy = 0 Entropy is maximized when there is an equal chance of all values for the target attribute (i.e. the result is random) If commute time = short in 3 instances, medium in 3 instances and long in 3 instances, entropy is maximized
- 14. Entropy Calculation of entropy Entropy(S) = ∑(i=1 to l)-|Si|/|S| * log2(|Si|/|S|) ○ S = set of examples ○ Si = subset of S with value vi under the target attribute ○ l = size of the range of the target attribute
- 15. ID3 ID3 splits on attributes with the lowest entropy We calculate the entropy for all values of an attribute as the weighted sum of subset entropies as follows: ∑(i = 1 to k) |Si|/|S| Entropy(Si), where k is the range of the attribute we are testing We can also measure information gain (which is inversely proportional to entropy) as follows: Entropy(S) - ∑(i = 1 to k) |Si|/|S| Entropy(Si)
- 16. ID3 Given our commute time sample set, we can calculate the entropy of each attribute at the root node Attribute Expected Entropy Information Gain Hour 0.6511 0.768449 Weather 1.28884 0.130719 Accident 0.92307 0.496479 Stall 1.17071 0.248842
- 17. Decision Tree Advantages • Inexpensive to construct • Extremely fast at classifying unknown records • Easy to interpret for small-sized trees • Accuracy is comparable to other classification techniques for many simple data sets
- 18. Decision Tree Limitations No backtracking local optimal solution not global optimal solution lookahead features may give us better trees Rectangular-shaped geometric regions in two-dimensional space ○ regions bounded by lines parallel to the x- and y- axes some linear relationships not parallel to the axes
- 19. Summary Decision trees can be used to help predict the future The trees are easy to understand Decision trees work more efficiently with discrete attributes The trees may suffer from error propagation