Entropy and information gain in decision tree.
0 likes1,483 views
Entropy is a measure of unpredictability or impurity in a data set. It is used in decision trees to determine the best way to split data at each node. High entropy means low purity with an equal mix of classes, while low entropy means high purity with mostly one class. Information gain is the reduction in entropy when splitting on an attribute, with the attribute with the highest information gain chosen as the split. For example, in a data set on restaurant patrons, splitting on the "patrons" attribute results in a higher information gain than splitting on "type of food" so "patrons" would be chosen as the root node.
![Choosing the best attribute
E(Type=French) = 1
E(Type=Italian) = 1
E(Type=Thai) = 1
E(Type=Burger) =1
Weighted average of entropy for each node
E(Type)=
[2/12 * 1 + 2/12 * 1 + 4/12 * 1 + 4/12 *1]
= 1
E(Type) ≈ 1
11
6 positive
6 negative
2 positive
2 negative
French
2
Italian
2
Thai
4
1 negative
1 positive
1 positive
1 negative
Burger
4
2 positive
2 negative
Type
OMega TechEd](https://image.slidesharecdn.com/2-231203160515-80cb685b/85/Entropy-and-information-gain-in-decision-tree-11-320.jpg)
Entropy and information gain in decision tree.
- 1. Entropy and Information Gain in Decision Tree OMega TechEd
- 2. Entropy Entropy is the machine learning metric that measures the unpredictability or impurity in the system. Entropy is the measurement of disorder or impurities in the information processed in machine learning. It determines how a decision tree chooses to split data. 2 High Entropy Low Entropy OMega TechEd
- 3. Entropy A random variable with only one value, a coin that always comes up heads, has no uncertainty and thus its entropy is defined as zero; thus, we gain no information by observing its value. Entropy always lies between 0 and 1, however depending on the number of classes in the dataset, it can be greater than 1. In general, the entropy of a random variable V with values vk, each with probability P(vk), is defined as Entropy: H(V ) = − ∑ k P(vk) log2 P(vk) . Entropy of a fair coin flip H(Fair ) = −(0.5 log2 0.5+0.5 log2 0.5) = 1 . 3 OMega TechEd
- 4. How to calculate Entropy? H(V ) = − ∑ k P(vk) log2 P(vk) . Example: If we had a total 10 data points in our dataset with 3 belonging to positive class and 7 belonging to negative class: -3/10 * log2 (3/10) – 7/10 * log2 (7/10) ≈ 0.876 The Entropy is approximately 0.88 . High entropy means low level of purity. 4 OMega TechEd
- 5. Entropy (Cont.) Different cases 5 Entropy=0 Entropy=1 Entropy=0.88 If dataset contain equal no of positive and negative data points entropy is 1. If dataset contain only positive or only negative data points entropy is 0. OMega TechEd
- 6. Information Gain Information gain is defined as the pattern observed in the dataset and reduction in the entropy. Mathematically, information gain can be expressed with the below formula: Information Gain = (Entropy of parent node)-(Entropy of child node) 6 OMega TechEd
- 7. Decision tree using information gain 1. An attribute with the highest information gain from a set should be selected as the parent (root) node. 2. Build child nodes for every value of attribute A. 3. Repeat iteratively until we finish constructing the whole tree. 7 OMega TechEd
- 8. Choosing the best attribute We need a measure of “good” and “bad” for attributes. One way to do is to compute the information gain. Example: At the root node of the restaurant problem, there are 6 True samples and 6 False samples. Entropy(Parent) =1 8 6 positive 6 negative 2 negative 4 positive 2 positive 4 negative None 2 Some 4 Full 6 0 positive 0 negative Patrons OMega TechEd
- 9. Choosing the best attribute E(Patrons=None) = 0 E(Patrons=Some) = 0 E(Patrons=Full) = = -2/6 * log2 (2/6) – 4/6 * log2 (4/6) = -1/3 * (-1.59) - 2/3 * (-0.59) = 0.53+0.39 ≈ 0.92 Weighted average of entropy for each node E(Patrons)= 2/12 * 0 + 4/12 * 0 + 6/12 * 0.92 = 0.46 E(Patrons) ≈ 0.46 9 6 positive 6 negative 2 negative 4 positive 2 positive 4 negative None 2 Some 4 Full 6 0 positive 0 negative Patrons
- 10. Choosing the best attribute 10 Information Gain = (Entropy of parent node)-(Entropy of child node) IG= 1-0.46 ≈ 0.54 Gain(Patrons) ≈ 0.54 OMega TechEd
- 11. Choosing the best attribute E(Type=French) = 1 E(Type=Italian) = 1 E(Type=Thai) = 1 E(Type=Burger) =1 Weighted average of entropy for each node E(Type)= [2/12 * 1 + 2/12 * 1 + 4/12 * 1 + 4/12 *1] = 1 E(Type) ≈ 1 11 6 positive 6 negative 2 positive 2 negative French 2 Italian 2 Thai 4 1 negative 1 positive 1 positive 1 negative Burger 4 2 positive 2 negative Type OMega TechEd
- 12. Choosing the best attribute 12 Information Gain = (Entropy of parent node)-(Entropy of child node) IG= 1 - 1 ≈ 0 Gain(Type) ≈ 0 Confirming that Patrons is a better attribute than Type. In fact, at the root Patrons gives the highest information gain. OMega TechEd
- 13. Thank you Reference: Artificial Intelligence: A Modern Approach, 3rd ed. Stuart Russell and Peter Norvig OMega TechEd