Fast Perceptron Decision Tree Learning from Evolving Data Streams
0 likes1,580 views
The document proposes using perceptron learners at the leaves of Hoeffding decision trees to improve performance on data streams. It introduces a new evaluation metric called RAM-Hours that considers both time and memory usage. The authors empirically evaluate different classifier models, including Hoeffding trees with perceptron and naive Bayes learners at leaves, on several datasets. Results show that hybrid models like Hoeffding naive Bayes perceptron trees often provide the best balance of accuracy, time and memory usage.
![Concept Drift Framework
t
f(t) f(t)
α
α
t0
W
0.5
1
Definition
Given two data streams a, b, we define c = a⊕W
t0
b as the data
stream built joining the two data streams a and b
Pr[c(t) = b(t)] = 1/(1+e−4(t−t0)/W ).
Pr[c(t) = a(t)] = 1−Pr[c(t) = b(t)]
20 / 28](https://image.slidesharecdn.com/pakdd10-101101164426-phpapp01/85/Fast-Perceptron-Decision-Tree-Learning-from-Evolving-Data-Streams-25-320.jpg)
Fast Perceptron Decision Tree Learning from Evolving Data Streams
- 1. Fast Perceptron Decision Tree Learning from Evolving Data Streams Albert Bifet, Geoff Holmes, Bernhard Pfahringer, and Eibe Frank University of Waikato Hamilton, New Zealand Hyderabad, 23 June 2010 14th Pacific-Asia Conference on Knowledge Discovery and Data Mining (PAKDD’10)
- 2. Motivation RAM Hours Time and Memory in one measure Hoeffding Decision Trees with Perceptron Learners at leaves Improve performance of classification methods for data streams 2 / 28
- 3. Outline 1 RAM-Hours 2 Perceptron Decision Tree Learning 3 Empirical evaluation 3 / 28
- 4. Mining Massive Data 2007 Digital Universe: 281 exabytes (billion gigabytes) The amount of information created exceeded available storage for the first time Web 2.0 106 million registered users 600 million search queries per day 3 billion requests a day via its API. 4 / 28
- 5. Green Computing Green Computing Study and practice of using computing resources efficiently. Algorithmic Efficiency A main approach of Green Computing Data Streams Fast methods without storing all dataset in memory 5 / 28
- 6. Data stream classification cycle 1 Process an example at a time, and inspect it only once (at most) 2 Use a limited amount of memory 3 Work in a limited amount of time 4 Be ready to predict at any point 6 / 28
- 7. Mining Massive Data Koichi Kawana Simplicity means the achievement of maximum effect with minimum means. time accuracy memory Data Streams 7 / 28
- 8. Evaluation Example Accuracy Time Memory Classifier A 70% 100 20 Classifier B 80% 20 40 Which classifier is performing better? 8 / 28
- 9. RAM-Hours RAM-Hour Every GB of RAM deployed for 1 hour Cloud Computing Rental Cost Options 9 / 28
- 10. Evaluation Example Accuracy Time Memory RAM-Hours Classifier A 70% 100 20 2,000 Classifier B 80% 20 40 800 Which classifier is performing better? 10 / 28
- 11. Outline 1 RAM-Hours 2 Perceptron Decision Tree Learning 3 Empirical evaluation 11 / 28
- 12. Hoeffding Trees Hoeffding Tree : VFDT Pedro Domingos and Geoff Hulten. Mining high-speed data streams. 2000 With high probability, constructs an identical model that a traditional (greedy) method would learn With theoretical guarantees on the error rate Time Contains “Money” YES Yes NO No Day YES Night 12 / 28
- 13. Hoeffding Naive Bayes Tree Hoeffding Tree Majority Class learner at leaves Hoeffding Naive Bayes Tree G. Holmes, R. Kirkby, and B. Pfahringer. Stress-testing Hoeffding trees, 2005. monitors accuracy of a Majority Class learner monitors accuracy of a Naive Bayes learner predicts using the most accurate method 13 / 28
- 14. Perceptron Attribute 1 Attribute 2 Attribute 3 Attribute 4 Attribute 5 Output hw (xi) w1 w2 w3 w4 w5 Data stream: xi,yi Classical perceptron: hw (xi) = sgn(wT xi), Minimize Mean-square error: J(w) = 1 2 ∑(yi −hw (xi))2 14 / 28
- 15. Perceptron Attribute 1 Attribute 2 Attribute 3 Attribute 4 Attribute 5 Output hw (xi) w1 w2 w3 w4 w5 We use sigmoid function hw = σ(wT x) where σ(x) = 1/(1+e−x ) σ (x) = σ(x)(1−σ(x)) 14 / 28
- 16. Perceptron Minimize Mean-square error: J(w) = 1 2 ∑(yi −hw (xi))2 Stochastic Gradient Descent: w = w +η∇Jxi Gradient of the error function: ∇J = −∑ i (yi −hw (xi))∇hw (xi) ∇hw (xi) = hw (xi)(1−hw (xi)) Weight update rule w = w +η ∑ i (yi −hw (xi))hw (xi)(1−hw (xi))xi 14 / 28
- 17. Perceptron PERCEPTRON LEARNING(Stream,η) 1 for each class 2 do PERCEPTRON LEARNING(Stream,class,η) PERCEPTRON LEARNING(Stream,class,η) 1 £ Let w0 and w be randomly initialized 2 for each example (x,y) in Stream 3 do if class = y 4 then δ = (1−hw (x))·hw (x)·(1−hw (x)) 5 else δ = (0−hw (x))·hw (x)·(1−hw (x)) 6 w = w +η ·δ ·x PERCEPTRON PREDICTION(x) 1 return argmaxclass hwclass (x) 15 / 28
- 18. Hybrid Hoeffding Trees Hoeffding Naive Bayes Tree Two learners at leaves: Naive Bayes and Majority Class Hoeffding Perceptron Tree Two learners at leaves: Perceptron and Majority Class Hoeffding Naive Bayes Perceptron Tree Three learners at leaves: Naive Bayes, Perceptron and Majority Class 16 / 28
- 19. Outline 1 RAM-Hours 2 Perceptron Decision Tree Learning 3 Empirical evaluation 17 / 28
- 20. What is MOA? {M}assive {O}nline {A}nalysis is a framework for online learning from data streams. It is closely related to WEKA It includes a collection of offline and online methods as well as tools for evaluation: boosting and bagging Hoeffding Trees with and without Na¨ıve Bayes classifiers at the leaves. 18 / 28
- 21. What is MOA? Easy to extend Easy to design and run experiments Philipp Kranen, Hardy Kremer, Timm Jansen, Thomas Seidl, Albert Bifet, Geoff Holmes, Bernhard Pfahringer RWTH Aachen University, University of Waikato Benchmarking Stream Clustering Algorithms within the MOA Framework KDD 2010 Demo 18 / 28
- 22. MOA: the bird The Moa (another native NZ bird) is not only flightless, like the Weka, but also extinct. 19 / 28
- 23. MOA: the bird The Moa (another native NZ bird) is not only flightless, like the Weka, but also extinct. 19 / 28
- 24. MOA: the bird The Moa (another native NZ bird) is not only flightless, like the Weka, but also extinct. 19 / 28
- 25. Concept Drift Framework t f(t) f(t) α α t0 W 0.5 1 Definition Given two data streams a, b, we define c = a⊕W t0 b as the data stream built joining the two data streams a and b Pr[c(t) = b(t)] = 1/(1+e−4(t−t0)/W ). Pr[c(t) = a(t)] = 1−Pr[c(t) = b(t)] 20 / 28
- 26. Concept Drift Framework t f(t) f(t) α α t0 W 0.5 1 Example (((a⊕W0 t0 b)⊕W1 t1 c)⊕W2 t2 d)... (((SEA9 ⊕W t0 SEA8)⊕W 2t0 SEA7)⊕W 3t0 SEA9.5) CovPokElec = (CoverType⊕5,000 581,012 Poker)⊕5,000 1,000,000 ELEC2 20 / 28
- 27. Empirical evaluation Accuracy 40 45 50 55 60 65 70 75 80 10.000 120.000 230.000 340.000 450.000 560.000 670.000 780.000 890.000 1.000.0 Instances Accuracy(%) htnbp htnb htp ht Figure: Accuracy on dataset LED with three concept drifts. 21 / 28
- 28. Empirical evaluation RunTime 0 5 10 15 20 25 30 35 10.000 120.000 230.000 340.000 450.000 560.000 670.000 780.000 890.000 Instances Time(sec.) htnbp htnb htp ht Figure: Time on dataset LED with three concept drifts. 22 / 28
- 29. Empirical evaluation Memory 0 0,5 1 1,5 2 2,5 3 3,5 4 4,5 5 10.000 130.000 250.000 370.000 490.000 610.000 730.000 850.000 970.000 Instances Memory(Mb) htnbp htnb htp ht Figure: Memory on dataset LED with three concept drifts. 23 / 28
- 30. Empirical evaluation RAM-Hours 0,00E+00 5,00E-06 1,00E-05 1,50E-05 2,00E-05 2,50E-05 3,00E-05 3,50E-05 4,00E-05 4,50E-05 10.000 130.000 250.000 370.000 490.000 610.000 730.000 850.000 970.000 Instances RAM-Hours htnbp htnb htp ht Figure: RAM-Hours on dataset LED with three concept drifts. 24 / 28
- 31. Empirical evaluation Cover Type Dataset Accuracy Time Mem RAM-Hours Perceptron 81.68 12.21 0.05 1.00 Na¨ıve Bayes 60.52 22.81 0.08 2.99 Hoeffding Tree 68.3 13.43 2.59 56.98 Trees Na¨ıve Bayes HT 81.06 24.73 2.59 104.92 Perceptron HT 83.59 16.53 3.46 93.68 NB Perceptron HT 85.77 22.16 3.46 125.59 Bagging Na¨ıve Bayes HT 85.73 165.75 0.8 217.20 Perceptron HT 86.33 50.06 1.66 136.12 NB Perceptron HT 87.88 115.58 1.25 236.65 25 / 28
- 32. Empirical evaluation Electricity Dataset Accuracy Time Mem RAM-Hours Perceptron 79.07 0.53 0.01 1.00 Na¨ıve Bayes 73.36 0.55 0.01 1.04 Hoeffding Tree 75.35 0.86 0.12 19.47 Trees Na¨ıve Bayes HT 80.69 0.96 0.12 21.74 Perceptron HT 84.24 0.93 0.21 36.85 NB Perceptron HT 84.34 1.07 0.21 42.40 Bagging Na¨ıve Bayes HT 84.36 3.17 0.13 77.75 Perceptron HT 85.22 2.59 0.44 215.02 NB Perceptron HT 86.44 3.55 0.3 200.94 26 / 28
- 33. Summary http://moa.cs.waikato.ac.nz/ Summary Sensor Networks use Perceptron Handheld Computers use Hoeffding Naive Bayes Perceptron Tree Servers use Bagging Hoeffding Naive Bayes Perceptron Tree 27 / 28
- 34. Summary http://moa.cs.waikato.ac.nz/ Conclusions RAM-Hours as a new measure of time and memory Hoeffding Perceptron Tree Hoeffding Naive Bayes Perceptron Tree Future Work Adaptive learning rate for the Perceptron. 28 / 28