1.5.ensemble learning with apache spark m llib 1.5
2 likes357 views
This document discusses ensemble learning techniques in Apache Spark MLlib 1.5. It begins by defining ensemble learning as combining multiple learning modules to increase model stability and predictive power. It then provides examples of sources of variation between models, such as different data samples, assumptions, modeling techniques, and initialization parameters. Next, it demonstrates mathematically how combining three 70% accurate binary classifiers through majority voting can achieve over 78% accuracy. The remainder of the document explains techniques like bagging, boosting, and stacking to reduce bias and variance errors by leveraging diverse models. It provides an example using stacking with Spark MLlib on a dataset to improve over a random forest baseline precision.
![Reference
[1] http://www.csdn.net/article/2015-03-02/2824069
[2] http://www.csdn.net/article/2015-09-07/2825629
[3] http://www.scholarpedia.org/article/Ensemble_learning](https://image.slidesharecdn.com/1-160127064705/85/1-5-ensemble-learning-with-apache-spark-m-llib-1-5-2-320.jpg)
1.5.ensemble learning with apache spark m llib 1.5
- 1. Ensemble Learning with Apache Spark MLlib 1.5 leoricklin@gmail.com
- 2. Reference [1] http://www.csdn.net/article/2015-03-02/2824069 [2] http://www.csdn.net/article/2015-09-07/2825629 [3] http://www.scholarpedia.org/article/Ensemble_learning
- 3. What is Ensemble Learning (集成学习) ? ● 结合不同的学习模块（单个模型）来加强模型的稳定性和预 测能力 ● 导致模型不同的4个主要因素。这些因素的组合也可能会造 成模型不同： ● 集成学习是典型的实践驱动的研究方向，它一开始先在实践 中证明有效，而后才有学者从理论上进行各种分析 ● 不同种类 ● 不同假设 ● 不同建模技术 ● 初始化参数不同
- 4. A pinch of math ● There are 3 (independent) binary classifiers (A,B,C) with a 70% accuracy ● For a majority vote with 3 members we can expect 4 outcomes: ● All three are correct 0.7 * 0.7 * 0.7 = 0.3429 ● Two are correct 0.7 * 0.7 * 0.3 + 0.7 * 0.3 * 0.7 + 0.3 * 0.7 * 0.7 = 0.4409 ● Two are wrong 0.3 * 0.3 * 0.7 + 0.3 * 0.7 * 0.3 + 0.7 * 0.3 * 0.3 = 0.189 ● All three are wrong 0.3 * 0.3 * 0.3 = 0.027 0.3429 + 0.4409 = 0.7838 > 0.7
- 5. Model Error ● 任何模型中出现的误差都可以在 数学上分解成三个分量： ○ Bias error 是用来度量预测值与实 际值的差异 ○ Variance 则是度量基于同一观测 值，预测值之间的差异
- 6. Trade-off management of bias-variance errors ● 通当模型复杂性增加时，最 终会过拟合，因此模型开始 出现Variance ● 优良的模型应该在这两种 误差之间保持平衡 ● 集成学习就是执行折衷权 衡的一种方法 ○ 怎么训练每个算法？ ○ 怎么融合每个算法？
- 7. EL techniques (1): Bagging ● 试图在小样本集上实现相 似的学习模块，然后对预 测值求平均值 ● 可以帮助减少Variance
- 8. EL techniques (2): Boosting ● 是一项迭代技术 ● 它在上一次分类的基础上 调整观测值的权重。如果 观测值被错误分类，它就 会增加这个观测值的权重 ● 会减少Bias error，但是有 些时候会在训练数据上过 拟合
- 9. EL techniques (3): Stacking ● 用一个学习模块与来自 不同学习模块的输出结 合起来 ● 可以减少Bias error和 Variance ● 选择合适的集成模块与 其说是纯粹的科研问题， 不如说是一种艺术
- 12. Stacking with Apache MLLib (1) ● Dataset：UCI Covtype (Ch04, Adv. Analytic w/ Spark) ● Baseline: RandomForest (Best from 8 hyper-parameters with 3-folds C.V.) ○ precision = 0.956144 ○ recall = 0.956144 Training set X RF(θ1 ) fits Training set Y predicts h1 (Y,θ1 ) #trees = 32 θ1 : #bins=300, #depth=30, entropy
- 13. ● Using Meta-features Stacking with Apache MLLib (2) Training set X RF(θ1 ) RF(θ2 ) RF(θ3 ) RF(θ1 ) h1 (Y,θ1 ) #trees = 32 θ1 : #bins=300, #depth=30, entropy θ2 : #bins=40, #depth=30, entropy θ3 : #bins=300, #depth=30, gini fits predicts 3-folds C.V of Training set X h1 (X,θ1 ) h2 (X,θ2 ) h3 (X,θ3 ) Label fits Training set Y predicts RF(θ1 ) RF(θ2 ) RF(θ3 ) h1 (Y,θ1 ) h2 (Y,θ2 ) h3 (Y,θ3 ) predicts sort by precision Baseline Current precision 0.956144 0.951056 recall 0.956144 0.951056
- 14. Stacking with Apache MLLib (3) ● Using Original features & Meta-features Training set X RF(θ1 ) RF(θ2 ) RF(θ3 ) RF(θ1 ) h1 (Y,θ1 ) #trees = 32 θ1 : #bins=300, #depth=30, entropy θ2 : #bins=40, #depth=30, entropy θ3 : #bins=300, #depth=30, gini fits predicts 3-folds C.V of Training set X h1 (X,θ1 ) h2 (X,θ2 ) h3 (X,θ3 ) Label fits Training set Y predicts RF(θ1 ) RF(θ2 ) RF(θ3 ) h1 (Y,θ1 ) h2 (Y,θ2 ) h3 (Y,θ3 ) predicts sort by precision Baseline Current precision 0.956144 0.951094 recall 0.956144 0.951094 f1 fn ……….. f1 ...fn
- 15. Stacking with Apache MLLib (4) ● Retrain tier-1 models and stacking with all features Training set X RF(θ1 ) RF(θ2 ) RF(θ3 ) RF(θ1 ) h1 (Y,θ1 ) #trees = 32 θ1 : #bins=300, #depth=30, entropy θ2 : #bins=40, #depth=30, entropy θ3 : #bins=300, #depth=30, gini fits predicts Training set X h1 (X,θ1 ) h2 (X,θ2 ) h3 (X,θ3 ) Label fits Training set Y predicts h1 (Y,θ1 ) h2 (Y,θ2 ) h3 (Y,θ3 ) predicts sort by precision Baseline Current precision 0.956144 0.956836 recall 0.956144 0.956836 f1 fn ……….. f1 ...fn RF(θ1 ) RF(θ2 ) RF(θ3 )