Svm soft margin hyperplanes
1 like188 views
This document provides an overview of soft margin hyperplanes in support vector machines (SVMs). It discusses how soft margins allow for misclassified samples by introducing slack variables ξi that measure the degree of misclassification. The objective is to minimize the model complexity while maximizing the margin and allowing for some misclassification. This is achieved by minimizing ||w||2 + CΣξi, where C controls the trade-off between margin maximization and error minimization. Lagrange multipliers are used to solve the optimization problem to obtain the dual formulation and derive the soft margin SVM classifier. Hyperparameter tuning of C is also discussed to find an optimal value that gives good generalization performance.
![Optimization using Lagrange multipliers
Expression: 𝑀𝑖𝑛𝑖𝑚𝑖𝑧𝑒 =
1
2
||𝑤||
2
Constraint: 𝑦𝑖(𝑤. 𝑥𝑖 + b) −1 = 0
Primal problem:
L=
1
2
||𝑤||
2
- 𝛼𝑖 [𝑦𝑖(𝑤. 𝑥𝑖 + b) − 1]
𝜕𝐿
𝜕𝑤
= 𝑤- 𝛼𝑖 𝑦𝑖 𝑥𝑖 = 0
𝑤 = 𝛼𝑖 𝑦𝑖 𝑥𝑖 6
𝜕𝐿
𝜕𝑏
= - 𝛼𝑖 𝑦𝑖 = 0
𝛼𝑖 𝑦𝑖 = 0 7
5
Department of Computer Science and Engineering, LBS College of Engineering, Kasaragod
5
𝑥+
𝑥−
𝑥+ + 𝑥−](https://image.slidesharecdn.com/svm-softmarginhyperplanes-200602140421/85/Svm-soft-margin-hyperplanes-5-320.jpg)
![Optimization using Lagrange multipliers
7
L=
1
2
||𝑤||
2
- 𝛼𝑖 [𝑦𝑖(𝑤. 𝑥𝑖 + b) − 1]
𝑤 = 𝛼𝑖 𝑦𝑖 𝑥𝑖
5
6
𝛼𝑖 𝑦𝑖 = 0
L=
1
2
( 𝛼𝑖 𝑦𝑖 𝑥𝑖 ).( 𝛼𝑗 𝑦𝑗 𝑥𝑗 )- 𝛼𝑖 [𝑦𝑖( 𝛼𝑗 𝑦𝑗 𝑥𝑗 . 𝑥𝑖 + b) − 1]
L=
1
2
( 𝛼𝑖 𝑦𝑖 𝑥𝑖 ).( 𝛼𝑗 𝑦𝑗 𝑥𝑗 )- ( 𝛼𝑖 𝑦𝑖 𝑥𝑖). ( 𝛼𝑗 𝑦𝑗 𝑥𝑗) − 𝛼𝑖 𝑦𝑖 b + 𝛼𝑖
= 0
L= 𝛼𝑖
+
1
2
( 𝛼𝑖 𝑦𝑖 𝑥𝑖 ).( 𝛼𝑗 𝑦𝑗 𝑥𝑗 )- ( 𝛼𝑖 𝑦𝑖 𝑥𝑖). ( 𝛼𝑗 𝑦𝑗 𝑥𝑗)
L= 𝛼𝑖 −
1
2
𝛼𝑖 𝑦𝑖 𝛼𝑗 𝑦𝑗 (𝑥𝑖. 𝑥𝑗)
Department of Computer Science and Engineering, LBS College of Engineering, Kasaragod
6](https://image.slidesharecdn.com/svm-softmarginhyperplanes-200602140421/85/Svm-soft-margin-hyperplanes-6-320.jpg)
![8
SVM Classifier Function: f( 𝑥) =(𝑤. 𝑥) - b
Department of Computer Science and Engineering, LBS College of Engineering, Kasaragod
= −
1
2
, −
1
2
. (𝑥1, 𝑥2) +
5
2
= −
1
2
𝑥1 −
1
2
𝑥2 +
5
2
= −
1
2
[𝑥1 + 𝑥2−5]
Equation of maximal margin line
f( 𝑥) = 0 𝑥1 + 𝑥2 = 5
0 1 2 3 4 5 6
1
2
3
4
5
6
Sampl
e
F1 F2 Class
1 2 1 +1
2 4 3 -1](https://image.slidesharecdn.com/svm-softmarginhyperplanes-200602140421/85/Svm-soft-margin-hyperplanes-8-320.jpg)
![Hyperparameter Tuning
Parameter Range Step Length
C [20, 211] 21
20
Parameter Cherkassky et. al. [11]
C max( 𝑦 + 3𝜎 , | 𝑦 − 3𝜎|)
Heuristic Methods
Search Space
Department of Computer Science and Engineering, LBS College of Engineering, Kasaragod](https://image.slidesharecdn.com/svm-softmarginhyperplanes-200602140421/85/Svm-soft-margin-hyperplanes-20-320.jpg)
Svm soft margin hyperplanes
- 1. SVM – Soft margin hyperplanes Sarith Divakar M LBS College of Engineering, Kasaragod sarith@lbscek.ac.in
- 2. RECAP OF SESSION 1 2 Department of Computer Science and Engineering, LBS College of Engineering, Kasaragod
- 3. Width of the street 𝑊𝑖𝑑𝑡ℎ = (𝑥+ - 𝑥− ) . 𝑤 ||𝑤|| For +ve samples 𝑦𝑖=1 and –ve samples 𝑦𝑖 = -1 𝑦𝑖(𝑤. 𝑥𝑖 + b) −1 = 0 3 1.(𝑤. 𝑥𝑖 + b) −1 = 0 𝑤. 𝑥𝑖 = 1-b i -1.(𝑤. 𝑥𝑖 + b) −1 = 0 𝑤. 𝑥𝑖 = -1-b ii 𝑊𝑖𝑑𝑡ℎ = (1-b – (-1-b)) . 1 ||𝑤|| 𝑥+ 𝑥− 𝑥+ - 𝑥− 𝑊𝑖𝑑𝑡ℎ = 2 ||𝑤|| Department of Computer Science and Engineering, LBS College of Engineering, Kasaragod 3
- 4. Maximize Width of the street 𝑀𝑎𝑥𝑖𝑚𝑖𝑧𝑒 = 2 ||𝑤|| 𝑥+ 𝑥− 𝑥+ + 𝑥− 𝑀𝑎𝑥𝑖𝑚𝑖𝑧𝑒 = 1 ||𝑤|| 𝑀𝑖𝑛𝑖𝑚𝑖𝑧𝑒 = ||𝑤|| 𝑀𝑖𝑛𝑖𝑚𝑖𝑧𝑒 = 1 2 ||𝑤|| 2 4 𝑦𝑖(𝑤. 𝑥𝑖 + b) −1 = 0 Department of Computer Science and Engineering, LBS College of Engineering, Kasaragod 4
- 5. Optimization using Lagrange multipliers Expression: 𝑀𝑖𝑛𝑖𝑚𝑖𝑧𝑒 = 1 2 ||𝑤|| 2 Constraint: 𝑦𝑖(𝑤. 𝑥𝑖 + b) −1 = 0 Primal problem: L= 1 2 ||𝑤|| 2 - 𝛼𝑖 [𝑦𝑖(𝑤. 𝑥𝑖 + b) − 1] 𝜕𝐿 𝜕𝑤 = 𝑤- 𝛼𝑖 𝑦𝑖 𝑥𝑖 = 0 𝑤 = 𝛼𝑖 𝑦𝑖 𝑥𝑖 6 𝜕𝐿 𝜕𝑏 = - 𝛼𝑖 𝑦𝑖 = 0 𝛼𝑖 𝑦𝑖 = 0 7 5 Department of Computer Science and Engineering, LBS College of Engineering, Kasaragod 5 𝑥+ 𝑥− 𝑥+ + 𝑥−
- 6. Optimization using Lagrange multipliers 7 L= 1 2 ||𝑤|| 2 - 𝛼𝑖 [𝑦𝑖(𝑤. 𝑥𝑖 + b) − 1] 𝑤 = 𝛼𝑖 𝑦𝑖 𝑥𝑖 5 6 𝛼𝑖 𝑦𝑖 = 0 L= 1 2 ( 𝛼𝑖 𝑦𝑖 𝑥𝑖 ).( 𝛼𝑗 𝑦𝑗 𝑥𝑗 )- 𝛼𝑖 [𝑦𝑖( 𝛼𝑗 𝑦𝑗 𝑥𝑗 . 𝑥𝑖 + b) − 1] L= 1 2 ( 𝛼𝑖 𝑦𝑖 𝑥𝑖 ).( 𝛼𝑗 𝑦𝑗 𝑥𝑗 )- ( 𝛼𝑖 𝑦𝑖 𝑥𝑖). ( 𝛼𝑗 𝑦𝑗 𝑥𝑗) − 𝛼𝑖 𝑦𝑖 b + 𝛼𝑖 = 0 L= 𝛼𝑖 + 1 2 ( 𝛼𝑖 𝑦𝑖 𝑥𝑖 ).( 𝛼𝑗 𝑦𝑗 𝑥𝑗 )- ( 𝛼𝑖 𝑦𝑖 𝑥𝑖). ( 𝛼𝑗 𝑦𝑗 𝑥𝑗) L= 𝛼𝑖 − 1 2 𝛼𝑖 𝑦𝑖 𝛼𝑗 𝑦𝑗 (𝑥𝑖. 𝑥𝑗) Department of Computer Science and Engineering, LBS College of Engineering, Kasaragod 6
- 7. SVM Classifier 𝜙 𝛼 = 𝛼 𝑖 − 1 2 𝛼𝑖 𝑦𝑖 𝛼𝑗 𝑦𝑗 (𝑥𝑖. 𝑥𝑗) Compute 𝑤 and b Dual Problem: Find vector 𝛼 which maximizes Subject to 𝛼𝑖 𝑦𝑖 = 0 SVM Classifier Function: 𝑏 = 1 2 (𝑚𝑖𝑛𝑖:𝑦 𝑖=+1(𝑤. 𝑥𝑖) + 𝑚𝑎𝑥𝑖:𝑦 𝑖=−1(𝑤. 𝑥𝑖)) 𝑤 = 𝛼𝑖 𝑦𝑖 𝑥𝑖 f( 𝑥) =(𝑤. 𝑥) - b Department of Computer Science and Engineering, LBS College of Engineering, Kasaragod 7 𝛼𝑖 ≥ 0
- 8. 8 SVM Classifier Function: f( 𝑥) =(𝑤. 𝑥) - b Department of Computer Science and Engineering, LBS College of Engineering, Kasaragod = − 1 2 , − 1 2 . (𝑥1, 𝑥2) + 5 2 = − 1 2 𝑥1 − 1 2 𝑥2 + 5 2 = − 1 2 [𝑥1 + 𝑥2−5] Equation of maximal margin line f( 𝑥) = 0 𝑥1 + 𝑥2 = 5 0 1 2 3 4 5 6 1 2 3 4 5 6 Sampl e F1 F2 Class 1 2 1 +1 2 4 3 -1
- 9. Soft margin hyperplanes Department of Computer Science and Engineering, LBS College of Engineering, Kasaragod 9
- 10. 10 Department of Computer Science and Engineering, LBS College of Engineering, Kasaragod
- 11. Soft margin hyperplanes 11 Outside the margin or on the margin Inside the margin correctly classified Inside the margin misclassified Soft Error 𝐶 𝜉𝑖 𝜉𝑖 = 0 0 < 𝜉𝑖 < 1 𝜉𝑖 > 1 Department of Computer Science and Engineering, LBS College of Engineering, Kasaragod 𝑤 𝑢 0 < 𝜉𝑖 < 1 𝜉𝑖 > 1 𝜉𝑖 = 0 𝜉𝑖 = 0
- 12. Optimization using Lagrange multipliers 12 Primal problem: Expression: 𝑀𝑖𝑛𝑖𝑚𝑖𝑧𝑒 = 1 2 ||𝑤|| 2 + C 𝜉𝑖 Constraint: 𝑦𝑖(𝑤. 𝑥𝑖 + b)≥ 1 − 𝜉𝑖 𝜉𝑖 ≥ 0 L= 1 2 ||𝑤|| 2 + C 𝜉𝑖 - 𝛼𝑖 𝑦𝑖(𝑤. 𝑥𝑖 + b) − 1 + 𝜉𝑖 − 𝜇𝑖 𝜉𝑖 𝛼𝑖 ≥ 0, 𝜇𝑖 ≥ 0 Department of Computer Science and Engineering, LBS College of Engineering, Kasaragod 𝑤 𝑢 0 < 𝜉𝑖 < 1 𝜉𝑖 > 1 𝜉𝑖 = 0 𝜉𝑖 = 0
- 13. 13 𝜕𝐿 𝜕𝑤 = 𝑤- 𝛼𝑖 𝑦𝑖 𝑥𝑖 = 0 𝑤 = 𝛼𝑖 𝑦𝑖 𝑥𝑖 𝜕𝐿 𝜕𝑏 = - 𝛼𝑖 𝑦𝑖 = 0 𝛼𝑖 𝑦𝑖 = 0 𝜕𝐿 𝜕𝜉 𝑖 = C - 𝛼𝑖 - 𝜇𝑖= 0 𝜇𝑖= C -𝛼𝑖 𝜇𝑖≥ 0 ⇒ 𝛼𝑖 ≤ 𝐶 L= 1 2 ||𝑤|| 2 + C 𝜉𝑖 - 𝛼𝑖 𝑦𝑖(𝑤. 𝑥𝑖 + b) − 1 + 𝜉𝑖 − 𝜇𝑖 𝜉𝑖 Optimization using Lagrange multipliers Department of Computer Science and Engineering, LBS College of Engineering, Kasaragod
- 14. Optimization using Lagrange multipliers L= 1 2 ( 𝛼𝑖 𝑦𝑖 𝑥𝑖 ).( 𝛼𝑗 𝑦𝑗 𝑥𝑗 ) + C 𝜉𝑖 − 𝛼𝑖 𝑦𝑖( 𝛼𝑗 𝑦𝑗 𝑥𝑗 . 𝑥𝑖 + b) − 1 + 𝜉𝑖 - 𝐶𝜉𝑖 + 𝛼𝑖 𝜉𝑖 L= 1 2 ( 𝛼𝑖 𝑦𝑖 𝑥𝑖 ).( 𝛼𝑗 𝑦𝑗 𝑥𝑗 ) + C 𝜉𝑖- ( 𝛼𝑖 𝑦𝑖 𝑥𝑖). ( 𝛼𝑗 𝑦𝑗 𝑥𝑗) − 𝛼𝑖 𝑦𝑖 b + 𝛼𝑖 − 𝛼𝑖 𝜉𝑖 −C 𝜉𝑖 + 𝛼𝑖 𝜉𝑖 = 0 L= 𝛼𝑖 + 1 2 ( 𝛼𝑖 𝑦𝑖 𝑥𝑖 ).( 𝛼𝑗 𝑦𝑗 𝑥𝑗 )- ( 𝛼𝑖 𝑦𝑖 𝑥𝑖). ( 𝛼𝑗 𝑦𝑗 𝑥𝑗) L= 𝛼𝑖 − 1 2 𝛼𝑖 𝑦𝑖 𝛼𝑗 𝑦𝑗 (𝑥𝑖. 𝑥𝑗) Department of Computer Science and Engineering, LBS College of Engineering, Kasaragod 14 L= 1 2 ||𝑤|| 2 + C 𝜉𝑖 - 𝛼𝑖 𝑦𝑖(𝑤. 𝑥𝑖 + b) − 1 + 𝜉𝑖 − 𝜇𝑖 𝜉𝑖 𝑤 = 𝛼𝑖 𝑦𝑖 𝑥𝑖 𝛼𝑖 𝑦𝑖 = 0 𝜇𝑖= C -𝛼𝑖
- 15. SVM Soft Margin Classifier 𝜙 𝛼 = 𝛼 𝑖 − 1 2 𝛼𝑖 𝑦𝑖 𝛼𝑗 𝑦𝑗 (𝑥𝑖. 𝑥𝑗) Compute 𝑤 and b Dual Problem: Find vector 𝛼 which maximizes Subject to 𝛼𝑖 𝑦𝑖 = 0 SVM Classifier Function: 𝑏 = 1 2 (𝑚𝑖𝑛𝑖:𝑦 𝑖=+1(𝑤. 𝑥𝑖) + 𝑚𝑎𝑥𝑖:𝑦 𝑖=−1(𝑤. 𝑥𝑖)) 𝑤 = 𝛼𝑖 𝑦𝑖 𝑥𝑖 f( 𝑥) =(𝑤. 𝑥) - b Department of Computer Science and Engineering, LBS College of Engineering, Kasaragod 15 0 ≤ 𝛼𝑖 ≤ 𝐶
- 16. Hyperparamter Tuning 16 Department of Computer Science and Engineering, LBS College of Engineering, Kasaragod
- 17. C=1 17 Department of Computer Science and Engineering, LBS College of Engineering, Kasaragod
- 18. C=2000 18 Department of Computer Science and Engineering, LBS College of Engineering, Kasaragod
- 19. C=78 19 Department of Computer Science and Engineering, LBS College of Engineering, Kasaragod
- 20. Hyperparameter Tuning Parameter Range Step Length C [20, 211] 21 20 Parameter Cherkassky et. al. [11] C max( 𝑦 + 3𝜎 , | 𝑦 − 3𝜎|) Heuristic Methods Search Space Department of Computer Science and Engineering, LBS College of Engineering, Kasaragod
- 21. Reference 1. Sudheep Elayidom, M. Data mining and warehousing, Cengage. 2. V. Vapnik, The Nature of Statistical Learning Theory, Springer, 1995 3. Vapnik, V. Statistical Learning Theory. John Wiley & Sons. Inc., New York, 1998 4. B. Schölkopf and A. J. Smola, Learning with Kernels, MIT Press, Cambridge, MA, 2002 5. Davide, M. and Simon, H. Advances in Kernel Methods, 1999, 226-227. 6. Jaiwei Han, Micheline Kamber, “Data Mining Concepts and Techniques”, Elsevier, 2006. 7. Pang-Ning Tan, Michael Steinbach, “Introduction to Data Mining”, Addison Wesley, 2006. 8. Dunham M H, “Data Mining: Introductory and Advanced Topics”, Pearson Education, New Delhi, 2003. 9. Mehmed Kantardzic, “Data Mining Concepts, Methods and Algorithms”, John Wiley and Sons, USA, 2003. 10. Sarith Divakar M, Sudheep Elayidom M, Rajesh R, “An efficient approach for crop yield forecasting using machine learning techniques based on normalized difference vegetation index and climatic indices”, JARDCS, Vol. 10, 15-Special Issue, 2018 11. Cherkassky, V. and Ma, Y. Selection of meta-parameters for support vector regression. International Con-ference on Artificial Neural Networks, 2002, 687-693 Department of Computer Science and Engineering, LBS College of Engineering, Kasaragod 21