Sparse Regularization
1 like760 views
1) The document discusses sparse methods for machine learning, including regularization techniques like L1 and L2 regularization. 2) L1 regularization results in sparse solutions with many zero weights, which can save memory and computation costs. 3) Compressive sensing is introduced as applying sparse recovery to problems where the underlying signal is sparse or compressible in some domain. 4) Dictionary learning is presented as a way to learn a representation dictionary from data where data points can be represented sparsely with few non-zero coefficients.
Sparse Regularization
- 1. Introduction to Sparse Methods Shadi Albarqouni, M.Sc. Research Assistant | PhD Candidate! shadi.albarqouni@tum.de ! Machine Learning in Medical Imaging! BioMedical Computing (BMC) Master Program Computer Aided Medical Procedures | Technische Universität München
- 3. Introduction Regularization Sparsity Extensions Model Announcement! Hiring Data Scientists Let’s apply! Boss Christoph
- 4. Introduction Regularization Sparsity Extensions Model Announcement! Hiring Data Scientists Let’s apply! Boss Christoph 1st day of his work
- 5. Introduction Regularization Sparsity Extensions Model Announcement! Hiring Data Scientists Let’s apply! We want to build a Predictive Model for the wages! Boss Christoph 1st day of his work
- 6. Introduction Regularization Sparsity Extensions Model Announcement! Hiring Data Scientists Let’s apply! We want to build a Predictive Model for the wages! What!! Boss Christoph 1st day of his work
- 7. Introduction Regularization Sparsity Extensions Model Announcement! Hiring Data Scientists Let’s apply! We want to build a Predictive Model for the wages! What!! What kind of data you have? How Big? Boss Christoph 1st day of his work
- 8. Introduction Regularization Sparsity Extensions Model Announcement! Hiring Data Scientists Let’s apply! We want to build a Predictive Model for the wages! What!! What kind of data you have? How Big? Data Cleanup, Pre- processing, overfitting! Boss Christoph 1st day of his work
- 9. Introduction Regularization Sparsity Extensions Model Monthly Wages January Wages 0 17,5 35 52,5 70 0 1 2 3 4 5 6 7 8 9 10 Shadi Chrsitian Alma Kathrina Average Observations Boss
- 10. Introduction Regularization Sparsity Extensions Model Monthly Wages October Wages 0 17,5 35 52,5 70 0 1 2 3 4 5 6 7 8 9 10 Shadi Chrsitian Alma Kathrina Average Observations Boss
- 11. Introduction Regularization Sparsity Extensions Model Monthly Wages October Wages 0 17,5 35 52,5 70 0 1 2 3 4 5 6 7 8 9 10 Shadi Chrsitian Alma Kathrina Average Observations Boss Here the observations we have
- 12. Introduction Regularization Sparsity Extensions Model Curve fitting Monthly Wages Wages 0 17,5 35 52,5 70 0 1 2 3 4 5 6 7 8 9 10 Shadi Chrsitian Alma Kathrina Average x y Christoph
- 13. Introduction Regularization Sparsity Extensions Model Curve fitting y=a0 Monthly Wages Wages 0 17,5 35 52,5 70 0 1 2 3 4 5 6 7 8 9 10 Shadi Chrsitian Alma Kathrina Average x y Christoph
- 14. Introduction Regularization Sparsity Extensions Model Curve fitting y=a0 Monthly Wages Wages 0 17,5 35 52,5 70 0 1 2 3 4 5 6 7 8 9 10 Shadi Chrsitian Alma Kathrina Average x y Under-fitting Christoph
- 15. Introduction Regularization Sparsity Extensions Model Curve fitting y=a0 Monthly Wages Wages 0 17,5 35 52,5 70 0 1 2 3 4 5 6 7 8 9 10 Shadi Chrsitian Alma Kathrina Average x y Christoph
- 16. Introduction Regularization Sparsity Extensions Model Curve fitting Monthly Wages Wages 0 17,5 35 52,5 70 0 1 2 3 4 5 6 7 8 9 10 Shadi Chrsitian Alma Kathrina Average x y Christoph
- 17. Introduction Regularization Sparsity Extensions Model Curve fitting Monthly Wages Wages 0 17,5 35 52,5 70 0 1 2 3 4 5 6 7 8 9 10 Shadi Chrsitian Alma Kathrina Average y=a1x x y Christoph
- 18. Introduction Regularization Sparsity Extensions Model Curve fitting Monthly Wages Wages 0 17,5 35 52,5 70 0 1 2 3 4 5 6 7 8 9 10 Shadi Chrsitian Alma Kathrina Average x y Christoph
- 19. Introduction Regularization Sparsity Extensions Model Curve fitting Monthly Wages Wages 0 17,5 35 52,5 70 0 1 2 3 4 5 6 7 8 9 10 Shadi Chrsitian Alma Kathrina Average y=a0+a1x x y Christoph
- 20. Introduction Regularization Sparsity Extensions Model Curve fitting Monthly Wages Wages 0 17,5 35 52,5 70 0 1 2 3 4 5 6 7 8 9 10 Shadi Chrsitian Alma Kathrina Average x y Christoph
- 21. Introduction Regularization Sparsity Extensions Model Curve fitting Monthly Wages Wages 0 17,5 35 52,5 70 0 1 2 3 4 5 6 7 8 9 10 Shadi Chrsitian Alma Kathrina Average y=a0+a1x+a2x^2+…+aNx^N x y Christoph
- 22. Introduction Regularization Sparsity Extensions Model Curve fitting Monthly Wages Wages 0 17,5 35 52,5 70 0 1 2 3 4 5 6 7 8 9 10 Shadi Chrsitian Alma Kathrina Average y=a0+a1x+a2x^2+…+aNx^N x y Over-fitting Christoph
- 23. Introduction Regularization Sparsity Extensions Model Curve fitting Monthly Wages Wages 0 17,5 35 52,5 70 0 1 2 3 4 5 6 7 8 9 10 Shadi Chrsitian Alma Kathrina Average y=a0+a1x+a2x^2+…+aNx^N x y Over-fitting Christoph Which one will make the Boss happy!!
- 24. Introduction Regularization Sparsity Extensions Model Wages 0 17,5 35 52,5 70 0 1 2 3 4 5 6 7 8 9 10 Shadi Chrsitian Alma Kathrina Average_Female Average_Male x y More Interesting and Complicated Boss
- 25. Introduction Regularization Sparsity Extensions Model Wages 0 17,5 35 52,5 70 0 1 2 3 4 5 6 7 8 9 10 Shadi Chrsitian Alma Kathrina Average_Female Average_Male x y More Interesting and Complicated Boss I see! Based on your Model, we should hire in the first quarter, the question is Male or Female? Age? Experience?
- 26. Introduction Regularization Sparsity Extensions Model Building a Model / Formulation Christoph y x 7 9 20 35 1 2 1 7 3 1 12 26 Obser. Features Wages Month Age Gender Exp. …. = Linear Predictive Model few data points -> under-fitting more data points -> robust …. over-fitting? AModel 0.2 0.4 0.1 0.8 0.1 . y = A.x + e Where e is a white gaussian noise
- 27. Introduction Regularization Sparsity Extensions Model Building a Model / Formulation Christoph I decided for a Linear Predictive Model … Pretty Easy! y x 7 9 20 35 1 2 1 7 3 1 12 26 Obser. Features Wages Month Age Gender Exp. …. = Linear Predictive Model few data points -> under-fitting more data points -> robust …. over-fitting? AModel 0.2 0.4 0.1 0.8 0.1 . y = A.x + e Where e is a white gaussian noise
- 29. Introduction Regularization Sparsity Extensions Model Generalized Model Boss You did a good job Christoph! What about a generalized model, we would enter the image processing market, so we may need Image Restoration or Reconstruction!
- 31. Introduction Regularization Sparsity Extensions Model Challenges Christoph OMG, i haven’t heard about that …. how we can solve such a problem!
- 33. Introduction Regularization Sparsity Extensions Model Challenges Christoph Well, it seems not that easy! We need a regularization!
- 35. Introduction Regularization Sparsity Extensions Model Regularization Christoph Ah! Which one would be the best? … I can’t understand how this penalty function would solve the posedness?
- 36. Introduction Regularization Sparsity Extensions Model L2 & L1 Regularization Christoph
- 37. Introduction Regularization Sparsity Extensions Model L2 & L1 Regularization Christoph Let’s have a simple example with only two features!
- 38. Introduction Regularization Sparsity Extensions Model L2 & L1 Regularization Christoph Let’s have a simple example with only two features! x1 x2
- 39. Introduction Regularization Sparsity Extensions Model L2 & L1 Regularization Christoph Let’s have a simple example with only two features! x1 x2
- 40. Introduction Regularization Sparsity Extensions Model L2 & L1 Regularization Christoph Let’s have a simple example with only two features! x1 x2
- 41. Introduction Regularization Sparsity Extensions Model L2 & L1 Regularization Christoph Let’s have a simple example with only two features! x1 x2
- 42. Introduction Regularization Sparsity Extensions Model L2 & L1 Regularization Christoph Let’s have a simple example with only two features! x1 x2
- 43. Introduction Regularization Sparsity Extensions Model L2 & L1 Regularization Christoph Let’s have a simple example with only two features! x1 x2
- 44. Introduction Regularization Sparsity Extensions Model L2 & L1 Regularization Christoph Let’s have a simple example with only two features! x1 x2
- 45. Introduction Regularization Sparsity Extensions Model L2 & L1 Regularization Christoph Let’s have a simple example with only two features! x1 x2
- 46. Introduction Regularization Sparsity Extensions Model L2 & L1 Regularization Christoph Let’s have a simple example with only two features! x1 x2
- 47. Introduction Regularization Sparsity Extensions Model L2 & L1 Regularization Christoph Let’s have a simple example with only two features! x1 x2
- 48. Introduction Regularization Sparsity Extensions Model L2 & L1 Regularization Christoph Let’s have a simple example with only two features! x1 x2
- 50. Introduction Regularization Sparsity Extensions Model Compressive Sensing Christoph Boss Interesting! We got few elements (features/weights) with L1 This would definitely help us in Memory saving, computations costs … less Money!
- 51. Introduction Regularization Sparsity Extensions Model Compressive Sensing Christoph Boss Interesting! We got few elements (features/weights) with L1 This would definitely help us in Memory saving, computations costs … less Money! That’s true! However, we don’t have prior info. about the latent variable (Sparse or not)? … Wavelet, Fourier, … Dictionary
- 54. Introduction Regularization Sparsity Extensions Model Dictionary Learning D xi =a5d5 +a8d8 +a35d35 …. +a120d120 xi Data points are represented sparely with few non-zero elements
- 58. Introduction Regularization Sparsity Extensions Model Boss Christoph Hey Christoph! We have a plan to work more in this sparse coding topic, please keep working on that.
- 59. Introduction Regularization Sparsity Extensions Model Boss Christoph Hey Christoph! We have a plan to work more in this sparse coding topic, please keep working on that. I have read some materials on Graph Sparse Coding … etc.
- 60. Introduction Regularization Sparsity Extensions Model Boss Christoph Hey Christoph! We have a plan to work more in this sparse coding topic, please keep working on that. I have read some materials on Graph Sparse Coding … etc. Great! Keep doing that! Please if you have any question, don’t hesitate to contact me: shadi.albarqouni@tum.de