![Back-propagation (1980)
• Back-propagation is a learning algorithm for multi-layer neural
networks
• It was invented independently several times
• Bryson an Ho [1969]
• Werbos [1974]
• Parker [1985]
• Rumelhart et al. [1986]
Parallel Distributed Processing - Vol. 1
Foundations
David E. Rumelhart, James L. McClelland and the PDP Research
Group
What makes people smarter than computers? These volumes by
a pioneering neurocomputing.....](https://image.slidesharecdn.com/1in21-240913161729-0ad8dbf3/85/MachineLearning_Road-to-deep-learning-pdf-44-320.jpg)
MachineLearning_Road to deep learning.pdf
- 1. Lecture 1: Machine Learning Andreas Wichert Department of Computer Science and Engineering Técnico Lisboa
- 2. Corpo docente – Alameda/Tagus • Andreas (Andrzej) Wichert • andreas.wichert@tecnico.ulisboa.pt • tel: 214233231 • room: N2 5-7 (Taguspark) • http://web.tecnico.ulisboa.pt/andreas.wichert/
- 3. Main Literature • Christopher M. Bishop, Pattern Recognition and Machine Learning (Information Science and Statistics), Springer 2006 • https://www.microsoft.com/en-us/research/people/cmbishop/#!prml-book • Simon O. Haykin, Neural Networks and Learning Machine, (3rd Edition), Pearson 2008 • Deep Learning, I. Goodfellow, Y. Bengio, A. Courville MIT Press 2016 • https://www.deeplearningbook.org
- 4. Main Literature • Machine Learning - A Journey to Deep Learning, A. Wichert, Luis Sa-Couto, World Scientific, 2021 • Intelligent Big Multimedia Databases, A. Wichert, World Scientific, 2015 • Preprocessing, Feature Extraction like DFT, Wavelets, will be not covered in the lecture….
- 5. Additional Literature • Machine Learning: A Probabilistic Perspective, K. Murphy, MIT Press 2012 • Introduction To The Theory Of Neural Computation (Santa Fe Institute Series Book 1), John A. Hertz, Anders S. Krogh, Richard G. Palmer, Addison-Wesley Pub. Co, Redwood City, CA; 1 edition (January 1, 1991) • I find this book to be one of the best written mathematical guides for Neural Networks. See Perceptron, Backpropagation…
- 6. Literature Software • Hands-On Machine Learning with Scikit-Learn and TensorFlow: Concepts, Tools, and Techniques to Build Intelligent Systems 1st Edition, Aurélien Géron , O'Reilly Media; 1 edition (April 9, 2017) • https://github.com/amitanalyste/aurelienGeron • https://scikit-learn.org/stable/index.html • http://www.numpy.org
- 7. I) Outline: Introduction: What is Machine Learning? 1. Introduction 2. Decision Trees Mathematical Tools: 3. Probability theory & Information (Naive Bayes) 4. Linear Algebra & Optimization (Simple NN) Road to deep learning: Error Minimization (Loss), Regularization, Optimization by Gradient descent 5. Linear Regression & Bayesian Linear Regression 6. Perceptron & Logistic Regression 7. Multilayer Perceptrons
- 8. II) Outline Why do the neural works work : 8. Learning theory, Bias-Variance 9. K-Means, EM-Clustering 10. Kernel Methods & RBF 11. Support Vector Machines How to use the models: 12. Model Selection
- 9. III) Outline Deep Learning solves the problem of high dimensionality which is related to the training database size! 13. Deep Learning 14. Convolutional Neural Networks 15. Recurrent Neural Networks Dimension Reduction: 16. PCA, ICA 17. Autoencoders
- 10. IV) Outline Alternative Road to Machine Learning (Classical Approach): 18. Feature Extraction (FFT, SFT, Edge Detection) 19. k Nearest Neighbour & Locally Weighted Regression 20. Ensemble Methods Probabilistic and Stochastic Approach: 21. Bayesian Networks 22. Stochastic Methods
- 11. What is machine Learning? • Parallels between “animals” and machine learning • Many techniques derived from efforts of psychologist / biologists to make more sense “animal” learning through computational models
- 12. Machine Learning • Statistical Machine Learning • Linear Regression • Clustering, Self Organizing Maps (SOM) • Artificial Neural Networks, Kernel Machines • Bayesian Network • We will not cover…. • Inductive Learning (ID3) • Knowledge Learning • Analogical Learning • SOAR: Model of Cognition and Learning
- 13. An Example of Symbolical Learning (Patrick Winston-1975)
- 14. An Example (Patrick Winston-1975)
- 15. An Example (Patrick Winston-1975)
- 16. An Example (Patrick Winston-1975)
- 17. Statistical Machine Learning • Changes in the system that perform tasks associated with AI • Recognition • Prediction • Planning • Diagnosis
- 18. Learning Input output functions • Supervised • With a teacher • Unsupervised • Without a teacher • Reinforcemet Learning • Actions within & responses from the environment • Absence of a designated teacher to give positive and negative examples
- 22. • We might add other features that are not correlated with the ones we already have. A precaution should be taken not to reduce the performance by adding such “noisy features” • Ideally, the best decision boundary should be the one which provides an optimal performance such as in the following figure:
- 24. • However, our satisfaction is premature because the central aim of designing a classifier is to correctly classify novel input Issue of generalization!
- 27. • 1040 Neurons • 104-5 connections per neuron
- 29. Perceptron (1957) • Linear threshold unit (LTU) S x1 x2 xn . . . w1 w2 wn w0 x0=1 o McCulloch-Pitts model of a neuron (1943) The “bias”, a constant term that does not depend on any input value
- 30. Linearly separable patterns X0=1, bias...
- 31. (a) The two classes 1 (indicated by a big point) and −1 (indicated by a small point) are separated by the line −1 + x1 + x2 = 0. (b) The hyperplane −1+x1+x2 =y defines the line for y=0.
- 32. • The goal of a perceptron is to correctly classify the set of pattern D={x1,x2,..xm} into one of the classes C1 and C2 • The output for class C1 is o=1 and for C2 is o=-1 • For n=2 è
- 33. Perceptron learning rule • Consider linearly separable problems • How to find appropriate weights • Initialize each vector w to some small random values • Look if the output pattern o belongs to the desired class, has the desired value d • h is called the learning rate • 0 < h ≤ 1 Δw =η⋅(d −o)⋅ x
- 34. • In supervised learning the network has its output compared with known correct answers • Supervised learning • Learning with a teacher • (d-o) plays the role of the error signal
- 36. Constructions
- 41. • Rosenblatt's bitter rival and professional nemesis was Marvin Minsky of Carnegie Mellon University • Minsky despised Rosenblatt, hated the concept of the perceptron, and wrote several polemics against him • For years Minsky crusaded against Rosenblatt on a very nasty and personal level, including contacting every group who funded Rosenblatt's research to denounce him as a charlatan, hoping to ruin Rosenblatt professionally and to cut off all funding for his research in neural nets
- 42. XOR problem and Perceptron • By Minsky and Papert in mid 1960
- 43. k Means Clustering (Unsupervised Learning) • The standard algorithm was first proposed by Stuart Lloyd in 1957
- 44. Back-propagation (1980) • Back-propagation is a learning algorithm for multi-layer neural networks • It was invented independently several times • Bryson an Ho [1969] • Werbos [1974] • Parker [1985] • Rumelhart et al. [1986] Parallel Distributed Processing - Vol. 1 Foundations David E. Rumelhart, James L. McClelland and the PDP Research Group What makes people smarter than computers? These volumes by a pioneering neurocomputing.....
- 45. The good old days…
- 46. Everyone was doing Back-propagation….
- 48. NETtalk Sejnowski et al 1987
- 52. Kunihiko Fukushima Kunihiko Fukushima received a B.Eng. degree in electronics in 1958 and a PhD degree in electrical engineering in 1966 from Kyoto University, Japan. He was a professor at Osaka University from 1989 to 1999, at the University of Electro-Communications from 1999 to 2001, at Tokyo University of Technology from 2001 to 2006; and a visiting professor at Kansai University from 2006 to 2010. Prior to his Professorship, he was a Senior Research Scientist at the NHK Science and Technology Research Laboratories. He is now a Senior Research Scientist at Fuzzy Logic Systems Institute (part-time position), and usually works at his home in Tokyo.
- 55. 2006
- 64. Data Set Size: 9th Order Polynomial
- 65. Data Set Size: 9th Order Polynomial
- 66. Regularization Penalize large coefficient values
- 67. Regularization:
- 68. Problem of Local Minima • The immediate solution to this is to build networks with more hidden layers with regularization • “Deep Learning”… • Déjà vu?
- 69. Artificial intelligence pioneer (Geoffrey Hinton ) says we need to start over • Back-propagation still has a core role in AI's future. • Entirely new methods will probably have to be invented • "I don't think it's how the brain works," he said. "We clearly don't need all the labeled data.
- 70. What is an „A“ ? • What makes something similar to something else (specifically what makes, for example, an uppercase letter 'A' recognisable as such) • Metamagical Themas, Douglas Hoffstader, Basic Books, 1985
- 72. •What is the essence of dogness or house-ness? •What is the essence of 'A'-ness? •What is the essence of a given person's face, that it will not be confused with other people's faces? • How to convey these things to computers, which seem to be best at dealing with hard-edged categories--categories having crystal-clear, perfectly sharp boundaries?
- 73. • What Next? • Example of what is machine learning: Decision Trees
- 74. Literature • Simon O. Haykin, Neural Networks and Learning Machine, (3rd Edition), Pearson 2008 • Chapter 1 • Christopher M. Bishop, Pattern Recognition and Machine Learning (Information Science and Statistics), Springer 2006 • Section 1.1
- 75. Literature • Machine Learning - A Journey to Deep Learning, A. Wichert, Luis Sa-Couto, World Scientific, 2021 • Chapter 1