![What is ELM?
• Extreme Learning Machines
• Used for Single Hidden Layer Feed Forward Networks
• Uses the Hidden Layer Output matrix Concept and
Generalized Pseudo Inverse
Hidden Layer Output Matrix
ELM-SLFN
𝑓𝐿 𝑥 =
𝑖=1
𝐿
β𝑖ℎ𝑖 𝑥 = ℎ 𝑥 β
where ,
β = β1, β2, … , β 𝐿
𝑇
, 𝑜𝑢𝑡𝑝𝑢𝑡 𝑤𝑒𝑖𝑔ℎ𝑡 𝑚𝑎𝑡𝑟𝑖𝑥
and
h(x)= [ℎ1 𝑥 , ℎ2 𝑥 , … , ℎ 𝐿 𝑥 ] , ℎ𝑖𝑑𝑑𝑒𝑛 𝑙𝑎𝑦𝑒𝑟 𝑜𝑢𝑡𝑝𝑢𝑡
𝑚𝑎𝑡𝑟𝑖𝑥
Generalized Pseudo Inverse
If the above problem is:
Ax = b
x = 𝑨ᵻ
𝒃](https://image.slidesharecdn.com/tffn-171212160144/85/TFFN-Two-Hidden-Layer-Feed-Forward-Network-using-the-randomness-of-Extreme-Learning-Machine-4-320.jpg)
![References
[1] Huang, Gao, et al. ”Trends in extreme learning machines: A review.” Neural Networks 61 (2015): 32-48.
[2] Cambria, Erik, et al. ”Extreme learning machines [trends & controversies].” IEEE Intelligent Systems 28.6
(2013): 30- 59.
[3] Huang, Guang-Bin. ”An insight into extreme learning machines: random neurons, random features and
kernels.” Cognitive Computation 6.3 (2014): 376-390.
[4] Huang, Guang-Bin, et al. ”Extreme learning machine for regression and multiclass classification.” IEEE
Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics) 42.2 (2012): 513-529.
[5] Huang, Guang-Bin, Qin-Yu Zhu, and Chee-Kheong Siew. ”Extreme learning machine: theory and
applications.” Neurocomputing 70.1 (2006): 489-501.
[6] Funahashi, Ken-Ichi. ”On the approximate realization of continuous mappings by neural networks.” Neural
networks 2.3 (1989): 183-192.
[7] Anthony, Martin, and Peter L. Bartlett. Neural network learning: Theoretical foundations. cambridge
university press, 2009. [8] Huang, Guang-Bin, and Lei Chen. ”Convex incremental extreme learning machine.”
Neurocomputing 70.16 (2007): 3056-3062.
[9] Albert, Arthur. Regression and the Moore-Penrose pseudoinverse. Elsevier, 1972.
[10] Penrose, Roger. ”A generalized inverse for matrices.” Mathematical proceedings of the Cambridge
philosophical society. Vol. 51. No. 3. Cambridge University Press, 1955.
[11] Golub, Gene H., and Charles F. Van Loan. ”An analysis of the total least squares problem.” SIAM Journal on
Numerical Analysis 17.6 (1980): 883-893.
[12] Lawson, Charles L., and Richard J. Hanson. Solving least squares problems. Society for Industrial and
Applied Mathematics, 1995.
[13] Huang, Guang-Bin, et al. ”Extreme learning machine for regression and multiclass classification.” IEEE
Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics) 42.2 (2012): 513-529.](https://image.slidesharecdn.com/tffn-171212160144/85/TFFN-Two-Hidden-Layer-Feed-Forward-Network-using-the-randomness-of-Extreme-Learning-Machine-10-320.jpg)
TFFN: Two Hidden Layer Feed Forward Network using the randomness of Extreme Learning Machine
- 1. TFFN: Two Hidden Layer Feed Forward Network using the randomness of Extreme Learning Machine Authors: Nimai Chand Das Adhikari Arpana Alka Dr. Raju K George Indian Institute of Space Science and Technology, Trivandrum
- 2. Problem Statement • Use of Back Propagation Algorithm for Feed Forward Neural Networks • Local Optima • Trivial Manual Intervention • Time Consumption
- 3. Proposal TFFN Uses Concept of ELM Has Better Generalization Capability Optimizes the Hyper- parameters
- 4. What is ELM? • Extreme Learning Machines • Used for Single Hidden Layer Feed Forward Networks • Uses the Hidden Layer Output matrix Concept and Generalized Pseudo Inverse Hidden Layer Output Matrix ELM-SLFN 𝑓𝐿 𝑥 = 𝑖=1 𝐿 β𝑖ℎ𝑖 𝑥 = ℎ 𝑥 β where , β = β1, β2, … , β 𝐿 𝑇 , 𝑜𝑢𝑡𝑝𝑢𝑡 𝑤𝑒𝑖𝑔ℎ𝑡 𝑚𝑎𝑡𝑟𝑖𝑥 and h(x)= [ℎ1 𝑥 , ℎ2 𝑥 , … , ℎ 𝐿 𝑥 ] , ℎ𝑖𝑑𝑑𝑒𝑛 𝑙𝑎𝑦𝑒𝑟 𝑜𝑢𝑡𝑝𝑢𝑡 𝑚𝑎𝑡𝑟𝑖𝑥 Generalized Pseudo Inverse If the above problem is: Ax = b x = 𝑨ᵻ 𝒃
- 5. Results Accuracy Comparison-Glass Accuracy Comparison-Diabetes ELM-SLFN Vs TFFN
- 6. TFFN: Two Hidden Layer Feed Forward Network Computation of Hidden layer output matrix Calculation of β
- 7. TFFN: Mathematical Formulation Two Hidden Layer Linear Formulation When closely seen, the above equation: Target 𝑊𝑖 𝑊𝑗
- 8. TFFN: Theorems behind it • Universal approximation capability • Classification Capability Theorem • Moore Penrose Generalized inverse • Least Squares Solution
- 9. Conclusions • The randomness concept avoids the term “Iterations” as in case of Back Propagation • It has a better “Generalization Capability” than Back Propagation • This automatically optimizes the Hyperparameters • The randomness can scrutinize the algorithm to “Move in one direction only” • The algorithm can’t assign priority on the basis of the training dataset
- 10. References [1] Huang, Gao, et al. ”Trends in extreme learning machines: A review.” Neural Networks 61 (2015): 32-48. [2] Cambria, Erik, et al. ”Extreme learning machines [trends & controversies].” IEEE Intelligent Systems 28.6 (2013): 30- 59. [3] Huang, Guang-Bin. ”An insight into extreme learning machines: random neurons, random features and kernels.” Cognitive Computation 6.3 (2014): 376-390. [4] Huang, Guang-Bin, et al. ”Extreme learning machine for regression and multiclass classification.” IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics) 42.2 (2012): 513-529. [5] Huang, Guang-Bin, Qin-Yu Zhu, and Chee-Kheong Siew. ”Extreme learning machine: theory and applications.” Neurocomputing 70.1 (2006): 489-501. [6] Funahashi, Ken-Ichi. ”On the approximate realization of continuous mappings by neural networks.” Neural networks 2.3 (1989): 183-192. [7] Anthony, Martin, and Peter L. Bartlett. Neural network learning: Theoretical foundations. cambridge university press, 2009. [8] Huang, Guang-Bin, and Lei Chen. ”Convex incremental extreme learning machine.” Neurocomputing 70.16 (2007): 3056-3062. [9] Albert, Arthur. Regression and the Moore-Penrose pseudoinverse. Elsevier, 1972. [10] Penrose, Roger. ”A generalized inverse for matrices.” Mathematical proceedings of the Cambridge philosophical society. Vol. 51. No. 3. Cambridge University Press, 1955. [11] Golub, Gene H., and Charles F. Van Loan. ”An analysis of the total least squares problem.” SIAM Journal on Numerical Analysis 17.6 (1980): 883-893. [12] Lawson, Charles L., and Richard J. Hanson. Solving least squares problems. Society for Industrial and Applied Mathematics, 1995. [13] Huang, Guang-Bin, et al. ”Extreme learning machine for regression and multiclass classification.” IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics) 42.2 (2012): 513-529.