2020: Phase transitions in layered neural networks: ReLU vs. sigmoidal activation
0 likes297 views
The document discusses phase transitions in neural networks with different activation functions (sigmoid vs ReLU). It finds that for sigmoid activations with more hidden units (K>2), there is a discontinuous first-order phase transition from an initial unspecialized state to a specialized state that facilitates learning. For ReLU activations, the transition is continuous, with both specialized and anti-specialized states achieving good generalization. Monte Carlo simulations show dynamics converging to specialized or anti-specialized states depending on initialization.
![CITEC June 2020
Phase transitions in layered neural networks:
rectified linear units vs. sigmoidal activation
Elisa Oostwal
Michiel Straat
Michael Biehl
Bernoulli Institute for Mathematics,
Computer Science and Artificial Intelligence
University of Groningen / NL
arXiv:1910.07476: Hidden unit Specialization in Layered Neural Networks:
ReLU vs. Sigmoidal Activation [updated May 27, 2020]](https://image.slidesharecdn.com/2020relusigmoidal-split-200707130320/85/2020-Phase-transitions-in-layered-neural-networks-ReLU-vs-sigmoidal-activation-1-320.jpg)
![CITEC June 2020
generalization error
on average over P({xi,xj
*})
[D. Saad, S. Solla, 1995]](https://image.slidesharecdn.com/2020relusigmoidal-split-200707130320/85/2020-Phase-transitions-in-layered-neural-networks-ReLU-vs-sigmoidal-activation-27-320.jpg)
![CITEC June 2020
generalization error
on average over P({xi,xj
*})
[D. Saad, S. Solla, 1995]
[M. Straat, 2019]](https://image.slidesharecdn.com/2020relusigmoidal-split-200707130320/85/2020-Phase-transitions-in-layered-neural-networks-ReLU-vs-sigmoidal-activation-28-320.jpg)
2020: Phase transitions in layered neural networks: ReLU vs. sigmoidal activation
- 1. CITEC June 2020 Phase transitions in layered neural networks: rectified linear units vs. sigmoidal activation Elisa Oostwal Michiel Straat Michael Biehl Bernoulli Institute for Mathematics, Computer Science and Artificial Intelligence University of Groningen / NL arXiv:1910.07476: Hidden unit Specialization in Layered Neural Networks: ReLU vs. Sigmoidal Activation [updated May 27, 2020]
- 2. CITEC June 2020 motivation/research question ReLU activation, e.g. in deep learning folklore: • computationally cheap and fast • circumvents “vanishing gradient” problem • sparse activity (biologically plausible) ? more efficient training ? favorable generalization ability
- 3. CITEC June 2020 motivation/research question ReLU activation, e.g. in deep learning folklore: • computationally cheap and fast • circumvents “vanishing gradient” problem • sparse activity (biologically plausible) ? more efficient training ? favorable generalization ability Theoretical study • model scenarios, student/teacher setup • typical learning curves, statistical physics based approach • systematic comparison of ReLU and “classical” sigmoidal activation • significant differences or “only” practical issues?
- 4. CITEC June 2020 statistical physics (of learning) in a nutshell objective/cost/energy function with • training by stochastic optimization of adaptive weights (thresholds etc.)
- 5. CITEC June 2020 statistical physics (of learning) in a nutshell objective/cost/energy function with • Metropolis algorithm, noisy gradient descent (Langevin) with long time equilibrium (Gibbs-Boltzmann) control parameter: „inverse temperature“ β =1 / T • training by stochastic optimization of adaptive weights (thresholds etc.)
- 6. CITEC June 2020 statistical physics (of learning) in a nutshell objective/cost/energy function with • equilibrium state: compromise/competition between minimal energy (ground state) vs. huge number of available states with higher energy • Metropolis algorithm, noisy gradient descent (Langevin) with long time equilibrium (Gibbs-Boltzmann) control parameter: „inverse temperature“ β =1 / T • training by stochastic optimization of adaptive weights (thresholds etc.)
- 7. CITEC June 2020 statistical physics (of learning) in a nutshell objective/cost/energy function with • equilibrium state: compromise/competition between minimal energy (ground state) vs. huge number of available states with higher energy • Metropolis algorithm, noisy gradient descent (Langevin) with long time equilibrium (Gibbs-Boltzmann) control parameter: „inverse temperature“ β =1 / T • training by stochastic optimization of adaptive weights (thresholds etc.) minimal free energy (per weight)
- 8. CITEC June 2020 statistical physics (of learning) in a nutshell objective/cost/energy function with • equilibrium state: compromise/competition between minimal energy (ground state) vs. huge number of available states with higher energy • Metropolis algorithm, noisy gradient descent (Langevin) with long time equilibrium (Gibbs-Boltzmann) control parameter: „inverse temperature“ β =1 / T • training by stochastic optimization of adaptive weights (thresholds etc.) „thermal avg.“ over Peq , (microcanonical) entropy minimal free energy (per weight)
- 9. CITEC June 2020 • computation of system properties in equilibrium: thermal averages statistical physics (of learning) in a nutshell
- 10. CITEC June 2020 T =1 / β controls competition between energy and entropy singles out the lowest energy (groundstate) all states occur with equal probability, independent of energy thermal noise dominates the training • computation of system properties in equilibrium: thermal averages statistical physics (of learning) in a nutshell
- 11. CITEC June 2020 T =1 / β controls competition between energy and entropy singles out the lowest energy (groundstate) all states occur with equal probability, independent of energy thermal noise dominates the training • computation of system properties in equilibrium: thermal averages statistical physics (of learning) in a nutshell • practical machine learning: algorithm (hyper-) parameters control minimization of the cost function e.g. learning rate, weight decay ...
- 12. CITEC June 2020 T =1 / β controls competition between energy and entropy singles out the lowest energy (groundstate) all states occur with equal probability, independent of energy thermal noise dominates the training • computation of system properties in equilibrium: thermal averages statistical physics (of learning) in a nutshell • practical machine learning: algorithm (hyper-) parameters control minimization of the cost function e.g. learning rate, weight decay ... • statistical physics / stochastic optimization design of algorithms (Metrropolis, simulated annealing) theoretical analysis of model scenarios
- 13. CITEC June 2020 machine learning • optimization of adaptive quantities, e.g. weights of a network based on a given specific set of example data (input/output pairs) cost function: defined w.r.t.
- 14. CITEC June 2020 machine learning • optimization of adaptive quantities, e.g. weights of a network based on a given specific set of example data (input/output pairs) cost function: defined w.r.t. • typical results: additional average of thermal averages over (difficult!) even for the simplest model densities: with independent identically distributed (zero mean, unit vaiance) unstructured input density, information about the target through labels
- 15. CITEC June 2020 machine learning • optimization of adaptive quantities, e.g. weights of a network based on a given specific set of example data (input/output pairs) cost function: defined w.r.t. • typical results: additional average of thermal averages over (difficult!) even for the simplest model densities: with independent identically distributed (zero mean, unit vaiance) unstructured input density, information about the target through labels proper -average of the free energy requires replica trick or annealed approximation
- 16. CITEC June 2020 machine learning at high temperatures • a simplifying limit: training at high (formal) temperature
- 17. CITEC June 2020 machine learning at high temperatures • a simplifying limit: training at high (formal) temperature • independent i.i.d. training data: • extensive number of examples: (prop. to number of weights)
- 18. CITEC June 2020 machine learning at high temperatures • a simplifying limit: training at high (formal) temperature • independent i.i.d. training data: • extensive number of examples: (prop. to number of weights) with finite “ learn almost nothing... ” (high T ) ...from infinitely many examples ”
- 19. CITEC June 2020 machine learning at high temperatures • a simplifying limit: training at high (formal) temperature • independent i.i.d. training data: • extensive number of examples: (prop. to number of weights) with finite “ learn almost nothing... ” (high T ) ...from infinitely many examples ” limitations: - training error and generalization error are identical - number of examples / training temperature are coupled
- 20. CITEC June 2020 adaptive student N inputs (K) hidden units (M) teacher ? ? ? ? ? ? ? student teacher scenario: “soft committees”
- 21. CITEC June 2020 adaptive student N inputs (K) hidden units (M) teacher ? ? ? ? ? ? ? student teacher scenario: “soft committees” training: minimization of here: learnable rules, reliable data (outputs provided by teacher) perfectly matching complexity K=M
- 22. CITEC June 2020 adaptive student N inputs (K) hidden units (M) teacher ? ? ? ? ? ? ? student teacher scenario: “soft committees” training: minimization of here: learnable rules, reliable data (outputs provided by teacher) perfectly matching complexity K=M consider two prototypical activation functions: sigmoidal / ReLU in student and teacher
- 23. CITEC June 2020 exploit thermodynamic limit, CLT for normally distributed with zero mean and covariance matrix large N: Central Limit Theorem
- 24. CITEC June 2020 exploit thermodynamic limit, CLT for normally distributed with zero mean and covariance matrix large N: Central Limit Theorem order parameters: model parameters:macroscopic properties of the system
- 25. CITEC June 2020 exploit thermodynamic limit, CLT for normally distributed with zero mean and covariance matrix large N: Central Limit Theorem order parameters: model parameters:macroscopic properties of the system (+ constant) independent of details (e.g. activation)
- 26. CITEC June 2020 generalization error on average over P({xi,xj *})
- 27. CITEC June 2020 generalization error on average over P({xi,xj *}) [D. Saad, S. Solla, 1995]
- 28. CITEC June 2020 generalization error on average over P({xi,xj *}) [D. Saad, S. Solla, 1995] [M. Straat, 2019]
- 29. CITEC June 2020 site symmetry simplification: orthonormal teacher vectors, isotropic input density reflects permutation symmetry, allows for hidden unit specialization
- 30. CITEC June 2020 site symmetry simplification: orthonormal teacher vectors, isotropic input density reflects permutation symmetry, allows for hidden unit specialization entropy (+ constant)
- 31. CITEC June 2020 site symmetry simplification: orthonormal teacher vectors, isotropic input density reflects permutation symmetry, allows for hidden unit specialization sigmoidal hidden units entropy (+ constant)
- 32. CITEC June 2020 site symmetry simplification: orthonormal teacher vectors, isotropic input density reflects permutation symmetry, allows for hidden unit specialization sigmoidal hidden units ReLU activations entropy (+ constant)
- 33. CITEC June 2020 given: K, g(x), typical learning curves
- 34. CITEC June 2020 given: K, g(x), typical learning curves solve:
- 35. CITEC June 2020 determine (global and local) minima of given: K, g(x), obtain learning curves typical learning curves order parameters, generalization error (typical, average) as a function of the (scaled) training set size solve:
- 36. CITEC June 2020 sigmoidal (K=2)
- 37. CITEC June 2020 sigmoidal (K=2) invariance under exchange of the two hidden units R=S: both units ~ (w1 * + w2 *) + noise symmetry breaking phase transition (continuous, “second order”)...
- 38. CITEC June 2020 sigmoidal (K=2) invariance under exchange of the two hidden units R=S: both units ~ (w1 * + w2 *) + noise symmetry breaking phase transition (continuous, “second order”)... ... results in a kink in the typical learning curve
- 39. CITEC June 2020 sigmoidal (K=2) invariance under exchange of the two hidden units R=S: both units ~ (w1 * + w2 *) + noise symmetry breaking phase transition (continuous, “second order”)... ... results in a kink in the typical learning curve
- 40. CITEC June 2020 continuous transition (schematic) R<S R>S R=S βf βf
- 41. CITEC June 2020 ReLU (K=2) qualitatively identical behavior Note: num. values of and/or are irrelevant, scale depends a.o. on pre-factor of g(z)
- 42. CITEC June 2020 ReLU (K=2) qualitatively identical behavior Note: num. values of and/or are irrelevant, scale depends a.o. on pre-factor of g(z)
- 43. CITEC June 2020 sigmoidal (K>2) K=5 permutation symmetry of h.u. initial R=S phase
- 44. CITEC June 2020 sigmoidal (K>2) K=5 permutation symmetry of h.u. initial R=S phase first order transition, local min. R>S competes with R=S R>S becomes global minimum facilitates perfect learning
- 45. CITEC June 2020 sigmoidal (K>2) K=5 permutation symmetry of h.u. initial R=S phase first order transition, local min. R>S competes with R=S R>S becomes global minimum facilitates perfect learning additional transition: “anti-specialization” S>R (overlooked in 1998!)
- 46. CITEC June 2020 sigmoidal (K>2) K=5 permutation symmetry of h.u. initial R=S phase first order transition, local min. R>S competes with R=S R>S becomes global minimum facilitates perfect learning additional transition: “anti-specialization” S>R (overlooked in 1998!)
- 47. CITEC June 2020 sigmoidal (K>2) K=5 permutation symmetry of h.u. initial R=S phase first order transition, local min. R>S competes with R=S R>S becomes global minimum facilitates perfect learning additional transition: “anti-specialization” S>R (overlooked in 1998!)
- 48. CITEC June 2020 sigmoidal (K>2) K=5 permutation symmetry of h.u. initial R=S phase discontinuous jump in ε g coexistence of poor and good generalization weak / no effect of additional anti-specialization on gen. error first order transition, local min. R>S competes with R=S R>S becomes global minimum facilitates perfect learning additional transition: “anti-specialization” S>R (overlooked in 1998!)
- 49. CITEC June 2020 discontinuous transition (schematic) R=S R>S R>S R<S R=S R=S R>S βf βf βf βf
- 50. CITEC June 2020 ReLU (K>2) K=10 permutation symmetry of h.u. initial R=S phase
- 51. CITEC June 2020 ReLU (K>2) K=10 permutation symmetry of h.u. initial R=S phase continuous phase transtion global minimum: R>S local minimum: R<S
- 52. CITEC June 2020 ReLU (K>2) K=10 permutation symmetry of h.u. initial R=S phase continuous kink(s) in ε g competing minima of poor* vs. good generalization * pretty good continuous phase transtion global minimum: R>S local minimum: R<S
- 53. CITEC June 2020 ReLU (large K) permutation symmetry of h.u. initial R=S phase
- 54. CITEC June 2020 ReLU (large K) permutation symmetry of h.u. initial R=S phase continuous phase transtion at degenerate minima: R>S, R<S
- 55. CITEC June 2020 ReLU (large K) permutation symmetry of h.u. initial R=S phase specialized and anti-specialized branch both achieve perfect generalization, asymptotically ! continuous phase transtion at degenerate minima: R>S, R<S
- 56. CITEC June 2020 R=1: perfect agreement of x with x* R≈0: conditional avg. (linear!)
- 57. CITEC June 2020 perfectly aligned, specialized student = teacher R=1: perfect agreement of x with x* R≈0: conditional avg. (linear!)
- 58. CITEC June 2020 = “anti-specialized” student, large K perfectly aligned, specialized student = teacher R=1: perfect agreement of x with x* R≈0: conditional avg. (linear!) +
- 59. CITEC June 2020 unspecialized R=S state remains meta-stable up to large hidden layer: sigmoidal (large K) perfect generalization without prior knowledge impossible with order O(NK) examples ?
- 60. CITEC June 2020 Monte Carlo simulations continous Metropolis dynamics, K=4, N=50, β=1 (=T) generalization error vs. time, specialized and unspecialized initialization
- 61. CITEC June 2020 Monte Carlo simulations continous Metropolis dynamics, K=4, N=50, β=1 (=T) generalization error vs. time, specialized and unspecialized initialization
- 62. CITEC June 2020 Monte Carlo simulations histogram of observed R continous Metropolis dynamics, K=4, N=50, β=1 (=T) generalization error vs. time, specialized and unspecialized initialization unspecialized
- 63. CITEC June 2020 Monte Carlo simulations histogram of observed R continous Metropolis dynamics, K=4, N=50, β=1 (=T) generalization error vs. time, specialized and unspecialized initialization anti-specialized specialized unspecialized
- 64. CITEC June 2020 Monte Carlo simulations sigmoidal activation K=4
- 65. CITEC June 2020 Monte Carlo simulations sigmoidal activation ReLUK=4
- 66. CITEC June 2020 • formal equilibrium of training at high temperature in student/teacher model situations of supervised learning Summary
- 67. CITEC June 2020 • formal equilibrium of training at high temperature in student/teacher model situations of supervised learning • unspecialized and (partially) specialized configurations compete as local/global minima of the free energy • phase transitions with temperature / number of examples: Summary
- 68. CITEC June 2020 • formal equilibrium of training at high temperature in student/teacher model situations of supervised learning • unspecialized and (partially) specialized configurations compete as local/global minima of the free energy • phase transitions with temperature / number of examples: K=2: continuous symmetry-breaking transitions with equivalent competing states K>2, sigmoidal activations: first order transition with competing states of distinct generalization ability K>2, ReLU networks: continuous transition with competing states of similar performance Summary
- 69. CITEC June 2020 Outlook which is the decisive property of the activation? • consider various activation functions (leaky ReLU ✓, swish ... ) most important question:
- 70. CITEC June 2020 piece-wise linear „sigmoidal“ activation ReLU Outlook which is the decisive property of the activation? • consider various activation functions (leaky ReLU ✓, swish ... ) most important question:
- 71. CITEC June 2020 piece-wise linear „sigmoidal“ activation ReLU with increasing slope, change from discontinuous to continuous transition Outlook which is the decisive property of the activation? • consider various activation functions (leaky ReLU ✓, swish ... ) most important question:
- 72. CITEC June 2020 • annealed approximation / replica trick: - low temperatures, vary number of examples and temperature indep. - mismatched student/teacher networks - overfitting / underfitting effects Outlook
- 73. CITEC June 2020 • annealed approximation / replica trick: - low temperatures, vary number of examples and temperature indep. - mismatched student/teacher networks - overfitting / underfitting effects • universal approximators - adaptive thresholds in hidden units - adaptive hidden-to-output weights Outlook
- 74. CITEC June 2020 • annealed approximation / replica trick: - low temperatures, vary number of examples and temperature indep. - mismatched student/teacher networks - overfitting / underfitting effects • universal approximators - adaptive thresholds in hidden units - adaptive hidden-to-output weights • realistic input data - clustered / correlated data - recent developments: Zdeborova, Mezard, Goldt et al. Outlook
- 75. CITEC June 2020 • annealed approximation / replica trick: - low temperatures, vary number of examples and temperature indep. - mismatched student/teacher networks - overfitting / underfitting effects • universal approximators - adaptive thresholds in hidden units - adaptive hidden-to-output weights • deep networks - multi-layered networks - tree-like architectures • realistic input data - clustered / correlated data - recent developments: Zdeborova, Mezard, Goldt et al. Outlook