WeightWatcher LLM Update
0 likes155 views
This document introduces WeightWatcher, a tool for analyzing the eigenvalues of weight matrices in deep neural networks. It was created by Dr. Charles H. Martin and Calculation Consulting to provide "data free diagnostics" for deep learning models using insights from random matrix theory and statistical mechanics. WeightWatcher can analyze pre-trained models to evaluate layer quality, predict generalization performance, and compare different network architectures, without access to the training data. The document provides an overview of the theoretical foundations and empirical evidence supporting WeightWatcher's methods.
![c|c
(TM)
WeightWatcher: analyzes the ESD
(eigenvalues) of the layer weight matrices
(TM)
8
calculation | consulting why deep learning works
Well trained layers are heavy-tailed and well shaped
GPT-2 Fits a Power Law
(or Truncated Power Law)
alpha in [2, 6]
watcher.analyze(plot=True)
Good quality of
fi
t (D is small)](https://image.slidesharecdn.com/tiijune2023-230621052629-2069c747/85/WeightWatcher-LLM-Update-8-320.jpg)
![c|c
(TM)
WeightWatcher: predict trends in generalization
(TM)
20
calculation | consulting why deep learning works
Predict test accuracies across variations in hyper-parameters
The average Power Law exponent alpha
predicts generalization—at
fi
xed depth
Smaller average-alpha is better
Better models are easier to treat
Charles H. Martin, Michael W. Mahoney;
[Contest post-mortem paper]](https://image.slidesharecdn.com/tiijune2023-230621052629-2069c747/85/WeightWatcher-LLM-Update-20-320.jpg)
WeightWatcher LLM Update
- 1. weight | watcher Data Free Diagnostics for Deep Learning (TM) c|c (TM) charles@calculationconsulting.com
- 2. weight|watcher Data Free Diagnostics for Deep Learning (TM) charles@calculationconsulting.com
- 3. calculation | consulting why deep learning works Who Are We? c|c (TM) Dr. Charles H. Martin, PhD University of Chicago, Chemical Physics NSF Fellow in Theoretical Chemistry, UIUC Over 15 years experience in applied Machine Learning and AI ML algos for: Aardvark, acquired by Google (2010) Demand Media (eHow); fi rst $1B IPO since Google Wall Street: Barclays, BlackRock Fortune 500: Roche, France Telecom,Walmart BigTech: eBay, Aardvark (Google), GoDaddy Private Equity: Grif fi n Advisors Alt. Energy: Anthropocene Institute (Page Family) www.calculationconsulting.com charles@calculationconsulting.com (TM) 3
- 4. c|c (TM) (TM) 4 calculation | consulting why deep learning works weight|watcher
- 5. c|c (TM) (TM) 5 calculation | consulting why deep learning works Understanding deep learning requires rethinking generalization Motivations: WeightWatcher Theory The weightwatcher theory is a quant/physics-based approach based on: the Statistical Mechanics of Generalization, Random MatrixTheory, & the theory of Strongly Correlated Systems Motivated by the Self Organized Criticality / Critical Brain Hypothesis
- 6. c|c (TM) Research: Implicit Self-Regularization in Deep Learning (TM) 6 calculation | consulting why deep learning works • Predicting trends in the quality of state-of-the-art neural networks without access to training or testing data (JMLR 2021) (ICML 2019, SDM 2020) (KDD 2023) (Nature Communications 2021) Selected publications (with UC Berkeley) … • Implicit Self-Regularization in Deep Neural Networks: Evidence from Random Matrix Theory and Implications for Learning.
- 7. c|c (TM) WeightWatcher: analyzes the ESD (eigenvalues) of the layer weight matrices (TM) 7 calculation | consulting why deep learning works The tail of the ESD contains the information
- 8. c|c (TM) WeightWatcher: analyzes the ESD (eigenvalues) of the layer weight matrices (TM) 8 calculation | consulting why deep learning works Well trained layers are heavy-tailed and well shaped GPT-2 Fits a Power Law (or Truncated Power Law) alpha in [2, 6] watcher.analyze(plot=True) Good quality of fi t (D is small)
- 9. c|c (TM) WeightWatcher: analyzes the ESD (eigenvalues) of the layer weight matrices (TM) 9 calculation | consulting why deep learning works Better trained layers are more heavy-tailed and better shaped GPT GPT-2
- 10. c|c (TM) (TM) 10 calculation | consulting why deep learning works Random Matrix Theory: Marcenko Pastur plus Tracy-Widom fl uctuations very crisp edges Q RMT says if W is a simple random Gaussian matrix, then the ESD will have a very simple , known form Shape depends on Q=N/M (and variance ~ 1) Eigenvalues tightly bounded a few spikes may appear
- 11. c|c (TM) (TM) 11 calculation | consulting why deep learning works RMT: AlexNet Marchenko-Pastur Bulk-decay | Heavy Tailed FC1 zoomed in FC2 zoomed in
- 12. c|c (TM) (TM) 12 calculation | consulting why deep learning works Random Matrix Theory: Heavy Tailed But if W is heavy tailed, the ESD will also have heavy tails (i.e. its all spikes, bulk vanishes) If W is strongly correlated , then the ESD can be modeled as if W is drawn from a heavy tailed distribution Nearly all pre-trained DNNs display heavy tails…as shall soon see
- 13. c|c (TM) (TM) 13 calculation | consulting why deep learning works Experiments: just apply to pre-trained Models LeNet5 (1998) AlexNet (2012) InceptionV3 (2014) ResNet (2015) … DenseNet201 (2018) Conv2D MaxPool Conv2D MaxPool FC FC
- 14. c|c (TM) (TM) 14 calculation | consulting why deep learning works AlexNet, VGG11,VGG13, … ResNet, … Inception, DenseNet, BERT, RoBERT, … GPT, GPT2, … … Heavy-Tailed: Self-Regularization All large, well trained, modern DNNs exhibit heavy tailed self-regularization scale free HTSR
- 15. c|c (TM) (TM) 15 calculation | consulting why deep learning works Heavy Tailed Metrics: GPT vs GPT2 The original GPT is poorly trained on purpose; GPT2 is well trained alpha for every layer smaller alpha is better large alpha are bad fi ts
- 16. c|c (TM) (TM) 16 calculation | consulting why deep learning works Power Law Universality: ImageNet All ImageNet models display remarkable Heavy Tailed Universality 500 matrices ~50 architectures Linear layers & Conv2D feature maps 80-90% < 4
- 17. c|c (TM) (TM) 17 calculation | consulting why deep learning works Random Matrix Theory: detailed insight into WL DNN training induces breakdown of Gaussian random structure and the onset of a new kind of heavy tailed self-regularization Gaussian random matrix Bulk+ Spikes Heavy Tailed Small, older NNs Large, modern DNNs and/or Small batch sizes
- 18. c|c (TM) (TM) 18 calculation | consulting why deep learning works HT-SR Theory: 5+1 Phases of Training Implicit Self-Regularization in Deep Neural Networks: Evidence from Random Matrix Theory and Implications for Learning Charles H. Martin, Michael W. Mahoney; JMLR 22(165):1−73, 2021.
- 19. c|c (TM) (TM) 19 calculation | consulting why deep learning works Heavy Tailed RMT: Universality Classes The familiar Wigner/MP Gaussian class is not the only Universality class in RMT Charles H. Martin, Michael W. Mahoney; JMLR 22(165):1−73, 2021.
- 20. c|c (TM) WeightWatcher: predict trends in generalization (TM) 20 calculation | consulting why deep learning works Predict test accuracies across variations in hyper-parameters The average Power Law exponent alpha predicts generalization—at fi xed depth Smaller average-alpha is better Better models are easier to treat Charles H. Martin, Michael W. Mahoney; [Contest post-mortem paper]
- 21. c|c (TM) WeightWatcher: Shape vs Scale metrics (TM) 21 calculation | consulting why deep learning works Purely norm-based (scale) metrics (from SLT) can be correlated with depth but anti-correlated with hyper-parameter changes
- 22. c|c (TM) WeightWatcher: treat architecture changes (TM) 22 calculation | consulting why deep learning works Predict test accuracies across variations in hyper-parameters and depth The alpha-hat metric combines shape and scale metrics and corrects for different depths (grey line) can be derived from theory…
- 23. c|c (TM) WeightWatcher: predict test accuracies (TM) 23 calculation | consulting why deep learning works alpha-hat works for 100s of different CV and NLP models (Nature Communications 2021) We do not have access to The training or test data But we can still predict trends in the generalization
- 24. c|c (TM) WeightWatcher: predict test accuracies (TM) 24 calculation | consulting why deep learning works ResNet, DenseNet, etc. (Nature Communications 2021)
- 25. c|c (TM) (TM) 25 calculation | consulting why deep learning works Predicting test accuracies: 100 pretrained models The heavy tailed (shape) metrics perform best https://github.com/osmr/imgclsmob From an open source sandbox of nearly 500 pretrained CV models (picked >=5 models / regression) (Nature Communications 2021)
- 26. c|c (TM) (TM) 26 calculation | consulting why deep learning works Correlation Flow: CV Models We can study correlation fl ow looking at vs. depth VGG ResNet DenseNet (Nature Communications 2021)
- 27. c|c (TM) (TM) 27 calculation | consulting why deep learning works Comparing Transformers: BERT vs XLNet weightwatcher layer alphas over- fi t under- fi t very heavy-tailed weakly heavy-tailed
- 28. c|c (TM) (TM) 28 calculation | consulting why deep learning works Comparing Transformers: Bloom vs 560m weightwatcher layer alphas Smaller alpha is better
- 29. c|c (TM) (TM) 29 calculation | consulting why deep learning works Comparing LLMs: Falcon vs Llama Falcon Llama weightwatcher layer alphas
- 30. c|c (TM) (TM) 30 calculation | consulting why deep learning works Comparing LLMs: Correlation Flow Falcon Llama
- 31. c|c (TM) (TM) 31 calculation | consulting why deep learning works Fined-Tuned LLMs: Only the deltas Vicuna Dromedary weightwatcher layer alphas
- 32. c|c (TM) (TM) 32 calculation | consulting why deep learning works LLM Base Models: Truthfulness TruthfulQA metric layer-averaged / model alpha
- 33. c|c (TM) WeightWatcher: why Power Law fi ts ? (TM) 33 calculation | consulting why deep learning works Spiking (i.e real) neurons exhibit power law behavior weightwatcher supports several PL fi ts from experimental neuroscience plus totally new shape metrics we have invented (and published)
- 34. c|c (TM) WeightWatcher: why Power Law fi ts ? (TM) 34 calculation | consulting why deep learning works Spiking (i.e real) neurons exhibit (truncated) power law behavior The Critical Brain Hypothesis Evidence of Self-Organized Criticality (SOC) Per Bak (How Nature Works) As neural systems become more complex they exhibit power law behavior and then truncated power law behavior We see exactly this behavior in DNNs and it is predictive of learning capacity
- 35. c|c (TM) WeightWatcher: open-source, open-science (TM) 35 calculation | consulting why deep learning works We are looking for early adopters and collaborators https://github.com/CalculatedContent/WeightWatcher We have a Discord channel to support the tool 100K+ downloads; 1000 stars https://discord.com/invite/uVVsEAcfyF
- 36. c|c (TM) (TM) 36 calculation | consulting why deep learning works Statistical Mechanics derivation of the alpha-hat metric
- 37. c|c (TM) WeightWatcher: predict test accuracies (TM) 37 calculation | consulting why deep learning works alpha-hat works for 100s of different CV and NLP models (Nature Communications 2021) We do not have access to The training or test data But we can still predict trends in the generalization
- 38. c|c (TM) (TM) 38 calculation | consulting why deep learning works Classic Set Up: Student-Teacher model Statistical Mechanics of Learning Engle &Van den Broeck (2001) MultiLayer Feed Forward Network Perceptron
- 39. c|c (TM) (TM) 39 calculation | consulting why deep learning works Classic Set Up: Student-Teacher model Average overlap over random students J Use to compute the typical / average case generalization error
- 40. c|c (TM) (TM) 40 calculation | consulting why deep learning works New Set Up: Matrix-generalized Student-Teacher real DNN matrices: NxM Strongly correlated Heavy-Tailed correlation matrices Solve for Free Energy associated with the generalization error This Free Energy is the weightwatcher layer quality metric
- 41. c|c (TM) (TM) 41 calculation | consulting why deep learning works Layer Quality Metrics : SemiEmpirical Theory “Generalized Norm” simple, functional form can infer from empirical fi t Eigenvalues of Teacher empirical fi t to: “Asymptotics of HCZI integrals …” Tanaka (2008) WeightWatcher PowerLaw metric
- 42. c|c (TM) WeightWatcher: global and local convexity metrics (TM) 42 calculation | consulting why deep learning works Smaller alpha corresponds to more convex energy landscapes Transformers (alpha ~ 3-4 or more) alpha 2-3 (or less) Rational Decisions, Random Matrices and Spin Glasses" (1998) by Galluccio, Bouchaud, and Potters:
- 43. c|c (TM) WeightWatcher: global and local convexity metrics (TM) 43 calculation | consulting why deep learning works When the layer alpha < 2, we think this means the layer is over fi t We suspect that the early layers of some Convolutional Nets may be slightly overtrained Some alpha < 2 This is predicted from our HTSR theory