WeightWatcher Introduction
1 like672 views
WeightWatcher is an open-source diagnostic tool designed for analyzing deep neural networks without requiring access to training or test data. It utilizes theoretical research to examine the correlations in layer weight matrices, providing insights into model performance and potential issues during training and compression. The tool can predict test accuracies and detect problems using various metrics, making it a valuable resource for deep learning practitioners.
WeightWatcher Introduction
- 1. WeightWatcher: A Diagnostic Tool for Deep Neural Networks Charles H. Martin PhD, Calculation Consulting pip install weightwatcher Martin WeightWatcher March 2021 1 / 9
- 2. Open source tool: weightwatcher https://github.com/CalculatedContent/WeightWatcher WeightWatcher (WW): is an open-source, diagnostic tool for analyzing Deep Neural Networks (DNN), without needing access to training or even test data. It can be used to: analyze pre/trained pyTorch and keras models inspect models that are difficult to train gauge improvements in model performance predict test accuracies across different models detect potential problems when compressing or fine-tuning pretrained models It is based on theoretical research (done injoint with UC Berkeley) into Why Deep Learning Works, using ideas from Random Matrix Theory (RMT), Statistical Mechanics, and Strongly Correlated Systems. pip install weightwatcher Martin WeightWatcher March 2021 2 / 9
- 3. Shape and Scale Metrics WeightWatcher (WW): analyzes the shape and scale of the correlations in the layer weight matrices: WW: extracts, plots, and fits the Empirical Spectral Density (ESD, or eigenvalues) for each layer weight matrix (or tensor slice). The tail of the ESD contains the most informative components. The shape of the tail carries useful information! Martin WeightWatcher March 2021 3 / 9
- 4. WeightWatcher: Usage Usage import weightwatcher as ww watcher = ww.WeightWatcher(model=model) details = watcher.analyze(plot=True) summary = watcher.get_summary(details) summary = watcher.get_summary(details) Martin WeightWatcher March 2021 4 / 9
- 5. Layer-by-Layer Analysis WW layer metrics: can detect potential problems in the ESD shapes Poorly trained models (orange) can have unusually large layer α’s. Martin WeightWatcher March 2021 5 / 9
- 6. Layer-by-Layer Analysis WW layer metrics: can detect potential problems in the ESD Scales Compressed models (red) can show unexpected scale changes example from Intel distiller Group Regularization technique Martin WeightWatcher March 2021 6 / 9
- 7. α: a regularization metric The WW hαi metric: predicts test accuracy for a given model (i.e same depth) when varying the regularization hyper-parameters (such as batch size, weight decay, momentum, etc.)—without access to the test or training data. Martin WeightWatcher March 2021 7 / 9
- 8. α̂: a multi-purpose metric The WW α̂ metric: predicts test accuracy for models in the same architecture series across varying depth and other architecture parameters and regularization hyper-parameters—-without access to the test or training data. Martin WeightWatcher March 2021 8 / 9