Beyond TensorBoard: AutoML을 위한 interactive visual analytics 서비스 개발 경험 공유
3 likes446 views
HyperTendril is a visual analytics system for interactive hyperparameter tuning of deep neural networks. It addresses challenges of visualizing large numbers of models from autoML by providing an overview and enabling switching to detailed analysis views. The system aims to support the open-ended tuning task through human-in-the-loop interaction, with the goal of refining models based on insights gained from visual exploration of results. User studies found different interaction patterns depending on user roles like fine-tuner or research-oriented tuner, suggesting the need for an extensible design. Future work includes supporting multi-metric model comparison and neural architecture search.
![References
HyperTendril
- Knowledge Generation Models for Visual Analytics [1]
- Visual Analytics in Deep Learning [2]
- Tensorboard graph visualizer [3]
- CNNVis [4]
- GanLab [5]
- RNNVis [6]
- LSTMVis [7]
- ActiVis [8]
- D3.js [9]
- Reservoir sampling [10]
28
[1] Sacha, Dominik, et al. "Knowledge generation model for visual analytics." IEEE transactions on visualization and computer graphics 20.12 (2014): 1604-1613.
[2] Hohman, Fred Matthew, et al. "Visual analytics in deep learning: An interrogative survey for the next frontiers."
IEEE transactions on visualization and computer graphics (2018).
[3] Girija, Sanjay Surendranath. "Tensorflow: Large-scale machine learning on heterogeneous distributed systems." Software available from tensorflow. org (2016).
[4] Liu, Mengchen, et al. "Towards better analysis of deep convolutional neural networks." IEEE transactions on visualization and computer graphics 23.1 (2017): 91-100.
[5] Kahng, Minsuk, et al. "GAN Lab: Understanding Complex Deep Generative Models using Interactive Visual Experimentation.”
IEEE transactions on visualization and computer graphics 25.1 (2019): 310-320.
[6] Karpathy, Andrej, Justin Johnson, and Li Fei-Fei. "Visualizing and understanding recurrent networks." arXiv preprint arXiv:1506.02078 (2015).
[7] Strobelt, Hendrik, et al. "Lstmvis: A tool for visual analysis of hidden state dynamics in recurrent neural networks."
IEEE transactions on visualization and computer graphics 24.1 (2018): 667-676.
[8] Kahng, Minsuk, et al. "Activis: Visual exploration of industry-scale deep neural network models."
IEEE transactions on visualization and computer graphics 24.1 (2018): 88-97.
[9] D3.js https://d3js.org/
[10] J. S. Vitter. Random sampling with a reservoir. ACM Transactions on Mathematical Software (TOMS), 11(1):37–57, 1985.](https://image.slidesharecdn.com/fedevtalkvisualanalyticshsplatest-190528011711/85/Beyond-TensorBoard-AutoML-interactive-visual-analytics-27-320.jpg)
Beyond TensorBoard: AutoML을 위한 interactive visual analytics 서비스 개발 경험 공유
- 1. Beyond TensorBoard AutoML을 위한 interactive visual analytics 서비스 개발 경험 공유 Clova AI 박 흥 석 1
- 2. Contents Introduction Introduction to Visual Analytics & AutoML Problem in Context Design Goals HyperTendril: Visual Analytics for AutoML Discussion & Conclusion 2
- 3. Visual Analytics? Introduction What is visual analytics? 3
- 4. Visual Analytics? Introduction 4 Data Visualization Interpretation Human Typical (interactive) visualization - focuses on visualizing given data as it is. - but, big data cannot be typically visualized due to the limited screen space and human perception (for large number of data items and features with lots of noise, …). Interaction
- 5. Visual Analytics? Introduction 5 Data Model Visualization Human Computer interpretation feedback Visual Analytics - focuses on real-world data analysis tasks. - machine learning/data supportingmining techniques are employed to extract meaningful information from data. - “Computers are incredibly fast, accurate, and stupid, Human beings are incredibly slow, inaccurate, and brilliant. Together they are powerful beyond imagination”, Albert Einstein.
- 6. Visual Analytics for Deep Learning Introduction Visual analytics for deep learning? 6
- 7. Visual Analytics for Deep Learning Introduction Many visual analytics for Deep learning 7 https://medium.com/multiple-views-visualization-research-explained/visualization-in-deep-learning-b29f0ec4f136
- 8. Visual Analytics in DL: for interpretability, diagnosis, refinement of models Introduction 8 GAN LabTensorBoard CNNVis RNNVis LSTMVis ActiVis
- 9. 데이터 전처리 Feature 엔지니어링 모델 아키텍처 선택 모델 파라미터 선택 모델 평가 서비스 배포 Properly handling: § Imbalanced data § Outliers § Missing values § High cardinality features § Highly correlated features § Target leakage § Inconsistent feature definition § Data that doesn’t fit local memory Selecting right preprocessing for: § Numbers § Classes § Dates § Lists § Nested fields Multiple options per column, 100s of columns in table Selecting the best model architecture from dozens available § Linear § Feed forward § Decision tree § Residual nets Keeping up with the onslaught of newest state of the art For each architecture, selecting the right values for each hyperparameter § Learning rate § Regularization § Layers § Hidden nodes § Activation fxn Potentially more than a dozen values to set Evaluating model at § Dataset-level § Feature-level § Prediction-level Ensuring behavior is fully understood before deployment Formalized workflow of deep learning model development Introduction Deploying service 9
- 10. Problems in developing deep learning model Introduction 데이터 전처리 Feature 엔지니어링 모델 아키텍처 선택 모델 파라미터 선택 모델 평가 서비스 배포 10 Tedious episodes of trial and error!
- 11. Problems in developing deep learning model => AutoML as a solution Introduction Black-box optimization Number of hyperparameter combination is infinite! A number of computation, time, and human resources are needed :( 11 데이터 전처리 Feature 엔지니어링 모델 아키텍처 선택 모델 파라미터 선택 모델 평가 서비스 배포 Þ AutoML: Automatic & Systematic approach
- 12. Problems in developing deep learning model => AutoML as a solution Introduction 12 NSML AutoML Black-box optimization 데이터 전처리 Feature 엔지니어링 모델 아키텍처 선택 모델 파라미터 선택 모델 평가 서비스 배포
- 13. Visual Analytics for AutoML Problem in context 어쨌건 visual analytics를 이용해서 모델 분석도 하고, 해석도 어느 정도 가능한 것 같긴한데.. AutoML은 보통 수백, 수천 개의 모델을 만드는데 이를 어떻게 표현하고, 분석할 수 있지? 13
- 14. Visual Analytics for AutoML Problem in context No visual analytics system for AutoML 14
- 15. Challenges - Too many models to be shown (n > 100, 1000, 10000, ...) - High dimensionality and complexity of hyperparameter space 15 DL models Batch_size Learning_rate Num_epoch Layer_depth Activation _function … Test/acc. Model 0 100 0.001 74 3 relu … 0.9231 Model 1 100 0.001231 68 6 sigmoid … 0.8951 Model 2 1000 0.00125 48 9 tanh … 0.5789 Model 3 500 0.00534 24 128 relu … 0.9483 Model 4 500 0.01541 24 128 sigmoid … 0.832 Model 5 500 0.05929 24 32 Tanh … 0.748 … … … … … … … … Example results of hyperparameter optimization Problem in context Hyperparameter configuration space
- 16. - 수만 개의 모델을 만들어도 최적의 모델이라는 보장은 없음 (infinite search space) - 매번 수만, 수십만 개의 모델을 만들 수 없음 => 모든 space를 탐색할 시간이 한정적이고, GPU도 한정적 => 몇백 개의 탐색 결과를 보고, 결과를 기반으로 또 다시 몇백 개의 탐색을 반복 - AutoML algorithm에 대한 configuration도 정답이 없음 => 실행 결과를 보고 진단하고 바로 잡는 등의 과정 Þ 최적화는 한 번의 trial로 끝나지 않음 (Open-ended task) More Challenges Problem in context 16
- 17. Design Goals - Show an overview of results with effective visual interfaces - Enable switching to detail analysis view from the overview by coordinated visual components - Steer the open-ended tuning task with human-in-the-loop approach Hyperparameter optimization process through visual analytics 17 Design Goals - How to effectively visualize the result of hyper- parameter optimization? - How can visual analytics support the open-ended hyperparameter optimization task? Design Challenges Design Goals
- 18. Interaction flow design - Overview 보여주고, 여러 관점에서의 분석 환경을 제공해서 결과 분석 할 수 있는 환경 제공 - 결과 분석으로 얻은 insight를 바탕으로 모델의 refinement도 도와주는 환경 제공 18 Design Goals A Visual exploration of overall optimization results Switch overview to details B Hyperparameter-level Model-level Method-level analysis Action C User-driven model refinement - Parallel coordinates plot - effective visualization for high dimensional data - Hyperparameter-level: - to find effective hyperparameter - to find effective range of hyperparameter - Model-level: - to validate model generalizability - to analyze the value of loss function by time/iteration - Method-level: - to validate/diagnose the algorithm configurations - to compare the performance of algorithm - Support interactive tuning process - easy to access autoML system with the gained insights
- 19. D3.js HyperTendril 19 - Domain specific language for data vis. - More flexible to make own visual component than higher-level libraries - A number of examples, documents, tutorials https://d3js.org/
- 20. HyperTendril: Visual Analytics for User-Driven hyperparameter Tuning of Deep Neural Networks HyperTendril 20
- 21. Interaction flow of HyperTendril Visual Analytics HyperTendril 22 A Visual exploration of overall optimization results Switch overview to details B Hyperparameter-level Model-level Method-level analysis Action C User-driven model refinement
- 22. - To understand usage behaviors of visual analytics - Log collection w/ Google analytics - User feedback w/ UX interview - Findings: the usage behaviors and volume of interactions are various by their tasks and purposes. User study HyperTendril Click-stream analysis with representative users and their sessions 23 Interaction patterns are quite different!
- 23. - Behaviors of each user are various, but can be categorized - Fine-tuner - Service-oriented tuner - Research-oriented tuner Þ Should consider an extendable design of visual analytics to satisfy various types of users. Lessons learned Discussion & Conclusion 24 Knowledge generation loop w/ HyperTendril Complexity & volume of interactions
- 24. Conclusion & Future work - Interactive hyperparameter optimization in real-time - Visual analytics for multi-metric based model comparison - including latency, classification performance (e.q., confusion matrix), and etc. - Visual analytics for Neural Architecture Search (NAS) - for automating the design of artificial neural network. 25 Discussion & Conclusion - Defining problem to solve (which task can be supported by visual analytics) is important. - Extendable design should be considered in developing visual analytics. Future work Conclusion
- 25. Thank you :) 26
- 26. Issues in development HyperTendril - Performance Issues - Browser down in drawling line chart with numerous data points => reservoir sampling 27
- 27. References HyperTendril - Knowledge Generation Models for Visual Analytics [1] - Visual Analytics in Deep Learning [2] - Tensorboard graph visualizer [3] - CNNVis [4] - GanLab [5] - RNNVis [6] - LSTMVis [7] - ActiVis [8] - D3.js [9] - Reservoir sampling [10] 28 [1] Sacha, Dominik, et al. "Knowledge generation model for visual analytics." IEEE transactions on visualization and computer graphics 20.12 (2014): 1604-1613. [2] Hohman, Fred Matthew, et al. "Visual analytics in deep learning: An interrogative survey for the next frontiers." IEEE transactions on visualization and computer graphics (2018). [3] Girija, Sanjay Surendranath. "Tensorflow: Large-scale machine learning on heterogeneous distributed systems." Software available from tensorflow. org (2016). [4] Liu, Mengchen, et al. "Towards better analysis of deep convolutional neural networks." IEEE transactions on visualization and computer graphics 23.1 (2017): 91-100. [5] Kahng, Minsuk, et al. "GAN Lab: Understanding Complex Deep Generative Models using Interactive Visual Experimentation.” IEEE transactions on visualization and computer graphics 25.1 (2019): 310-320. [6] Karpathy, Andrej, Justin Johnson, and Li Fei-Fei. "Visualizing and understanding recurrent networks." arXiv preprint arXiv:1506.02078 (2015). [7] Strobelt, Hendrik, et al. "Lstmvis: A tool for visual analysis of hidden state dynamics in recurrent neural networks." IEEE transactions on visualization and computer graphics 24.1 (2018): 667-676. [8] Kahng, Minsuk, et al. "Activis: Visual exploration of industry-scale deep neural network models." IEEE transactions on visualization and computer graphics 24.1 (2018): 88-97. [9] D3.js https://d3js.org/ [10] J. S. Vitter. Random sampling with a reservoir. ACM Transactions on Mathematical Software (TOMS), 11(1):37–57, 1985.