[2A4]DeepLearningAtNAVER
93 likes18,970 views
This document discusses deep learning and its applications. It provides an overview of deep learning, including how it is used for tasks like speech recognition, machine translation, and image classification. It then discusses deep learning applications at NAVER, including using convolutional neural networks for image classification and recurrent neural networks for language modeling. The document also covers important aspects of deep learning like new algorithms, large datasets, and specialized hardware.
![• Weight
matrix 를 두 개의 matrix
곱으로 분리하는 matrix
factorizaNon
• Weight
matrix는 low-‐rank
matrix
Dimension
ReducNon
[7000 ×3000] =[7000 × R][R×3000]](https://image.slidesharecdn.com/2a4deeplearningatnaver-140929210707-phpapp01/85/2A4-DeepLearningAtNAVER-71-320.jpg)
[2A4]DeepLearningAtNAVER
- 1. Deep Learning at NAVER DEVIEW 2014 NAVER LABS 김 정 희 수석연구원 2014. 09. 30
- 2. Contents 2 1. Deep learning overview 2. Deep learning at NAVER 3. Deep learning?
- 3. 1. Deep Learning Overview
- 4. Deep Learning Overview 4 Machine learning with Deep neural networks
- 5. Machine Learning 5 Observable Data Modeling (Learning) Informa?on
- 6. Machine Learning 6 Observable Data Modeling (Learning) Informa?on Speech DEVIEW 2014 Recogni?on Speech Text
- 7. Machine Learning 7 Observable Data Modeling (Learning) Informa?on One Language Another Language Machine I am a boy Transla?on 나는 소년이다
- 8. Machine Learning 8 Observable Data Modeling (Learning) Informa?on Image 고양이 Classify Image Category
- 9. Machine Learning 9 Observable Data Modeling (Learning) Informa?on HOW ?
- 10. Machine Learning 10 Observable Data Modeling (Learning) Informa?on LDA
- 11. Machine Learning 11 Observable Data Modeling (Learning) Informa?on Gaussian Mixture
- 12. Machine Learning 12 Observable Data Modeling (Learning) Informa?on Non-‐nega8ve Matrix Factoriza8on
- 13. Machine Learning 13 Observable Data Modeling (Learning) Informa?on Deep Neural Networks
- 14. Deep Neural Networks -‐ Learning 14 Input Data Weight Matrix Probability of Each Class Back propaga8on 1980년대 확립
- 15. Why Deep Learning? Yoshua Bengio 2014 KDDNN 15 • NAVER Speech recogniNon § 20 % error rate 개선
- 16. Why Deep Learning? Yoshua Bengio 2014 KDDNN 16 • ImageNet ClassificaNon 2012 § Krizhevsky et al (convnet) -‐ 16.4 % error ( top – 5 ) § Next best ( non – convnet ) -‐ 26.2 % error
- 17. Why Deep Learning? Yoshua Bengio 2014 KDDNN 17
- 18. 2. Deep Learning at NAVER
- 19. Rebirth of Deep Learning 19 New Algorithms Big Data Hardware
- 20. Primary Check Points of Deep Learning 20 New Algorithms Big Data Hardware
- 21. ConvoluNonal neural networks 2012 21 FFNN CNN Alex Krizhevsky et al. 2012 NIPSNN
- 22. Image classificaNon (m.ndrive.naver.com) 22 FFNN CNN RNN
- 23. ConvoluNonal neural networks 2014 23 FFNN CNN Chris?an Szegedy et al. 2014 ILSVRCNN
- 24. Deep Neural Networks 24 C_1 C_2 ŸŸŸ C_n FFNN CNN Non Linear Ac?va?on Sigmoid ŸŸŸ ŸŸŸ ŸŸŸ Output Layer Hidden Layer Input Layer
- 25. Back -‐ propagaNon 25 C_1 C_2 ŸŸŸ C_n FFNN CNN Non Linear Ac?va?on Sigmoid ŸŸŸ ŸŸŸ ŸŸŸ Back-‐ propgata?on Error 의 미분값
- 26. Exploding or Vanishing 26 FFNN CNN
- 27. Recurrent neural networks 27 FFNN CNN RNN
- 28. Recurrent Neural Networks 28 • SequenNal data Analysis § Video § RoboNcs § Language model
- 29. Recurrent neural networks 29 FFNN CNN T = R1N0N0 ?
- 30. Recurrent Neural Networks 30 100 개의 Hidden Layer
- 31. Long Short-‐Term Memory 31 FFNN CNN T = R1N0N0 ?
- 32. Language Model 32 • Language model § w(1), w(2), w(3), … , w(n) 의 단어열이 주어졌을 때, § 확률 p( w(1), w(2), w(3), … , w(m) ) 위한 확률 모델
- 33. LM ApplicaNon (1) 33 • Speech recogniNon § 같은 발음 / 다른 철자 o 위성이 지구 ( 궤도 / 괴도 ) 를 돌고 있다. o (궤도 / 괴도) 루팡을 읽었다.
- 34. LM ApplicaNon (2) 34 • Machine translaNon § 같은 철자 / 다른 의미 o The town lies on the coat. o She lies about her age.
- 35. Recurrent Neural Networks -‐ LM 35 • 에측 정확도 향상 § 기존 방법 : KN-‐5 § 실험 Data set : Penn tree bank § 척도 : perplexity § 20% 향상
- 36. Primary Check Points of Deep Learning 36 New Algorithms Big Data Hardware
- 37. Neural Networks 37 • Data-‐driven § 학습 Data 에 기초해 해당 Class 의 확률이 최대가 되도록 § Prior knowledge 를 사용하지 않는다. ! 학습 Data를 그대로 모사하는 것
- 38. Neural Networks 38 • Big data 가 아니면 문제가 생긴다 § Why? § 보유한 학습 Data 가 이 세상의 모든 Data 라고 할 수 있나?
- 39. Before – Big Data 39 Real World Data Small Learning Data
- 40. Before – Big Data 40 Real World Data
- 41. Before – Big Data 41 Real World Data Small Learning Data
- 42. Before – Big Data 42 Real World Data
- 43. Before – Big Data 43 Real World Data
- 44. Before – Big Data 44 Real World Data ERROR !!!
- 45. Human Case 45 MiYe 광고 이미지
- 46. Before – Big Data 46 Real World Data ERROR !!!
- 47. Prior Know-‐ ledge Before – Big Data 47 Real World Data Small Leaning Data ERROR !!!
- 48. GMM Before – Big Data 48 Real World Data Small Speech Data ERROR !!!
- 49. Gabor Filter Before – Big Data 49 Real World Data Small Image Data ERROR !!!
- 50. Image classificaNon (m.ndrive.naver.com) 50 FFNN CNN RNN
- 51. ConvoluNonal neural networks 51 FFNN CNN Alex Krizhevsky et al. 2012 NIPSNN
- 52. 기존 Approach 52 FFNN CNN RNN Image Pixels Hand-‐ designed Feature Extractor Trainable Classifier Object Class
- 53. Deep Learning 53 FFNN CNN RNN Image Pixels Trainable Feature Extractor Trainable Classifier Object Class
- 54. ConvoluNonal neural networks 54 FFNN CNN Hierarchical Trained Filters Alex Krizhevsky et al. 2012 NIPSNN
- 55. RepresentaNon Learning 55 RNN Rob Fergus 2013 NIPSNN
- 56. Prior Know-‐ ledge Afer – Big Data 56 Real World Data Small Leaning Data ERROR !!! Big Learning Data
- 57. Big Data 57 • Neural networks § Supervised learning § 무작정 data 만 많이 있어서는 안된다 § 정답이 있는 data가 많이 있어야 한다
- 58. Big Data 58 • 정답이 있는 DB를 대량으로 구축하기 힘들다면… § Semi-‐supervised learning § Supervised Learning + Unsupervised Learning
- 59. Big Data 59 • Big Data 를 구축하기 힘들다면… § 서비스 개발 진입 장벽
- 60. Big Data 60 • Big Data 를 구축하기 힘들다면… § 일단 만들고 beta 서비스 하자 !
- 61. Transfer Learning 61 • Deep learning ! RepresentaNon learning § 유사 Domain 에서 학습된 내용을 다른 도메인으로 § 한국어 음성인식 ! 일본어 음성인식 § 중국어 OCR ! 일본어 OCR § 이미지 feature extractor
- 62. Transfer Learning – Image 62 FFNN CNN Supervised Trained Feature Extractor Alex Krizhevsky et al. 2012 NIPSNN
- 63. Transfer Learning – Speech RecogniNon 63
- 64. Speech recogniNon 64 FFNN CNN RNN • Transfer learning § 일본어, 영어 인식 엔진 개발 § 소량의 데이터 § 한국어와 유사한 정확도
- 65. Primary Check Points of Deep Learning 65 New Algorithms Big Data Hardware
- 66. 계산량 7000 3000 … 3000 500 5 hidden layers 58.5 M parameters
- 67. Sparseness • 작은 weight 값들 존재
- 68. 계산량 감소 Idea 모든 weight가 필요하지는 않다.
- 69. Dimension ReducNon • PCA와 같은 Dimension ReducNon § Layer의 output을 Feature § Linear TransformaNon을 이용 § 필요없는 Dimension을 줄이는 ReducNon
- 70. 7000 R 3000 Dimension ReducNon • 7000 x 3000 >= 7000 x R + 3000 x R • 2100 >= R
- 71. • Weight matrix 를 두 개의 matrix 곱으로 분리하는 matrix factorizaNon • Weight matrix는 low-‐rank matrix Dimension ReducNon [7000 ×3000] =[7000 × R][R×3000]
- 72. Dimension ReducNon • NAVER § Weight 64% 감소 § 성능 저하 없음
- 73. GPU • OpNmizaNon 의 어려움 § Deep learning 용 GPU library 들 release § Ex) Nvidia -‐ CuNet
- 75. Deep Learning – The One Ring? 75 State of the art by Abstruse Goose
- 76. Deep Learning – The One Ring? 76 State of the art by Abstruse Goose
- 77. Deep Learning – The One Ring? 77 State of the art by Abstruse Goose
- 78. Deep Learning – Current Status 78 Yoshua Bengio KDD 2014
- 79. 79