NIPS 2016 Overview and Deep Learning Topics
28 likes14,661 views
The document provides an overview of the NIPS 2016 conference, detailing its agenda, topics like deep learning, generative adversarial networks (GANs), and recurrent neural networks (RNNs). It highlights the increase in participation and key features such as acceptance rates and types of presentations. Additionally, it discusses various recent developments and research presented at the conference in deep learning and GANs.
NIPS 2016 Overview and Deep Learning Topics
- 1. Copyright (C) 2016 DeNA Co.,Ltd. All Rights Reserved. NIPS 2016 読み会 @Preferred Networks 2017/1/19 NIPS 2016 Overview and Deep Learning Topics @hamadakoichi 濱田晃一 Copyright (C) 2016 DeNA Co.,Ltd. All Rights Reserved.
- 2. Copyright (C) 2016 DeNA Co.,Ltd. All Rights Reserved. 2 AGENDA ◆Deep Learning Topics ◆NIPS 2016 Overview ◆Generative Adversarial Networks(GANs) ◆Recurrent Neural Networks(RNNs) ◆GANs ◆GANs in NIPS2016 ◆Recent GANs ◆RNNs in NIPS2016
- 3. 3 Copyright (C) 2014 DeNA Co.,Ltd. All Rights Reserved. 講師 ・TokyoWebmining 主催者 - 機械学習の実活用コミュニティ。登録人数 1500人超。 - 7年継続、累積59回開催 濱田晃一 (@hamadakoichi) ・執筆：Mobageを支える技術 Analytics Architect ・博士 : 量子統計場の理論 (理論物理) ・DeNA全サービスを対象とし、大規模機械学習活用したサービス開発 - 数千万ユーザー、50億アクション/日、テキスト、画像、ソーシャルグラフ - 体験設計から、分散学習アルゴリズムの設計・実装まで ・Deep Learning - 画像表現学習・画像生成 対話・キャラクター表現学習、等
- 4. 4 Copyright (C) 2014 DeNA Co.,Ltd. All Rights Reserved. AGENDA ◆Deep Learning Topics ◆NIPS 2016 Overview ◆Generative Adversarial Networks(GANs) ◆Recurrent Neural Networks(RNNs) ◆GANs ◆GANs in NIPS2016 ◆Recent GANs ◆RNNs in NIPS2016
- 5. 5 Copyright (C) 2014 DeNA Co.,Ltd. All Rights Reserved. AGENDA ◆Deep Learning Topics ◆NIPS 2016 Overview ◆Generative Adversarial Networks(GANs) ◆Recurrent Neural Networks(RNNs) ◆GANs ◆GANs in NIPS2016 ◆Recent GANs ◆RNNs in NIPS2016
- 6. Copyright (C) 2016 DeNA Co.,Ltd. All Rights Reserved. 6 NIPS 2016 ・第30回の開催 ・期間： 2016年12月5-10日 ・ICML 33回に続き長い伝統 ・チュートリアル: 5(1日) ・本会議: 5-8(4日) ・ワークショップ: 9-10(2日) ・開催地： バルセロナ(スペイン) 貼る：会場雰囲気
- 7. Copyright (C) 2016 DeNA Co.,Ltd. All Rights Reserved. 7 NIPS 2016 参加者が 6000人に増加 (2015年の1.5倍) ※Terrence Sejnowskiは NIPS foundationの President
- 8. Copyright (C) 2016 DeNA Co.,Ltd. All Rights Reserved. 8 NIPS Features ・採択の92%はポスター ・採択率: 23% ・投稿数: 2500+、採択数： 568 ・Oral(45) : 20分の口頭発表 + ポスター ・Poster(523) ： ポスターのみ ・少数トラックでの進行(最大3) （昨年までシングルトラックだったがパラレルに）
- 9. Copyright (C) 2016 DeNA Co.,Ltd. All Rights Reserved. 9 NIPS Features ・ポスター発表による活発な議論 (昨年までの19-24時の5時間ポスターからは時間縮小したが、最後まで活発な議論） ・210 min(3.5 hour)/ day ・130 Poster x 4 days
- 10. Copyright (C) 2016 DeNA Co.,Ltd. All Rights Reserved. 10 NIPS2016 Hot Topics 引用元： The review process for NIPS 2016 http://www.tml.cs.uni-tuebingen.de/team/ luxburg/misc/nips2016/index.php Deep Learning Computer Vision Large Scale Learning Learning Theory Optimization Sparsity
- 11. Copyright (C) 2016 DeNA Co.,Ltd. All Rights Reserved. 11 NIPS2016 Hot Topics Tutorial 3/9、Symposium 2/3 が Deep Learning Reinforcement Learning, Generative Adversarial Net, Recurrent Net Tutorial Symposium
- 12. Copyright (C) 2016 DeNA Co.,Ltd. All Rights Reserved. 12 NIPS2016 Hot Topics Tutorial Symposium Tutorial 3/9、Symposium 2/3 が Deep Learning Reinforcement Learning, Generative Adversarial Net, Recurrent Net 上記2トピックに関し、本会議論文をピックアップし概要紹介します (Reinforcement Learningは、このNIPS読み会での個別論文の発表も多いため)
- 13. Copyright (C) 2016 DeNA Co.,Ltd. All Rights Reserved. 13 AGENDA ◆Deep Learning Topics ◆NIPS 2016 Overview ◆Generative Adversarial Networks(GANs) ◆Recurrent Neural Networks(RNNs) ◆GANs ◆GANs in NIPS2016 ◆Recent GANs ◆RNNs in NIPS2016
- 14. Copyright (C) 2016 DeNA Co.,Ltd. All Rights Reserved. 14 AGENDA ◆Deep Learning Topics ◆NIPS 2016 Overview ◆Generative Adversarial Networks(GANs) ◆Recurrent Neural Networks(RNNs) ◆GANs ◆GANs in NIPS2016 ◆Recent GANs ◆RNNs in NIPS2016
- 15. Copyright (C) 2016 DeNA Co.,Ltd. All Rights Reserved. 15 AGENDA ◆Deep Learning Topics ◆NIPS 2016 Overview ◆Generative Adversarial Networks(GANs) ◆Recurrent Neural Networks(RNNs) ◆GANs ◆GANs in NIPS2016 ◆Recent GANs ◆RNNs in NIPS2016
- 16. Copyright (C) 2016 DeNA Co.,Ltd. All Rights Reserved. 16 Generative Adversarial Network (GAN) Generative Adversarial Nets(GAN) Goodfellow+, NIPS2014
- 17. Copyright (C) 2016 DeNA Co.,Ltd. All Rights Reserved. 17 Generative Adversarial Network (GAN) Generator(生成器)と Discriminator(識別器)を戦わせ 生成精度を向上させる 識別器: “本物画像”と “生成器が作った偽画像”を識別する 生成器: 生成画像を識別器に“本物画像”と誤識別させようとする (Goodfellow+, NIPS2014, Deep Learning Workshop, Presentation)
- 18. Copyright (C) 2016 DeNA Co.,Ltd. All Rights Reserved. 18 Generative Adversarial Network (GAN) Minimax Objective function Discriminator が 「本物画像」を「本物」と識別 (Goodfellow+, NIPS2014, Deep Learning Workshop, Presentation) Discriminator が 「生成画像」を「偽物」と識別する Discriminatorは 正しく識別しようとする （最大化） Generatorは Discriminator に誤識別させようとする（最小化） Generator(生成器)と Discriminator(識別器)を戦わせ 生成精度を向上させる
- 19. Copyright (C) 2016 DeNA Co.,Ltd. All Rights Reserved. 19 自然画像の表現ベクトル空間学習・演算・画像生成 ICLR16: Deep Convolutional GAN : DCGAN (Radford+) 自然画像のクリアな画像生成 画像演算 Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks. Alec Radford, Luke Metz, Soumith Chintala. arXiv:1511.06434. In ICLR 2016.
- 20. Copyright (C) 2016 DeNA Co.,Ltd. All Rights Reserved. 20 ICML16: Autoencoding beyond pixels (Larsen+) Autoencoding beyond pixels using a learned similarity metric. Anders Boesen Lindbo Larsen, Søren Kaae Sønderby, Hugo Larochelle, Ole Winther. arXiv:1512.09300. In ICML 2016. 自然画像の表現ベクトル空間学習・演算・画像生成
- 21. Copyright (C) 2016 DeNA Co.,Ltd. All Rights Reserved. 21 ICML16: Generative Adversarial Text to Image Synthesis(Reed+) Generative Adversarial Text to Image Synthesis. Scott Reed, Zeynep Akata, Xinchen Yan, Lajanugen Logeswaran, Bernt Schiele, Honglak Lee. arXiv:1605.05396. In ICML 2016. 文章からの画像生成 文章で条件付したGAN
- 22. Copyright (C) 2016 DeNA Co.,Ltd. All Rights Reserved. 22 AGENDA ◆Deep Learning Topics ◆NIPS 2016 Overview ◆Generative Adversarial Networks(GANs) ◆Recurrent Neural Networks(RNNs) ◆GANs ◆GANs in NIPS2016 ◆Recent GANs ◆RNNs in NIPS2016
- 23. Copyright (C) 2016 DeNA Co.,Ltd. All Rights Reserved. 23 Generative Adversarial Text to Image Synthesis(Reed+) Learning What and Where to Draw. Scott Reed, Zeynep Akata, Santosh Mohan, Samuel Tenka, Bernt Schiele, Honglak Lee. arXiv:1610.02454. In NIPS 2016. 文章からの画像生成 表示位置情報も条件付したGAN
- 24. Copyright (C) 2016 DeNA Co.,Ltd. All Rights Reserved. 24 InfoGAN (Chen+) InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets. Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, Pieter Abbeel. arXiv:1606.03657. In NIPS 2016 Latent code c、Generator 出力との Mutual Information を加え GANで狙って表現ベクトル空間を学習
- 25. Copyright (C) 2016 DeNA Co.,Ltd. All Rights Reserved. 25 3Dモデルの表現ベクトル空間学習・演算・生成 3D GAN (Wu+) 3Dモデルの生成 3Dモデル演算 写真からの3Dモデル生成 3D VAE-GAN 3D GAN Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling. Jiajun Wu, Chengkai Zhang, Tianfan Xue, William T. Freeman, Joshua B. Tenenbaum. arXiv:1610.07584. In NIPS 2016.
- 26. Copyright (C) 2016 DeNA Co.,Ltd. All Rights Reserved. 26 Generating Videos with Scene Dynamics(Vondrick+) 動画の表現ベクトル空間学習・動画生成 Generating Videos with Scene Dynamics. Carl Vondrick, Hamed Pirsiavash, Antonio Torralba. In NIPS 2016. http://web.mit.edu/vondrick/tinyvideo/ 動画生成 1画像からその後の動画生成
- 27. Copyright (C) 2016 DeNA Co.,Ltd. All Rights Reserved. 27 f-GAN (Nowozin+) GAN目的関数を Symmetric JS-divergence から f-divergence に一般化。各Divergence を用い学習・評価 f-GAN: Training Generative Neural samplers using variational Divergence Minimization. Sebastian Nowozin, Botond Cseke, Ryota Tomioka. arXiv:1606.00709. In NIPS 2016. Kernel Density Estimation on the MNIST f-divergence LSUN
- 28. Copyright (C) 2016 DeNA Co.,Ltd. All Rights Reserved. 28 Improved Techniques for Training GANs (Salimans+) Improved Techniques for Training GANs. Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, Xi Chen. arXiv:1606.03498. In NIPS 2016. 収束が難しいGANの学習方法論 GAN半教師あり学習 1. Feature Matching 2. Minibatch discrimination 3. Historical averaging 4. One-sided label smoothing 5. Virtual batch normalization Techniques Semi-supervised learning MNIST Semi-supervised training with feature matching Semi-supervised training with feature matching and minibatch discrimination CIFAR-10 Generated samples
- 29. Copyright (C) 2016 DeNA Co.,Ltd. All Rights Reserved. 29 AGENDA ◆Deep Learning Topics ◆NIPS 2016 Overview ◆Generative Adversarial Networks(GANs) ◆Recurrent Neural Networks(RNNs) ◆GANs ◆GANs in NIPS2016 ◆Recent GANs ◆RNNs in NIPS2016
- 30. Copyright (C) 2016 DeNA Co.,Ltd. All Rights Reserved. 30 Extended Architectures for Generative Adversarial Nets 2016 Extended Architectures for GANs Figure by Chris Olah (2016) : https://twitter.com/ch402/status/793535193835417601 Ex) Conditional Image Synthesis With Auxiliary Classifier GANs. Augustus Odena, Christopher Olah, Jonathon Shlens. arXiv:1610.09585. Generative Adversarial Net の各種拡張
- 31. Copyright (C) 2016 DeNA Co.,Ltd. All Rights Reserved. 31 Stack GAN: Text to PhotoRealistic Image Synthesis(Zhang+2016) 1段目で文章から低解像度画像を生成 2段目で低解像度画像から高解像度画像を生成 StackGAN: Text to Photo-realistic Image Synthesis with Stacked Generative Adversarial Networks. Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaolei Huang, Xiaogang Wang, Dimitris Metaxas. arXiv:1612.03242.
- 32. Copyright (C) 2016 DeNA Co.,Ltd. All Rights Reserved. 32 Plug & Play Generative Networks (Nguyen+2016) 高解像度な画像生成 227 x 227 ImageNet Plug & Play Generative Networks: Conditional Iterative Generation of Images in Latent Space. Anh Nguyen, Jason Yosinski, Yoshua Bengio, Alexey Dosovitskiy, Jeff Clune. arXiv:1612.00005.
- 33. Copyright (C) 2016 DeNA Co.,Ltd. All Rights Reserved. 33 AGENDA ◆Deep Learning Topics ◆NIPS 2016 Overview ◆Generative Adversarial Networks(GANs) ◆Recurrent Neural Networks(RNNs) ◆GANs ◆GANs in NIPS2016 ◆Recent GANs ◆RNNs in NIPS2016
- 34. Copyright (C) 2016 DeNA Co.,Ltd. All Rights Reserved. 34 Phased LSTM (Neil+) 時間で開閉するGateを導入した LSTM Sensor Data 等、Event 駆動の長期系列特徴を学習 Phased LSTM: Accelerating Recurrent Network Training for Long or Event-based Sequences. Daniel Neil, Michael Pfeiffer, Shih-Chii Liu. arXiv:1610.09513. In NIPS 2016. LSTM Phased LSTM Phased LSTM Behavior Frequency Discrimination Task
- 35. Copyright (C) 2016 DeNA Co.,Ltd. All Rights Reserved. 35 Using Fast Weights to Attend to the Recent Past (Ba+) 早く学習・減衰する Fast Weight 追加で、系列固有の情報を扱う Slow Weight での長期特徴とあわせ、双方の系列特徴を学習 Using Fast Weights to Attend to the Recent Past. Jimmy Ba, Geoffrey Hinton, Volodymyr Mnih, Joel Z. Leibo, Catalin Ionescu. arXiv:1610.06258. In NIPS 2016. Associative Retrieval Task Classification Error Test Log Likelihood
- 36. Copyright (C) 2016 DeNA Co.,Ltd. All Rights Reserved. 36 Learning to learn by GD by GD (Andrychowicz+) LSTMを用いたOptimizer Parameterごとに 勾配系列から適切な次の更新量を算出 Learning to learn by gradient descent by gradient descent. Marcin Andrychowicz, Misha Denil, Sergio Gomez, Matthew W. Hoffman, David Pfau, Tom Schaul, Brendan Shillingford, Nando de Freitas. arXiv:1606.04474. In NIPS 2016.
- 37. Copyright (C) 2016 DeNA Co.,Ltd. All Rights Reserved. 37 Matching Network for One Shot Learning (Vinyals+) Attention Mechanism を用いた One Shot Learning 参照構造を学習しておき、新規小規模データセットでも高精度で動作 Matching Networks for One Shot Learning. Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Koray Kavukcuoglu, Daan Wierstra. arXiv:1606.04080. In NIPS 2016. Omniglot miniImageNet
- 38. Copyright (C) 2016 DeNA Co.,Ltd. All Rights Reserved. 38 AGENDA ◆Deep Learning Topics ◆NIPS 2016 Overview ◆Generative Adversarial Networks(GANs) ◆Recurrent Neural Networks(RNNs) ◆GANs ◆GANs in NIPS2016 ◆Recent GANs ◆RNNs in NIPS2016