SlideShare a Scribd company logo
Introduction of ICLR2017
Takeru Miyato
Preferred Networks, Inc.
Takeru Miyato (https://takerum.github.io/)
Researcher @ Preferred Networks, Inc.
● 04/2014-03/2016, M.S. Informatics @ Kyoto Univ.
● 01/2016-05/2016, Intern @ Google Brain
● 06/2016-08/2016, Reearch engineer @ ATR
● 09/2016-(now)
○ Researcher @ Preferred Networks, Inc.
○ Visiting Researcher @ ATR
Current research interests:
● Generative Adversarial Networks (GANs)
● Semi-supervised and Unsupervised Learing on Neural Networks
● Learning on Extremely Large Distributed Systems
https://arxiv.org/abs/1507.00677, https://arxiv.org/abs/1605.07725, https://arxiv.org/abs/1702.08720, https://arxiv.org/abs/1704.03976
https://openreview.net/forum?id=H1hLmF4Fx, https://arxiv.org/abs/1705.10941
(Self introduction)
Contents
● What’s ICLR?
○ Main features of ICLR
● ICLR2017
● GANs
What’s ICLR?
● International Conferene on Learning Representations. Called ‘I CLeaR’.
● 3 days conference, held at the beginning of May.
● ICLR2017 is the 5th-ICLR, the 1st-ICLR was held in 2013
● Website : http://www.iclr.cc/doku.php
○ Facebook page : https://www.facebook.com/iclr.cc/?fref=ts
○ Twitter account (This year) : https://twitter.com/iclr2017?lang=en
● http://www.kdnuggets.com/2016/02/iclr-deep-learning-scientific-publishing-ex
periment.html
Main features of ICLR (1)
● Focus on Deep Learning and its Application
○ Most of the accepted papers concern neural networks.
Variational Auto-Encoder
(ICLR2014)
ADAM optimizer
(ICLR2015)
Main features of ICLR (2)
● Open Review System https://openreview.net/
○ Single blind submission
○ Everyone can see all of the reviews and rebuttals.
○ Everyone can “join” the reviewing process.
Other features
● 2 tracks : Conference track and Workshop track
● Single oral track
● Lots of IT giants sponsors
https://blogs.sap.com/2017/05/22/dive-deep-into-deep-learning-
sap-at-the-international-conference-on-learning-representations
-iclr/
Submissions and Attendees
http://www.iclr.cc/lib/exe/fetch.php?media=iclr201
7:ranzato_introduction_iclr2017.pdf
ICLR2017 http://www.iclr.cc/doku.php?id=ICLR2017:main (Toulon, France)
● Attendees : 1100 (~300 last year)
● Accepted papers
○ Conference track : 196, 15 oral
○ Workshop track : ~110
https://twitter.com/iclr2017/status/864863545867079680
taken by Okumura-san
http://www.iclr.cc/lib/exe/fetch.php?media=iclr201
7:ranzato_introduction_iclr2017.pdf
New
(Last year) 28%
http://www.iclr.cc/lib/exe/fetch.php?media=iclr201
7:ranzato_introduction_iclr2017.pdf
● Google, Facebook,
Microsoft, OpenAI, … etc.
https://prlz77.github.io/iclr2017-stats/
My presentations
● 1 conference track poster and 1 workshop track poster
https://arxiv.org/abs/1605.07725 https://openreview.net/pdf?id=H1hLmF4Fx
http://www.iclr.cc/lib/exe/fetch.php?media=iclr201
7:ranzato_introduction_iclr2017.pdf
Oral sessions:
● Single oral track
● Two sessions on each day : morning and afternoon
○ 1 invited talk and 2~3 oral selected presentations in each session.
● Live streamining: https://www.facebook.com/iclr.cc/?fref=ts
Poster presentations
● Two poster sessions : morning and afternoon
● Conference track
○ ~40 posters at each session
○ 10~20 audiences around each poster
● Workshop track
○ ~20 posters
○ ~5 audiences
● (last year, poster)
○ ~5 audiences
(Taken by Okumura-san)
After hours parties
Individually hosted by Google, Facebook,
Salesforce ...etc.
○ Networking
○ Discussing with researchers and
engineers
■ until late at night!
Major & popular topics at ICLR2017
● Generalization ability of Neural Networks
○ Rethinking generalization https://arxiv.org/abs/1611.03530
○ Large batch training coverges to sharp minima https://arxiv.org/abs/1609.04836
● Generative Adversarial Networks
○ Towards principled methods of training GANs https://openreview.net/pdf?id=Hk4_qw5xe
○ Energy based GANs https://arxiv.org/abs/1702.01691
○ Two-sample tests by classifier https://openreview.net/forum?id=SJkXfE5xx&noteId=SJkXfE5xx
● Deep Reinforcement Learning
○ Reinforcement Learning with Unsupervised Auxiliary Tasks https://openreview.net/pdf?id=SJ6yPD5xg
○ (Not RL) Learning to act by predicting future https://arxiv.org/abs/1611.01779
● Neural Programmer
○ Making Neural Programming Architectures Generalize via Recursion
https://openreview.net/forum?id=BkbY4psgg&noteId=BkbY4psgg
Generative Adversarial Networks (GANs)
● Originally proposed by Ian Goodfellow et al. (2014)
● Quite a lot of researchers have been conducting works on GANs
○ GAN Zoo https://github.com/hindupuravinash/the-gan-zoo
https://github.com/hindupuravinash/the-gan-zoo
(GANs)
GANs frameworks
● Discriminator D(x) : trained to
discriminate between real
(dataset) examples and
generated examples by the
Generator G(z)
● Generator G(z) : trained to fool
the Discriminator D(x).
(GANs)
What is good about GANs?
● We don’t need explicit expression of the denstiy for
the generative models pG
(x)
○ Only requires a stochastic generative
process : x ~ pG
(x)
● The training process can be incorporated into
semi-supervised learning https://arxiv.org/abs/1606.03498
○ Achieved the state of the art performance, especially on a few
labeled semi-supervised dataset.
https://arxiv.org/pdf/1702.08896.pdf
(GANs)
Important GANs works (submitted to ICLR2017)
● b-GAN (uehara et al. https://arxiv.org/pdf/1610.02920.pdf )
○ The discriminator of GANs is the density ratio estimator rD
(x) of
r(x)=q(x) / pG
(x)
○ Directly learn q(x) / pG
(x) by minimizing Bregman Divergence between
rD
(x) and q(x) / pG
(x).
● Implicit Generative Models (Mohamed et al. https://arxiv.org/abs/1610.03483 )
○ Likelihood-free estimation through the GANs algorithm
● Deep and Hireachical Implicit models (Tran et al. https://arxiv.org/pdf/1702.08896.pdf ) (※Not
subimitted to ICLR)
○ Likelihood-free variational inference (LFVI) through the GANs
algorithm
■ it only requires that we can sample from qVI
(x, z) and pModel
(x, z)
(My selection of)
Thanks! I enjoyed water taxi ⚓
(Hotel <-> Conference)

More Related Content

PPTX
Deploying your Predictive Models as a Service via Domino
PDF
Skilling-up-in-research-data-management-20181128
PPTX
Kaggle Competitions, New Friends, New Skills and New Opportunities
PDF
Tds — big science dec 2021
PDF
2015 FOSS4G Track: Spatiotemporal Interface Development: Using Web Technologi...
PPTX
Toulouse Data Science meetup - Apache zeppelin
PDF
2015 GIS in Colorado Track: Challenges Standardizing and Implementing Metadat...
PDF
VALA Tech Camp 2017: Intro to Wikidata & SPARQL
Deploying your Predictive Models as a Service via Domino
Skilling-up-in-research-data-management-20181128
Kaggle Competitions, New Friends, New Skills and New Opportunities
Tds — big science dec 2021
2015 FOSS4G Track: Spatiotemporal Interface Development: Using Web Technologi...
Toulouse Data Science meetup - Apache zeppelin
2015 GIS in Colorado Track: Challenges Standardizing and Implementing Metadat...
VALA Tech Camp 2017: Intro to Wikidata & SPARQL

What's hot (18)

PPTX
Rich Data? Poor Data? Depends on...
PDF
2015 FOSS4G Track: Open Specifications for the Storage, Transport and Process...
PDF
Web Archive Profiling Through Fulltext Search
PDF
Introducing Web Archiving and WSDL Research Group
PDF
A Closer Look at the Changing Dynamics of DBpedia Mappings
PDF
Open Data and Data Journalism in Russia
PPT
Semantic web and Drupal: an introduction
PDF
A Decentralized Approach to Dissemination, Retrieval, and Archiving of Data
PDF
2015 FOSS4G Track: What is Free and Open Source Software for Geospatial Appli...
PDF
2015 FOSS4G Track: Integrating FOSS4G Into a Government Web-Editing Applicati...
PDF
ICT4D course 2013 - Sugar
PDF
2015 FOSS4G Track: Getting Started with FOSS4G: The Newbie’s Perspective by M...
PDF
The drawbridge to knowledge - Linking scholarly publications and research inf...
ODP
Charper.lawdi.20130531
PDF
Collected List of Business Intelligence Software
PDF
Some Russian Open data and Data Journalism projects
PDF
Python in Data Science Work
ODP
OpenRefine - Data Science Training for Librarians
Rich Data? Poor Data? Depends on...
2015 FOSS4G Track: Open Specifications for the Storage, Transport and Process...
Web Archive Profiling Through Fulltext Search
Introducing Web Archiving and WSDL Research Group
A Closer Look at the Changing Dynamics of DBpedia Mappings
Open Data and Data Journalism in Russia
Semantic web and Drupal: an introduction
A Decentralized Approach to Dissemination, Retrieval, and Archiving of Data
2015 FOSS4G Track: What is Free and Open Source Software for Geospatial Appli...
2015 FOSS4G Track: Integrating FOSS4G Into a Government Web-Editing Applicati...
ICT4D course 2013 - Sugar
2015 FOSS4G Track: Getting Started with FOSS4G: The Newbie’s Perspective by M...
The drawbridge to knowledge - Linking scholarly publications and research inf...
Charper.lawdi.20130531
Collected List of Business Intelligence Software
Some Russian Open data and Data Journalism projects
Python in Data Science Work
OpenRefine - Data Science Training for Librarians
Ad

Similar to [ICLR2017 Reading Meeting@DeNA] Introduction of ICLR2017 (20)

PDF
IA377 Seminar FEEC-UNICAMP Intro classpdf
PDF
SWAD Timeline 4:3
PDF
Swad Timeline
PPTX
Linking Software: citations, roles, references and more
PDF
 Towards Reproducible Data Analysis Using Cloud and Container Technologies
PDF
Local Weather Information and GNOME Shell Extension
PDF
Transformers in 2021
PDF
An Empirical Comparison of Knowledge Graph Embeddings for Item Recommendation
PDF
Pathways to Technology Transfer and Adoption: Achievements and Challenges
PDF
Towards Universitas Indonesia Next Generation Firewall Service - Tonny | GNOM...
PDF
Hobbit presentation at Apache Big Data Europe 2016
PDF
VII Jornadas eMadrid "Education in exponential times". "Open Analytics in an ...
PDF
Manila Project Onboarding - Denver Open Infrastructure Summit - May 2019
PPTX
2017 Q2 uPortal Open Source Support Briefing
PDF
SDEC2011 Mahout - the what, the how and the why
PDF
A Plan for Sustainable MIR Evaluation
PDF
Learning
PDF
Critical Thinking Learning MOOC Preparation of
PPTX
T13 (1).pptx
PDF
Overview of Hugging Face platform - 2024-10-24
IA377 Seminar FEEC-UNICAMP Intro classpdf
SWAD Timeline 4:3
Swad Timeline
Linking Software: citations, roles, references and more
 Towards Reproducible Data Analysis Using Cloud and Container Technologies
Local Weather Information and GNOME Shell Extension
Transformers in 2021
An Empirical Comparison of Knowledge Graph Embeddings for Item Recommendation
Pathways to Technology Transfer and Adoption: Achievements and Challenges
Towards Universitas Indonesia Next Generation Firewall Service - Tonny | GNOM...
Hobbit presentation at Apache Big Data Europe 2016
VII Jornadas eMadrid "Education in exponential times". "Open Analytics in an ...
Manila Project Onboarding - Denver Open Infrastructure Summit - May 2019
2017 Q2 uPortal Open Source Support Briefing
SDEC2011 Mahout - the what, the how and the why
A Plan for Sustainable MIR Evaluation
Learning
Critical Thinking Learning MOOC Preparation of
T13 (1).pptx
Overview of Hugging Face platform - 2024-10-24
Ad

Recently uploaded (20)

PPTX
ALIMENTARY AND BILIARY CONDITIONS 3-1.pptx
PDF
“Getting Started with Data Analytics Using R – Concepts, Tools & Case Studies”
PDF
22.Patil - Early prediction of Alzheimer’s disease using convolutional neural...
PPTX
STUDY DESIGN details- Lt Col Maksud (21).pptx
PPTX
Introduction to Knowledge Engineering Part 1
PDF
TRAFFIC-MANAGEMENT-AND-ACCIDENT-INVESTIGATION-WITH-DRIVING-PDF-FILE.pdf
PDF
BF and FI - Blockchain, fintech and Financial Innovation Lesson 2.pdf
PPTX
Business Acumen Training GuidePresentation.pptx
PPTX
advance b rammar.pptxfdgdfgdfsgdfgsdgfdfgdfgsdfgdfgdfg
PPTX
Moving the Public Sector (Government) to a Digital Adoption
PDF
Lecture1 pattern recognition............
PPT
Miokarditis (Inflamasi pada Otot Jantung)
PPTX
Introduction to Basics of Ethical Hacking and Penetration Testing -Unit No. 1...
PPT
Reliability_Chapter_ presentation 1221.5784
PPTX
DISORDERS OF THE LIVER, GALLBLADDER AND PANCREASE (1).pptx
PPTX
IBA_Chapter_11_Slides_Final_Accessible.pptx
PDF
Clinical guidelines as a resource for EBP(1).pdf
PPTX
oil_refinery_comprehensive_20250804084928 (1).pptx
PPTX
CEE 2 REPORT G7.pptxbdbshjdgsgjgsjfiuhsd
PPTX
IB Computer Science - Internal Assessment.pptx
ALIMENTARY AND BILIARY CONDITIONS 3-1.pptx
“Getting Started with Data Analytics Using R – Concepts, Tools & Case Studies”
22.Patil - Early prediction of Alzheimer’s disease using convolutional neural...
STUDY DESIGN details- Lt Col Maksud (21).pptx
Introduction to Knowledge Engineering Part 1
TRAFFIC-MANAGEMENT-AND-ACCIDENT-INVESTIGATION-WITH-DRIVING-PDF-FILE.pdf
BF and FI - Blockchain, fintech and Financial Innovation Lesson 2.pdf
Business Acumen Training GuidePresentation.pptx
advance b rammar.pptxfdgdfgdfsgdfgsdgfdfgdfgsdfgdfgdfg
Moving the Public Sector (Government) to a Digital Adoption
Lecture1 pattern recognition............
Miokarditis (Inflamasi pada Otot Jantung)
Introduction to Basics of Ethical Hacking and Penetration Testing -Unit No. 1...
Reliability_Chapter_ presentation 1221.5784
DISORDERS OF THE LIVER, GALLBLADDER AND PANCREASE (1).pptx
IBA_Chapter_11_Slides_Final_Accessible.pptx
Clinical guidelines as a resource for EBP(1).pdf
oil_refinery_comprehensive_20250804084928 (1).pptx
CEE 2 REPORT G7.pptxbdbshjdgsgjgsjfiuhsd
IB Computer Science - Internal Assessment.pptx

[ICLR2017 Reading Meeting@DeNA] Introduction of ICLR2017

  • 1. Introduction of ICLR2017 Takeru Miyato Preferred Networks, Inc.
  • 2. Takeru Miyato (https://takerum.github.io/) Researcher @ Preferred Networks, Inc. ● 04/2014-03/2016, M.S. Informatics @ Kyoto Univ. ● 01/2016-05/2016, Intern @ Google Brain ● 06/2016-08/2016, Reearch engineer @ ATR ● 09/2016-(now) ○ Researcher @ Preferred Networks, Inc. ○ Visiting Researcher @ ATR Current research interests: ● Generative Adversarial Networks (GANs) ● Semi-supervised and Unsupervised Learing on Neural Networks ● Learning on Extremely Large Distributed Systems https://arxiv.org/abs/1507.00677, https://arxiv.org/abs/1605.07725, https://arxiv.org/abs/1702.08720, https://arxiv.org/abs/1704.03976 https://openreview.net/forum?id=H1hLmF4Fx, https://arxiv.org/abs/1705.10941 (Self introduction)
  • 3. Contents ● What’s ICLR? ○ Main features of ICLR ● ICLR2017 ● GANs
  • 4. What’s ICLR? ● International Conferene on Learning Representations. Called ‘I CLeaR’. ● 3 days conference, held at the beginning of May. ● ICLR2017 is the 5th-ICLR, the 1st-ICLR was held in 2013 ● Website : http://www.iclr.cc/doku.php ○ Facebook page : https://www.facebook.com/iclr.cc/?fref=ts ○ Twitter account (This year) : https://twitter.com/iclr2017?lang=en ● http://www.kdnuggets.com/2016/02/iclr-deep-learning-scientific-publishing-ex periment.html
  • 5. Main features of ICLR (1) ● Focus on Deep Learning and its Application ○ Most of the accepted papers concern neural networks. Variational Auto-Encoder (ICLR2014) ADAM optimizer (ICLR2015)
  • 6. Main features of ICLR (2) ● Open Review System https://openreview.net/ ○ Single blind submission ○ Everyone can see all of the reviews and rebuttals. ○ Everyone can “join” the reviewing process.
  • 7. Other features ● 2 tracks : Conference track and Workshop track ● Single oral track ● Lots of IT giants sponsors https://blogs.sap.com/2017/05/22/dive-deep-into-deep-learning- sap-at-the-international-conference-on-learning-representations -iclr/
  • 9. ICLR2017 http://www.iclr.cc/doku.php?id=ICLR2017:main (Toulon, France) ● Attendees : 1100 (~300 last year) ● Accepted papers ○ Conference track : 196, 15 oral ○ Workshop track : ~110 https://twitter.com/iclr2017/status/864863545867079680 taken by Okumura-san
  • 12. ● Google, Facebook, Microsoft, OpenAI, … etc. https://prlz77.github.io/iclr2017-stats/
  • 13. My presentations ● 1 conference track poster and 1 workshop track poster https://arxiv.org/abs/1605.07725 https://openreview.net/pdf?id=H1hLmF4Fx
  • 15. Oral sessions: ● Single oral track ● Two sessions on each day : morning and afternoon ○ 1 invited talk and 2~3 oral selected presentations in each session. ● Live streamining: https://www.facebook.com/iclr.cc/?fref=ts
  • 16. Poster presentations ● Two poster sessions : morning and afternoon ● Conference track ○ ~40 posters at each session ○ 10~20 audiences around each poster ● Workshop track ○ ~20 posters ○ ~5 audiences ● (last year, poster) ○ ~5 audiences (Taken by Okumura-san)
  • 17. After hours parties Individually hosted by Google, Facebook, Salesforce ...etc. ○ Networking ○ Discussing with researchers and engineers ■ until late at night!
  • 18. Major & popular topics at ICLR2017 ● Generalization ability of Neural Networks ○ Rethinking generalization https://arxiv.org/abs/1611.03530 ○ Large batch training coverges to sharp minima https://arxiv.org/abs/1609.04836 ● Generative Adversarial Networks ○ Towards principled methods of training GANs https://openreview.net/pdf?id=Hk4_qw5xe ○ Energy based GANs https://arxiv.org/abs/1702.01691 ○ Two-sample tests by classifier https://openreview.net/forum?id=SJkXfE5xx&noteId=SJkXfE5xx ● Deep Reinforcement Learning ○ Reinforcement Learning with Unsupervised Auxiliary Tasks https://openreview.net/pdf?id=SJ6yPD5xg ○ (Not RL) Learning to act by predicting future https://arxiv.org/abs/1611.01779 ● Neural Programmer ○ Making Neural Programming Architectures Generalize via Recursion https://openreview.net/forum?id=BkbY4psgg&noteId=BkbY4psgg
  • 19. Generative Adversarial Networks (GANs) ● Originally proposed by Ian Goodfellow et al. (2014) ● Quite a lot of researchers have been conducting works on GANs ○ GAN Zoo https://github.com/hindupuravinash/the-gan-zoo https://github.com/hindupuravinash/the-gan-zoo (GANs)
  • 20. GANs frameworks ● Discriminator D(x) : trained to discriminate between real (dataset) examples and generated examples by the Generator G(z) ● Generator G(z) : trained to fool the Discriminator D(x). (GANs)
  • 21. What is good about GANs? ● We don’t need explicit expression of the denstiy for the generative models pG (x) ○ Only requires a stochastic generative process : x ~ pG (x) ● The training process can be incorporated into semi-supervised learning https://arxiv.org/abs/1606.03498 ○ Achieved the state of the art performance, especially on a few labeled semi-supervised dataset. https://arxiv.org/pdf/1702.08896.pdf (GANs)
  • 22. Important GANs works (submitted to ICLR2017) ● b-GAN (uehara et al. https://arxiv.org/pdf/1610.02920.pdf ) ○ The discriminator of GANs is the density ratio estimator rD (x) of r(x)=q(x) / pG (x) ○ Directly learn q(x) / pG (x) by minimizing Bregman Divergence between rD (x) and q(x) / pG (x). ● Implicit Generative Models (Mohamed et al. https://arxiv.org/abs/1610.03483 ) ○ Likelihood-free estimation through the GANs algorithm ● Deep and Hireachical Implicit models (Tran et al. https://arxiv.org/pdf/1702.08896.pdf ) (※Not subimitted to ICLR) ○ Likelihood-free variational inference (LFVI) through the GANs algorithm ■ it only requires that we can sample from qVI (x, z) and pModel (x, z) (My selection of)
  • 23. Thanks! I enjoyed water taxi ⚓ (Hotel <-> Conference)