SlideShare a Scribd company logo

Skill-Based Meta-Reinforcement Learning

M
MLAI2

The document discusses skill-based meta-reinforcement learning, detailing a framework that utilizes prior knowledge to accelerate learning for various tasks such as cooking. It presents a three-phase approach: skill extraction from offline data, skill-based meta-training, and target task learning. The framework reportedly allows for faster adaptation and learning of new long-horizon and sparse-reward tasks compared to existing methods.

Technology
1 of 37
Download to read offline
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
Skill-Based Meta-Reinforcement Learning Taewook Nam Shao-Hua Sun Karl Pertsch Sung Ju Hwang Joseph J. Lim
Human Leverages Prior Knowledge “Cook a pancake”
Human Leverages Prior Knowledge “Cook a pancake” SAC Policy
Human Leverages Prior Knowledge “Cook a pancake” Prior knowledge Prior knowledge
Human Leverages Prior Knowledge “Cook a pancake” How to hold frying pan How to turn on the stove
Human Leverages Prior Knowledge “Cook a pancake” “Make a sandwich” “Fry an egg”
Human Leverages Prior Knowledge “Cook a pancake” “Make a sandwich” “Fry an egg” How to hold frying pan How to turn on the stove Skill-based RL Meta-RL
Skill-Based Reinforcement Learning[1, 2] Task-Agnostic Dataset How to turn on a stove How to hold a frying pan Skill [1] Accelerating Reinforcement Learning with learned Skill Prior. Pertsch et al. CoRL 2020
 [2] Opal: O ffl ine Primitive Discovery for Accelerating O ffl ine Reinforcement Learning. Ajay et al. ICLR 2021
Skill-Based Reinforcement Learning[1, 2] Task-Agnostic Dataset How to turn on a stove How to hold a frying pan Skill [1] Accelerating Reinforcement Learning with learned Skill Prior. Pertsch et al. CoRL 2020
 [2] Opal: O ffl ine Primitive Discovery for Accelerating O ffl ine Reinforcement Learning. Ajay et al. ICLR 2021 Reward
Skill-Based Reinforcement Learning[1, 2] T1 T2 T3 T4 TT T5 Target Task Task-Agnostic Dataset How to turn on a stove How to hold a frying pan Skill [1] Accelerating Reinforcement Learning with learned Skill Prior. Pertsch et al. CoRL 2020
 [2] Opal: O ffl ine Primitive Discovery for Accelerating O ffl ine Reinforcement Learning. Ajay et al. ICLR 2021 + E ffi cient exploration Reward
T1 T2 T3 T4 TT T5 Target Task Task-Agnostic Dataset How to turn on a stove How to hold a frying pan Skill Skill-Based Reinforcement Learning[1, 2] [1] Accelerating Reinforcement Learning with learned Skill Prior. Pertsch et al. CoRL 2020
 [2] Opal: O ffl ine Primitive Discovery for Accelerating O ffl ine Reinforcement Learning. Ajay et al. ICLR 2021 + E ffi cient exploration - How to learn quickly Reward
Meta Reinforcement Learning[1, 2] T1 T2 T3 T4 TT T5 Target Task Training Tasks T1 T2 T5 T3 T4 “Fry an egg” “Make a sandwich” [1] Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks. Finn et al. ICML 2017
 [2] E ffi cient O ff -Policy Meta-reinforcement Learning via Probabilistic Context Variables. Rakelly et al. ICML 2019 “Cook a pancake” + How to learn quickly
T1 T2 T3 T4 TT T5 Target Task Training Tasks T1 T2 T5 T3 T4 “Fry an egg” “Cook a pancake” “Make a sandwich” Meta Reinforcement Learning[1, 2] [1] Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks. Finn et al. ICML 2017
 [2] E ffi cient O ff -Policy Meta-reinforcement Learning via Probabilistic Context Variables. Rakelly et al. ICML 2019 + How to learn quickly - Limited to short-horizon task
This Work : Meta-RL + Skill-based RL Training Tasks T1 T2 T5 T3 T4 Task-Agnostic Dataset T1 T2 T3 T4 TT T5 Target Task
This Work : Meta-RL + Skill-based RL Useful skill T1 T2 T3 T4 TT T5 Target Task T1 T2 T5 T3 T4
Meta-RL + Skill-based RL How to learn quickly T1 T2 T3 T4 TT T5 Target Task T1 T2 T5 T3 T4
Skill-Based Meta-Reinforcement Learning T1 T2 T3 T4 TT T5 T1 T2 T5 T3 T4 Fast learning of new long horizon task
Skill-Based Meta-Reinforcement Learning T1 T2 T3 T4 TT T5 T1 T2 T5 T3 T4 Fast learning of new long horizon task SiMPL
Phase 1 : Skill Extraction Extract skill from task-agnostic o ffl ine data, following SPiRL[1]. Skill Task-Agnostic Data a0 a1 a2 a3 s0 s1 s2 s3 s4 … [1] Accelerating Reinforcement Learning with learned Skill Prior. Pertsch et al. CoRL 2020
Phase 2 : Skill-based Meta-training Meta-train based on extracted skill, following PEARL[1]. Meta Policy T1 T2 Meta-Training Tasks T5 T3 T4 Skill [1] E ffi cient O ff -Policy Meta-reinforcement Learning via Probabilistic Context Variables. Rakelly et al. ICML 2019
Phase 2 : Skill-based Meta-training Meta-train based on extracted skill, following PEARL[1]. Transitions Meta Policy Task Encoder T1 T2 Meta-Training Tasks T5 T3 T4 Skill [1] E ffi cient O ff -Policy Meta-reinforcement Learning via Probabilistic Context Variables. Rakelly et al. ICML 2019
Phase 3 : Target Task Learning Warm-start target task learning by task encoding. Target Task T1 T2 T3 T4 TT T5
Phase 3 : Target Task Learning Warm-start target task learning by task encoding. Initial Exploration Target Task T1 T2 T3 T4 TT T5 Task Encoder
Phase 3 : Target Task Learning Warm-start target task learning by task encoding. Policy Task Encoder Skill Target Task T1 T2 T3 T4 TT T5 Initial Exploration
Phase 3 : Target Task Learning Warm-start target task learning by task encoding. Policy Task Encoder Skill Target Task T1 T2 T3 T4 TT T5 Initial Exploration Fine-tune
Environment Maze Navigation 2000 steps / sparse reward for completion Kitchen Manipulation 280 steps / sparse reward for subtask completions
Environment Meta-Training Tasks Target Tasks arget Tasks Agent Meta-training Tasks Target Tasks top burner light switch slide cabinet hinge cabinet slide cabinet bottom burner bottom burner kettle bottom burner light switch top burner microwave kettle slide cabinet hinge cabinet light switch 1 2 3 4 vigation (b) Kitchen Manipulation Target Tasks Target Tasks rget Tasks Agent Meta-training Tasks Target Tasks top burner light switch slide cabinet hinge cabinet slide cabinet bottom burner bottom burner kettle bottom burner light switch top burner microwave kettle slide cabinet hinge cabinet light switch 1 2 3 4 vigation (b) Kitchen Manipulation Target Tasks Meta-Training Tasks
SiMPL Learns Quickly SiMPL can solve this task in 100 episode, but other baseline can’t. Meta-training Tasks Episode 0 Episodes 20 Episodes 80 SPiRL Ours Meta-Training Task Target Task Target Task Agent Trajectory Episode 0 Episode 20 Episode 100 SiMPL (Ours) SPiRL Episode 0 Episode 20 Episode 100 PEARL-ft MTRL Target Location Episode 0 Episodes 20 Episodes 80 SPiRL Ours Meta-Training Task Target Task Meta-training Tasks Agent Trajectory Start Location
SiMPL Learns Quickly SiMPL can solve this task in 100 episode, but other baseline can’t. Meta-training Tasks Episode 0 Episodes 20 Episodes 80 SPiRL Ours Meta-Training Task Target Task Target Task Agent Trajectory Episode 0 Episode 20 Episode 100 SiMPL (Ours) SPiRL Episode 0 Episode 20 Episode 100 PEARL-ft MTRL Target Location Episode 0 Episodes 20 Episodes 80 SPiRL Ours Meta-Training Task Target Task Meta-training Tasks Agent Trajectory Start Location
SiMPL Learns Quickly SiMPL can solve this task in 100 episode, but other baseline can’t. Meta-training Tasks Episode 0 Episodes 20 Episodes 80 SPiRL Ours Meta-Training Task Target Task Target Task Agent Trajectory Episode 0 Episode 20 Episode 100 SiMPL (Ours) SPiRL Episode 0 Episode 20 Episode 100 PEARL-ft MTRL Target Location Episode 0 Episodes 20 Episodes 80 SPiRL Ours Meta-Training Task Target Task Meta-training Tasks Agent Trajectory Start Location
SiMPL Learns Quickly SiMPL can solve this task in 100 episode, but other baseline can’t. Meta-training Tasks Episode 0 Episodes 20 Episodes 80 SPiRL Ours Meta-Training Task Target Task Target Task Agent Trajectory Episode 0 Episode 20 Episode 100 SiMPL (Ours) SPiRL Episode 0 Episode 20 Episode 100 PEARL-ft MTRL Target Location Episode 0 Episodes 20 Episodes 80 SPiRL Ours Meta-Training Task Target Task Meta-training Tasks Agent Trajectory Start Location
SiMPL Learns Quickly SiMPL can solve this task in 100 episode, but other baseline can’t. Meta-training Tasks Episode 0 Episodes 20 Episodes 80 SPiRL Ours Meta-Training Task Target Task Target Task Agent Trajectory Episode 0 Episode 20 Episode 100 SiMPL (Ours) SPiRL Episode 0 Episode 20 Episode 100 PEARL-ft MTRL Target Location Episode 0 Episodes 20 Episodes 80 SPiRL Ours Meta-Training Task Target Task Meta-training Tasks Agent Trajectory Start Location
SiMPL Learns Quickly SiMPL can solve this task in 100 episode, but other baseline can’t. Meta-training Tasks Episode 0 Episodes 20 Episodes 80 SPiRL Ours Meta-Training Task Target Task Target Task Agent Trajectory Episode 0 Episode 20 Episode 100 SiMPL (Ours) SPiRL Episode 0 Episode 20 Episode 100 PEARL-ft MTRL Target Location Episode 0 Episodes 20 Episodes 80 SPiRL Ours Meta-Training Task Target Task Meta-training Tasks Agent Trajectory Start Location
SiMPL Learns Quickly SiMPL converges faster than MTRL / Skill-based RL / Meta-RL baselines. SiMPL (Ours) SPiRL MTRL PEARL-ft SAC PEARL SiMPL (Ours) SPiRL MTRL PEARL-ft SAC PEARL
• SiMPL can leverage both o ffl ine dataset and tasks by combining
 skill-based RL and meta-RL • SiMPL can learn new long-horizon and sparse-reward tasks faster Summary
Summary • SiMPL can leverage both o ffl ine dataset and tasks by combining
 skill-based RL and meta-RL • SiMPL can learn new long-horizon and sparse-reward tasks faster
Skill-Based Meta-Reinforcement Learning Taewook Nam Shao-Hua Sun Karl Pertsch Sung Ju Hwang Joseph J. Lim Paper & Code : namsan96.github.io/SiMPL

More Related Content

PDF
Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Va...
Dongmin Lee
 
PDF
Unsupervised Curricula for Visual Meta Reinforcement Learning(CARML)
Jeong-Gwan Lee
 
PDF
[DL輪読会] off-policyなメタ強化学習
Deep Learning JP
 
PDF
PR12-094: Model-Agnostic Meta-Learning for fast adaptation of deep networks
Taesu Kim
 
PDF
Reinforcement Learning
CloudxLab
 
PDF
Continuous Adaptation via Meta Learning in Nonstationary and Competitive Envi...
Yoonho Lee
 
PDF
Learning how to learn
Joaquin Vanschoren
 
PDF
Towards Machine Learning of Motor Skills
butest
 
Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Va...
Dongmin Lee
 
Unsupervised Curricula for Visual Meta Reinforcement Learning(CARML)
Jeong-Gwan Lee
 
[DL輪読会] off-policyなメタ強化学習
Deep Learning JP
 
PR12-094: Model-Agnostic Meta-Learning for fast adaptation of deep networks
Taesu Kim
 
Reinforcement Learning
CloudxLab
 
Continuous Adaptation via Meta Learning in Nonstationary and Competitive Envi...
Yoonho Lee
 
Learning how to learn
Joaquin Vanschoren
 
Towards Machine Learning of Motor Skills
butest
 

Similar to Skill-Based Meta-Reinforcement Learning (20)

PPTX
Introduction to Reinforcement Learning.pptx
Harsha Patil
 
PPTX
semi supervised Learning and Reinforcement learning (1).pptx
Dr.Shweta
 
PDF
anintroductiontoreinforcementlearning-180912151720.pdf
ssuseradaf5f
 
PPTX
An introduction to reinforcement learning
Subrat Panda, PhD
 
PDF
MILA DL & RL summer school highlights
Natalia Díaz Rodríguez
 
PDF
Meta Dropout: Learning to Perturb Latent Features for Generalization
MLAI2
 
PPTX
Designing an AI that gains experience for absolute beginners
Tanzim Saqib
 
PDF
Reinforcement learning, energy systems and deep neural nets
Université de Liège (ULg)
 
PPTX
Battlesnake AWS ML Meetup Victoria 2020
Xavier Raffin
 
PPTX
Reinforcement learning.pptx
aniketgupta16440
 
PPTX
An Introduction to Reinforcement Learning (December 2018)
Julien SIMON
 
PPTX
Reinforcement learning
Zahra Khoobi
 
PPTX
Machine Learning and its types with application
ShivangSingh81
 
PDF
Reinforcement Learning.pdf
hemayadav41
 
PDF
BOIL: Towards Representation Change for Few-shot Learning
Hyungjun Yoo
 
PDF
Lec0
Duncan Chan
 
PPTX
Information Theoretic aspect of reinforcement learning
JongsuHa
 
PDF
Context-aware Dynamics Model for Generalization in Model-Based Reinforcement ...
ALINLAB
 
PPTX
Learning in AI
Minakshi Atre
 
PPT
CS8082_MachineLearnigTechniques _Unit-1.ppt
pushpait
 
Introduction to Reinforcement Learning.pptx
Harsha Patil
 
semi supervised Learning and Reinforcement learning (1).pptx
Dr.Shweta
 
anintroductiontoreinforcementlearning-180912151720.pdf
ssuseradaf5f
 
An introduction to reinforcement learning
Subrat Panda, PhD
 
MILA DL & RL summer school highlights
Natalia Díaz Rodríguez
 
Meta Dropout: Learning to Perturb Latent Features for Generalization
MLAI2
 
Designing an AI that gains experience for absolute beginners
Tanzim Saqib
 
Reinforcement learning, energy systems and deep neural nets
Université de Liège (ULg)
 
Battlesnake AWS ML Meetup Victoria 2020
Xavier Raffin
 
Reinforcement learning.pptx
aniketgupta16440
 
An Introduction to Reinforcement Learning (December 2018)
Julien SIMON
 
Reinforcement learning
Zahra Khoobi
 
Machine Learning and its types with application
ShivangSingh81
 
Reinforcement Learning.pdf
hemayadav41
 
BOIL: Towards Representation Change for Few-shot Learning
Hyungjun Yoo
 
Lec0
Duncan Chan
 
Information Theoretic aspect of reinforcement learning
JongsuHa
 
Context-aware Dynamics Model for Generalization in Model-Based Reinforcement ...
ALINLAB
 
Learning in AI
Minakshi Atre
 
CS8082_MachineLearnigTechniques _Unit-1.ppt
pushpait
 
Ad

More from MLAI2 (20)

PDF
Meta Learning Low Rank Covariance Factors for Energy-Based Deterministic Unce...
MLAI2
 
PDF
Online Hyperparameter Meta-Learning with Hypergradient Distillation
MLAI2
 
PDF
Online Coreset Selection for Rehearsal-based Continual Learning
MLAI2
 
PDF
Representational Continuity for Unsupervised Continual Learning
MLAI2
 
PDF
Sequential Reptile_Inter-Task Gradient Alignment for Multilingual Learning
MLAI2
 
PDF
Edge Representation Learning with Hypergraphs
MLAI2
 
PDF
Hit and Lead Discovery with Explorative RL and Fragment-based Molecule Genera...
MLAI2
 
PDF
Mini-Batch Consistent Slot Set Encoder For Scalable Set Encoding
MLAI2
 
PDF
Task Adaptive Neural Network Search with Meta-Contrastive Learning
MLAI2
 
PDF
Federated Semi-Supervised Learning with Inter-Client Consistency & Disjoint L...
MLAI2
 
PDF
Meta-GMVAE: Mixture of Gaussian VAE for Unsupervised Meta-Learning
MLAI2
 
PDF
Accurate Learning of Graph Representations with Graph Multiset Pooling
MLAI2
 
PDF
Contrastive Learning with Adversarial Perturbations for Conditional Text Gene...
MLAI2
 
PDF
Clinical Risk Prediction with Temporal Probabilistic Asymmetric Multi-Task Le...
MLAI2
 
PDF
MetaPerturb: Transferable Regularizer for Heterogeneous Tasks and Architectures
MLAI2
 
PDF
Adversarial Self-Supervised Contrastive Learning
MLAI2
 
PDF
Learning to Extrapolate Knowledge: Transductive Few-shot Out-of-Graph Link Pr...
MLAI2
 
PDF
Neural Mask Generator : Learning to Generate Adaptive Word Maskings for Langu...
MLAI2
 
PDF
Cost-effective Interactive Attention Learning with Neural Attention Process
MLAI2
 
PDF
Adversarial Neural Pruning with Latent Vulnerability Suppression
MLAI2
 
Meta Learning Low Rank Covariance Factors for Energy-Based Deterministic Unce...
MLAI2
 
Online Hyperparameter Meta-Learning with Hypergradient Distillation
MLAI2
 
Online Coreset Selection for Rehearsal-based Continual Learning
MLAI2
 
Representational Continuity for Unsupervised Continual Learning
MLAI2
 
Sequential Reptile_Inter-Task Gradient Alignment for Multilingual Learning
MLAI2
 
Edge Representation Learning with Hypergraphs
MLAI2
 
Hit and Lead Discovery with Explorative RL and Fragment-based Molecule Genera...
MLAI2
 
Mini-Batch Consistent Slot Set Encoder For Scalable Set Encoding
MLAI2
 
Task Adaptive Neural Network Search with Meta-Contrastive Learning
MLAI2
 
Federated Semi-Supervised Learning with Inter-Client Consistency & Disjoint L...
MLAI2
 
Meta-GMVAE: Mixture of Gaussian VAE for Unsupervised Meta-Learning
MLAI2
 
Accurate Learning of Graph Representations with Graph Multiset Pooling
MLAI2
 
Contrastive Learning with Adversarial Perturbations for Conditional Text Gene...
MLAI2
 
Clinical Risk Prediction with Temporal Probabilistic Asymmetric Multi-Task Le...
MLAI2
 
MetaPerturb: Transferable Regularizer for Heterogeneous Tasks and Architectures
MLAI2
 
Adversarial Self-Supervised Contrastive Learning
MLAI2
 
Learning to Extrapolate Knowledge: Transductive Few-shot Out-of-Graph Link Pr...
MLAI2
 
Neural Mask Generator : Learning to Generate Adaptive Word Maskings for Langu...
MLAI2
 
Cost-effective Interactive Attention Learning with Neural Attention Process
MLAI2
 
Adversarial Neural Pruning with Latent Vulnerability Suppression
MLAI2
 
Ad

Recently uploaded (20)

PDF
Per capita expenditure prediction using model stacking based on satellite ima...
IAESIJAI
 
PDF
Machine learning based COVID-19 study performance prediction
IAESIJAI
 
PPTX
Cloud computing and distributed systems.
kkkkkhan
 
PPTX
Big Data Technologies - Introduction.pptx
kuthubussaman1
 
PPTX
MYSQL Presentation for SQL database connectivity
Swati270511
 
PPTX
sap open course for s4hana steps from ECC to s4
sreeni2106invites
 
PDF
Build a system with the filesystem maintained by OSTree @ COSCUP 2025
Jian-Hong Pan
 
PPTX
Understanding_Digital_Forensics_Presentation.pptx
ImranKhan423233
 
PDF
MIND Revenue Release Quarter 2 2025 Press Release
MIND CTI
 
PPTX
Detection-First SIEM: Rule Types, Dashboards, and Threat-Informed Strategy
ReZa AdineH
 
PDF
Blue Purple Modern Animated Computer Science Presentation.pdf.pdf
Akar16
 
PDF
7 ChatGPT Prompts to Help You Define Your Ideal Customer Profile.pdf
SOFTTECHHUB
 
PDF
Profit Center Accounting in SAP S/4HANA, S4F28 Col11
Libreria ERP
 
PPTX
KOM of Painting work and Equipment Insulation REV00 update 25-dec.pptx
muhammadmuneebnaeem7
 
PDF
Optimiser vos workloads AI/ML sur Amazon EC2 et AWS Graviton
Julien SIMON
 
PPT
Teaching material agriculture food technology
LiaRayya
 
PDF
Diabetes mellitus diagnosis method based random forest with bat algorithm
IAESIJAI
 
PPTX
Digital-Transformation-Roadmap-for-Companies.pptx
David Malick Dieng
 
PDF
Spectral efficient network and resource selection model in 5G networks
IAESIJAI
 
PDF
Approach and Philosophy of On baking technology
30anthuong17
 
Per capita expenditure prediction using model stacking based on satellite ima...
IAESIJAI
 
Machine learning based COVID-19 study performance prediction
IAESIJAI
 
Cloud computing and distributed systems.
kkkkkhan
 
Big Data Technologies - Introduction.pptx
kuthubussaman1
 
MYSQL Presentation for SQL database connectivity
Swati270511
 
sap open course for s4hana steps from ECC to s4
sreeni2106invites
 
Build a system with the filesystem maintained by OSTree @ COSCUP 2025
Jian-Hong Pan
 
Understanding_Digital_Forensics_Presentation.pptx
ImranKhan423233
 
MIND Revenue Release Quarter 2 2025 Press Release
MIND CTI
 
Detection-First SIEM: Rule Types, Dashboards, and Threat-Informed Strategy
ReZa AdineH
 
Blue Purple Modern Animated Computer Science Presentation.pdf.pdf
Akar16
 
7 ChatGPT Prompts to Help You Define Your Ideal Customer Profile.pdf
SOFTTECHHUB
 
Profit Center Accounting in SAP S/4HANA, S4F28 Col11
Libreria ERP
 
KOM of Painting work and Equipment Insulation REV00 update 25-dec.pptx
muhammadmuneebnaeem7
 
Optimiser vos workloads AI/ML sur Amazon EC2 et AWS Graviton
Julien SIMON
 
Teaching material agriculture food technology
LiaRayya
 
Diabetes mellitus diagnosis method based random forest with bat algorithm
IAESIJAI
 
Digital-Transformation-Roadmap-for-Companies.pptx
David Malick Dieng
 
Spectral efficient network and resource selection model in 5G networks
IAESIJAI
 
Approach and Philosophy of On baking technology
30anthuong17
 

Skill-Based Meta-Reinforcement Learning

  • 1. Skill-Based Meta-Reinforcement Learning Taewook Nam Shao-Hua Sun Karl Pertsch Sung Ju Hwang Joseph J. Lim
  • 2. Human Leverages Prior Knowledge “Cook a pancake”
  • 3. Human Leverages Prior Knowledge “Cook a pancake” SAC Policy
  • 4. Human Leverages Prior Knowledge “Cook a pancake” Prior knowledge Prior knowledge
  • 5. Human Leverages Prior Knowledge “Cook a pancake” How to hold frying pan How to turn on the stove
  • 6. Human Leverages Prior Knowledge “Cook a pancake” “Make a sandwich” “Fry an egg”
  • 7. Human Leverages Prior Knowledge “Cook a pancake” “Make a sandwich” “Fry an egg” How to hold frying pan How to turn on the stove Skill-based RL Meta-RL
  • 8. Skill-Based Reinforcement Learning[1, 2] Task-Agnostic Dataset How to turn on a stove How to hold a frying pan Skill [1] Accelerating Reinforcement Learning with learned Skill Prior. Pertsch et al. CoRL 2020
 [2] Opal: O ffl ine Primitive Discovery for Accelerating O ffl ine Reinforcement Learning. Ajay et al. ICLR 2021
  • 9. Skill-Based Reinforcement Learning[1, 2] Task-Agnostic Dataset How to turn on a stove How to hold a frying pan Skill [1] Accelerating Reinforcement Learning with learned Skill Prior. Pertsch et al. CoRL 2020
 [2] Opal: O ffl ine Primitive Discovery for Accelerating O ffl ine Reinforcement Learning. Ajay et al. ICLR 2021 Reward
  • 10. Skill-Based Reinforcement Learning[1, 2] T1 T2 T3 T4 TT T5 Target Task Task-Agnostic Dataset How to turn on a stove How to hold a frying pan Skill [1] Accelerating Reinforcement Learning with learned Skill Prior. Pertsch et al. CoRL 2020
 [2] Opal: O ffl ine Primitive Discovery for Accelerating O ffl ine Reinforcement Learning. Ajay et al. ICLR 2021 + E ffi cient exploration Reward
  • 11. T1 T2 T3 T4 TT T5 Target Task Task-Agnostic Dataset How to turn on a stove How to hold a frying pan Skill Skill-Based Reinforcement Learning[1, 2] [1] Accelerating Reinforcement Learning with learned Skill Prior. Pertsch et al. CoRL 2020
 [2] Opal: O ffl ine Primitive Discovery for Accelerating O ffl ine Reinforcement Learning. Ajay et al. ICLR 2021 + E ffi cient exploration - How to learn quickly Reward
  • 12. Meta Reinforcement Learning[1, 2] T1 T2 T3 T4 TT T5 Target Task Training Tasks T1 T2 T5 T3 T4 “Fry an egg” “Make a sandwich” [1] Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks. Finn et al. ICML 2017
 [2] E ffi cient O ff -Policy Meta-reinforcement Learning via Probabilistic Context Variables. Rakelly et al. ICML 2019 “Cook a pancake” + How to learn quickly
  • 13. T1 T2 T3 T4 TT T5 Target Task Training Tasks T1 T2 T5 T3 T4 “Fry an egg” “Cook a pancake” “Make a sandwich” Meta Reinforcement Learning[1, 2] [1] Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks. Finn et al. ICML 2017
 [2] E ffi cient O ff -Policy Meta-reinforcement Learning via Probabilistic Context Variables. Rakelly et al. ICML 2019 + How to learn quickly - Limited to short-horizon task
  • 14. This Work : Meta-RL + Skill-based RL Training Tasks T1 T2 T5 T3 T4 Task-Agnostic Dataset T1 T2 T3 T4 TT T5 Target Task
  • 15. This Work : Meta-RL + Skill-based RL Useful skill T1 T2 T3 T4 TT T5 Target Task T1 T2 T5 T3 T4
  • 16. Meta-RL + Skill-based RL How to learn quickly T1 T2 T3 T4 TT T5 Target Task T1 T2 T5 T3 T4
  • 19. Phase 1 : Skill Extraction Extract skill from task-agnostic o ffl ine data, following SPiRL[1]. Skill Task-Agnostic Data a0 a1 a2 a3 s0 s1 s2 s3 s4 … [1] Accelerating Reinforcement Learning with learned Skill Prior. Pertsch et al. CoRL 2020
  • 20. Phase 2 : Skill-based Meta-training Meta-train based on extracted skill, following PEARL[1]. Meta Policy T1 T2 Meta-Training Tasks T5 T3 T4 Skill [1] E ffi cient O ff -Policy Meta-reinforcement Learning via Probabilistic Context Variables. Rakelly et al. ICML 2019
  • 21. Phase 2 : Skill-based Meta-training Meta-train based on extracted skill, following PEARL[1]. Transitions Meta Policy Task Encoder T1 T2 Meta-Training Tasks T5 T3 T4 Skill [1] E ffi cient O ff -Policy Meta-reinforcement Learning via Probabilistic Context Variables. Rakelly et al. ICML 2019
  • 22. Phase 3 : Target Task Learning Warm-start target task learning by task encoding. Target Task T1 T2 T3 T4 TT T5
  • 23. Phase 3 : Target Task Learning Warm-start target task learning by task encoding. Initial Exploration Target Task T1 T2 T3 T4 TT T5 Task Encoder
  • 24. Phase 3 : Target Task Learning Warm-start target task learning by task encoding. Policy Task Encoder Skill Target Task T1 T2 T3 T4 TT T5 Initial Exploration
  • 25. Phase 3 : Target Task Learning Warm-start target task learning by task encoding. Policy Task Encoder Skill Target Task T1 T2 T3 T4 TT T5 Initial Exploration Fine-tune
  • 26. Environment Maze Navigation 2000 steps / sparse reward for completion Kitchen Manipulation 280 steps / sparse reward for subtask completions
  • 27. Environment Meta-Training Tasks Target Tasks arget Tasks Agent Meta-training Tasks Target Tasks top burner light switch slide cabinet hinge cabinet slide cabinet bottom burner bottom burner kettle bottom burner light switch top burner microwave kettle slide cabinet hinge cabinet light switch 1 2 3 4 vigation (b) Kitchen Manipulation Target Tasks Target Tasks rget Tasks Agent Meta-training Tasks Target Tasks top burner light switch slide cabinet hinge cabinet slide cabinet bottom burner bottom burner kettle bottom burner light switch top burner microwave kettle slide cabinet hinge cabinet light switch 1 2 3 4 vigation (b) Kitchen Manipulation Target Tasks Meta-Training Tasks
  • 28. SiMPL Learns Quickly SiMPL can solve this task in 100 episode, but other baseline can’t. Meta-training Tasks Episode 0 Episodes 20 Episodes 80 SPiRL Ours Meta-Training Task Target Task Target Task Agent Trajectory Episode 0 Episode 20 Episode 100 SiMPL (Ours) SPiRL Episode 0 Episode 20 Episode 100 PEARL-ft MTRL Target Location Episode 0 Episodes 20 Episodes 80 SPiRL Ours Meta-Training Task Target Task Meta-training Tasks Agent Trajectory Start Location
  • 29. SiMPL Learns Quickly SiMPL can solve this task in 100 episode, but other baseline can’t. Meta-training Tasks Episode 0 Episodes 20 Episodes 80 SPiRL Ours Meta-Training Task Target Task Target Task Agent Trajectory Episode 0 Episode 20 Episode 100 SiMPL (Ours) SPiRL Episode 0 Episode 20 Episode 100 PEARL-ft MTRL Target Location Episode 0 Episodes 20 Episodes 80 SPiRL Ours Meta-Training Task Target Task Meta-training Tasks Agent Trajectory Start Location
  • 30. SiMPL Learns Quickly SiMPL can solve this task in 100 episode, but other baseline can’t. Meta-training Tasks Episode 0 Episodes 20 Episodes 80 SPiRL Ours Meta-Training Task Target Task Target Task Agent Trajectory Episode 0 Episode 20 Episode 100 SiMPL (Ours) SPiRL Episode 0 Episode 20 Episode 100 PEARL-ft MTRL Target Location Episode 0 Episodes 20 Episodes 80 SPiRL Ours Meta-Training Task Target Task Meta-training Tasks Agent Trajectory Start Location
  • 31. SiMPL Learns Quickly SiMPL can solve this task in 100 episode, but other baseline can’t. Meta-training Tasks Episode 0 Episodes 20 Episodes 80 SPiRL Ours Meta-Training Task Target Task Target Task Agent Trajectory Episode 0 Episode 20 Episode 100 SiMPL (Ours) SPiRL Episode 0 Episode 20 Episode 100 PEARL-ft MTRL Target Location Episode 0 Episodes 20 Episodes 80 SPiRL Ours Meta-Training Task Target Task Meta-training Tasks Agent Trajectory Start Location
  • 32. SiMPL Learns Quickly SiMPL can solve this task in 100 episode, but other baseline can’t. Meta-training Tasks Episode 0 Episodes 20 Episodes 80 SPiRL Ours Meta-Training Task Target Task Target Task Agent Trajectory Episode 0 Episode 20 Episode 100 SiMPL (Ours) SPiRL Episode 0 Episode 20 Episode 100 PEARL-ft MTRL Target Location Episode 0 Episodes 20 Episodes 80 SPiRL Ours Meta-Training Task Target Task Meta-training Tasks Agent Trajectory Start Location
  • 33. SiMPL Learns Quickly SiMPL can solve this task in 100 episode, but other baseline can’t. Meta-training Tasks Episode 0 Episodes 20 Episodes 80 SPiRL Ours Meta-Training Task Target Task Target Task Agent Trajectory Episode 0 Episode 20 Episode 100 SiMPL (Ours) SPiRL Episode 0 Episode 20 Episode 100 PEARL-ft MTRL Target Location Episode 0 Episodes 20 Episodes 80 SPiRL Ours Meta-Training Task Target Task Meta-training Tasks Agent Trajectory Start Location
  • 34. SiMPL Learns Quickly SiMPL converges faster than MTRL / Skill-based RL / Meta-RL baselines. SiMPL (Ours) SPiRL MTRL PEARL-ft SAC PEARL SiMPL (Ours) SPiRL MTRL PEARL-ft SAC PEARL
  • 35. • SiMPL can leverage both o ffl ine dataset and tasks by combining
 skill-based RL and meta-RL • SiMPL can learn new long-horizon and sparse-reward tasks faster Summary
  • 36. Summary • SiMPL can leverage both o ffl ine dataset and tasks by combining
 skill-based RL and meta-RL • SiMPL can learn new long-horizon and sparse-reward tasks faster
  • 37. Skill-Based Meta-Reinforcement Learning Taewook Nam Shao-Hua Sun Karl Pertsch Sung Ju Hwang Joseph J. Lim Paper & Code : namsan96.github.io/SiMPL