[DL輪読会]Deep Dynamics Models for Learning Dexterous Manipulation

Editor's Notes

• #18: the model must have enough capacity to represent the complex dynamical system the use of ensembles is helpful, especially earlier in training when non-ensembled models can overfit badly and thus exhibit overconfident and harmful behavior there is not much difference between resetting model weights randomly at each training iteration versus warmstarting them from their previous values using a planning horizon that is either too long or too short can be detrimental: Short horizons lead to greedy planning, while long horizons suffer from compounding errors in the predictions PDDM, with action smoothing and soft updates, greatly outperforms the others medium values provide the best balance of dimensionality reduction and smooth integration of action samples versus loss of control authority. Here, too soft of a weighting leads to minimal movement of the hand, and too hard of a weighting leads to aggressive behaviors that frequently drop the objects
• #20: we confirm that most of the prior methods do in fact succeed, and we also see that even on this simpler task, policy gradient approaches such as NPG require prohibitively large amounts of data
• #21: when we increase the number of possible goals to 8 different options (90◦ and 45◦ rotations in the left, right, up, and down directions), we see that our method still succeeds, but the model-free approaches get stuck in local optima and are unable to fully achieve even the previously attainable goals. This inability to effectively address a “multi-task” or “multi-goal” setup is indeed a known drawback for model-free approaches, and it is particularly pronounced in such goal-conditioned tasks that require flexibility These additional goals do not make the task harder for PDDM, because even in learning 90◦ rotations, it is building a model of its interactions rather than specifically learning to get to those angles.
• #22: prior model-based approaches don’t actually solve this task (values below the grey line correspond to holding the pencil still near the middle of the paper)
• #23: This task is particularly challenging due to the inter-object interactions, which can lead to drastically discontinuous dynamics and frequent failures from dropping the objects. We were unable to get the other model-based or model-free methods to succeed at this task (Figure 8), but PDDM solves it using just 100,000 data points, or 2.7 hours worth of data moving a single ball to a goal location in the hand, posing the hand, and performing clockwise rotations instead of the learned counter-clockwise ones