Multimodal representation models for prediction and control from partial information

作     者：Zambelli, Martina Cully, Antoine Demiris, Yiannis 

作者机构：Imperial Coll London Dept Elect & Elect Engn Personal Robot Lab London England Imperial Coll London Personal Robot Lab London England DeepMind London London England Imperial Coll London Dept Comp London England 

出 版 物：《ROBOTICS AND AUTONOMOUS SYSTEMS》 (机器人学和自控系统)

年 卷 期：2020年第123卷

页      面：103312-000页

核心收录：

学科分类：12[管理学] 1201[管理学-管理科学与工程(可授管理学、工学学位)] 08[工学] 0811[工学-控制科学与工程] 0812[工学-计算机科学与技术（可授工学、理学学位）] 

基　　金：EPSRC, UK doctoral scholarship EU FP7 project WYSIWYD EU Horizon2020 project PAL [643783-RIA] 

主　　题：Multimodal learning Autonomous learning Variational autoencoder 

摘      要：Similar to humans, robots benefit from interacting with their environment through a number of different sensor modalities, such as vision, touch, sound. However, learning from different sensor modalities is difficult, because the learning model must be able to handle diverse types of signals, and learn a coherent representation even when parts of the sensor inputs are missing. In this paper, a multimodal variational autoencoder is proposed to enable an iCub humanoid robot to learn representations of its sensorimotor capabilities from different sensor modalities. The proposed model is able to (1) reconstruct missing sensory modalities, (2) predict the sensorimotor state of self and the visual trajectories of other agents actions, and (3) control the agent to imitate an observed visual trajectory. Also, the proposed multimodal variational autoencoder can capture the kinematic redundancy of the robot motion through the learned probability distribution. Training multimodal models is not trivial due to the combinatorial complexity given by the possibility of missing modalities. We propose a strategy to train multimodal models, which successfully achieves improved performance of different reconstruction models. Finally, extensive experiments have been carried out using an iCub humanoid robot, showing high performance in multiple reconstruction, prediction and imitation tasks. (C) 2019 Elsevier B.V. All rights reserved.