How Object Information Improves Skeleton-based Human Action Recognition in Assembly Tasks

作     者：Aganian, Dustin Köhler, Mona Baake, Sebastian Eisenbach, Markus Groß, Horst-Michael 

作者机构：Ilmenau University of Technology Neuroinformatics and Cognitive Robotics Lab Ilmenau98684 Germany 

出 版 物：《arXiv》 (arXiv)

年 卷 期：2023年

核心收录：

主　　题：Object detection 

摘      要：As the use of collaborative robots (cobots) in industrial manufacturing continues to grow, human action recognition for effective human-robot collaboration becomes increasingly important. This ability is crucial for cobots to act autonomously and assist in assembly tasks. Recently, skeleton-based approaches are often used as they tend to generalize better to different people and environments. However, when processing skeletons alone, information about the objects a human interacts with is lost. Therefore, we present a novel approach of integrating object information into skeleton-based action recognition. We enhance two state-of-the-art methods by treating object centers as further skeleton joints. Our experiments on the assembly dataset IKEA ASM show that our approach improves the performance of these state-of-the-art methods to a large extent when combining skeleton joints with objects predicted by a state-of-the-art instance segmentation model. Our research sheds light on the benefits of combining skeleton joints with object information for human action recognition in assembly tasks. We analyze the effect of the object detector on the combination for action classification and discuss the important factors that must be taken into account. Copyright © 2023, The Authors. All rights reserved.