检索结果-内蒙古大学图书馆

MOTPose: Multi-object 6D Pose Estimation for Dynamic video Sequences using Attention-based Temporal Fusion

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Periyasamy, Arul Selvam Behnke, Sven the Autonomous Intelligent Systems group Computer Science Institute VI – Intelligent Systems and Robotics the Center for Robotics the Lamarr Institute for Machine Learning and Artificial Intelligence University of Bonn Germany

Cluttered bin-picking environments are challenging for pose estimation models. Despite the impressive progress enabled by deep learning, single-view RGB pose estimation models perform poorly in cluttered dynamic environments. Imbuing the rich temporal information contained in the video of scenes has the potential to enhance models’ ability to deal with the adverse effects of occlusion and the dynamic nature of the environments. Moreover, joint object detection and pose estimation models are better suited to leverage the co-dependent nature of the tasks for improving the accuracy of both tasks. To this end, we propose attention-based temporal fusion for multi-object 6D pose estimation that accumulates information across multiple frames of a video sequence. Our MOTPose method takes a sequence of images as input and performs joint object detection and pose estimation for all objects in one forward pass. It learns to aggregate both object embeddings and object parameters over multiple time steps using cross-attention-based fusion modules. We evaluate our method on the physically-realistic cluttered bin-picking dataset SynPick and the YCB-video dataset and demonstrate improved pose estimation accuracy as well as better object detection accuracy. © 2024, CC BY.

关键词： Object detection

A Comparison of Prompt Engineering Techniques for Task Planning and Execution in Service robotics

学校读者我要写书评

暂无评论

A Comparison of Prompt Engineering Techniques for Task Plann...

IEEE-RAS International Conference on Humanoid Robots

作者： Jonas Bode Bastian Pätzold Raphael Memmesheimer Sven Behnke Autonomous Intelligent Systems group Computer Science Institute VI – Intelligent Systems and Robotics Lamarr Institute for Machine Learning and Artificial Intelligence and Center for Robotics University of Bonn Germany

ISBN: (数字)9798350373578

ISBN: (纸本)9798350373585

Recent advances in Large Language Models (LLMs) have been instrumental in autonomous robot control and human-robot interaction by leveraging their vast general knowledge and capabilities to understand and reason across a wide range of tasks and scenarios. Previous works have investigated various prompt engineering techniques for improving the performance of LLMs to accomplish tasks, while others have proposed methods that utilize LLMs to plan and execute tasks based on the available functionalities of a given robot platform. In this work, we consider both lines of research by comparing prompt engineering techniques and combinations thereof within the application of high-level task planning and execution in service robotics. We define a diverse set of tasks and a simple set of functionalities in simulation, and measure task completion accuracy and execution time for several state-of-the-art models. We make our code, including all prompts, available at https://***/AIS-Bonn/Prompt_Engineering.

关键词： Knowledge engineering Service robots Large language models Instruments Humanoid robots Reliability engineering Time measurement Planning Prompt engineering Tuning

Grasp Anything: Combining Teacher-Augmented Policy Gradient Learning with Instance Segmentation to Grasp Arbitrary Objects

学校读者我要写书评

暂无评论

Grasp Anything: Combining Teacher-Augmented Policy Gradient ...

IEEE International Conference on robotics and Automation (ICRA)

作者： Malte Mosbach Sven Behnke Autonomous Intelligent Systems Group Computer Science Institute VI – Intelligent Systems and Robotics – and the Center for Robotics and the Lamarr Institute for Machine Learning and Artificial Intelligence University of Bonn Germany

ISBN: (数字)9798350384574

ISBN: (纸本)9798350384581

Interactive grasping from clutter, akin to human dexterity, is one of the longest-standing problems in robot learning. Challenges stem from the intricacies of visual perception, the demand for precise motor skills, and the complex interplay between the two. In this work, we present Teacher-Augmented Policy Gradient (TAPG), a novel two-stage learning framework that synergizes reinforcement learning and policy distillation. After training a teacher policy to master the motor control based on object pose information, TAPG facilitates guided, yet adaptive, learning of a sensorimotor policy, based on object segmentation. We zero-shot transfer from simulation to a real robot by using Segment Anything Model for promptable object segmentation. Our trained policies adeptly grasp a wide variety of objects from cluttered scenarios in simulation and the real world based on human-understandable prompts. Furthermore, we show robust zero-shot transfer to novel objects. videos of our experiments are available at https://***/grasp_anything.

关键词： Training Instance segmentation Motor drives Object segmentation Reinforcement learning Robot sensing systems Motors

SLCF-Net: Sequential LiDAR-Camera Fusion for Semantic Scene Completion using a 3D Recurrent U-Net

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Cao, Helin Behnke, Sven Autonomous Intelligent Systems group Computer Science Institute VI-Intelligent Systems and Robotics Center for Robotics and the Lamarr Institute for Machine Learning and Artificial Intelligence University of Bonn Germany

We introduce SLCF-Net, a novel approach for the Semantic Scene Completion (SSC) task that sequentially fuses LiDAR and camera data. It jointly estimates missing geometry and semantics in a scene from sequences of RGB images and sparse LiDAR measurements. The images are semantically segmented by a pre-trained 2D U-Net and a dense depth prior is estimated from a depth-conditioned pipeline fueled by Depth Anything. To associate the 2D image features with the 3D scene volume, we introduce Gaussian-decay Depth-prior Projection (GDP). This module projects the 2D features into the 3D volume along the line of sight with a Gaussian-decay function, centered around the depth prior. Volumetric semantics is computed by a 3D U-Net. We propagate the hidden 3D U-Net state using the sensor motion and design a novel loss to ensure temporal consistency. We evaluate our approach on the SemanticKITTI dataset and compare it with leading SSC approaches. The SLCF-Net excels in all SSC metrics and shows great temporal consistency. Copyright © 2024, The Authors. All rights reserved.

关键词： Semantics

SLCF-Net: Sequential LiDAR-Camera Fusion for Semantic Scene Completion using a 3D Recurrent U-Net

学校读者我要写书评

暂无评论

SLCF-Net: Sequential LiDAR-Camera Fusion for Semantic Scene ...

IEEE International Conference on robotics and Automation (ICRA)

作者： Helin Cao Sven Behnke Autonomous Intelligent Systems Group Computer Science Institute VI – Intelligent Systems and Robotics – and the Center for Robotics and the Lamarr Institute for Machine Learning and Artificial Intelligence University of Bonn Germany

ISBN: (数字)9798350384574

ISBN: (纸本)9798350384581

关键词： Measurement Image segmentation Three-dimensional displays Laser radar Semantics Pipelines Robot sensing systems

Grasp Anything: Combining Teacher-Augmented Policy Gradient Learning with Instance Segmentation to Grasp Arbitrary Objects

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Mosbach, Malte Behnke, Sven The Autonomous Intelligent Systems group Computer Science Institute VI - Intelligent Systems and Robotics The Center for Robotics The Lamarr Institute for Machine Learning and Artificial Intelligence University of Bonn Germany

关键词： Distillation

MOTPose: Multi-object 6D Pose Estimation for Dynamic video Sequences using Attention-based Temporal Fusion

学校读者我要写书评

暂无评论

MOTPose: Multi-object 6D Pose Estimation for Dynamic Video S...

IEEE International Conference on robotics and Automation (ICRA)

作者： Arul Selvam Periyasamy Sven Behnke Autonomous Intelligent Systems Group Computer Science Institute VI – Intelligent Systems and Robotics – and the Center for Robotics and the Lamarr Institute for Machine Learning and Artificial Intelligence University of Bonn Germany

ISBN: (数字)9798350384574

ISBN: (纸本)9798350384581

关键词： Deep learning Accuracy Fuses Aggregates Pose estimation video sequences Object detection

PlaySlot: Learning Inverse Latent Dynamics for Controllable Object-Centric video Prediction and Planning

学校读者我要写书评

暂无评论

arXiv 2025年

作者： villar-Corrales, Angel Behnke, Sven Computer Science Institute VI – Intelligent Systems and Robotics Center for Robotics The Lamarr Institute for Machine Learning and Artificial Intelligence Germany

Predicting future scene representations is a crucial task for enabling robots to understand and interact with the environment. However, most existing methods rely on video sequences and simulations with precise action annotations, limiting their ability to leverage the large amount of available unlabeled video data. To address this challenge, we propose PlaySlot, an object-centric video prediction model that infers object representations and latent actions from unlabeled video sequences. It then uses these representations to forecast future object states and video frames. PlaySlot allows to generate multiple possible futures conditioned on latent actions, which can be inferred from video dynamics, provided by a user, or generated by a learned action policy, thus enabling versatile and interpretable world modeling. Our results show that PlaySlot outperforms both stochastic and object-centric baselines for video prediction across different environments. Furthermore, we show that our inferred latent actions can be used to learn robot behaviors sample-efficiently from unlabeled video demonstrations. videos and code are available at https://***/PlaySlot/. © 2025, CC BY.

关键词： Stochastic systems

MCDS-VSS: Moving Camera Dynamic Scene video Semantic Segmentation by Filtering with Self-Supervised Geometry and Motion

学校读者我要写书评

暂无评论

arXiv 2024年

作者： villar-Corrales, Angel Austermann, Moritz Behnke, Sven Autonomous Intelligent Systems Computer Science Institute VI Center for Robotics Lamarr Institute for Machine Learning and Artificial Intelligence University of Bonn Germany

autonomous systems, such as self-driving cars, rely on reliable semantic environment perception for decision making. Despite great advances in video semantic segmentation, existing approaches ignore important inductive biases and lack structured and interpretable internal representations. In this work, we propose MCDS-VSS, a structured filter model that learns in a self-supervised manner to estimate scene geometry and ego-motion of the camera, while also estimating the motion of external objects. Our model leverages these representations to improve the temporal consistency of semantic segmentation without sacrificing segmentation accuracy. MCDS-VSS follows a prediction-fusion approach in which scene geometry and camera motion are first used to compensate for ego-motion, then residual flow is used to compensate motion of dynamic objects, and finally the predicted scene features are fused with the current features to obtain a temporally consistent scene segmentation. Our model parses automotive scenes into multiple decoupled interpretable representations such as scene geometry, ego-motion, and object motion. Quantitative evaluation shows that MCDS-VSS achieves superior temporal consistency on video sequences while retaining competitive segmentation performance. Code and pretrained models are available in the project website. © 2024, CC BY.

关键词： Semantic Segmentation