检索结果-内蒙古大学图书馆

Mobile Manipulation Instruction Generation From Multiple Images With Automatic Metric Enhancement

IEEE ROBOTICS AND AUTOMATION LETTERS 2025年第3期10卷 3022-3029页

作者： Katsumata, Kei Kambara, Motonari Yashima, Daichi Korekata, Ryosuke Sugiura, Komei Keio Univ Yokohama Kanagawa 2238522 Japan

We consider the problem of generating free-form mobile manipulation instructions based on a target object image and receptacle image. Conventional image captioning models are not able to generate appropriate instructions because their architectures are typically optimized for single-image. In this study, we propose a model that handles both the target object and receptacle to generate free-form instruction sentences for mobile manipulation tasks. Moreover, we introduce a novel training method that effectively incorporates the scores from both learning-based and n-gram based automatic evaluation metrics as rewards. This method enables the model to learn the co-occurrence relationships between words and appropriate paraphrases. Results demonstrate that our proposed method outperforms baseline methods including representative multimodal large language models on standard automatic evaluation metrics. Moreover, physical experiments reveal that using our method to augment data on language instructions improves the performance of an existing multimodal language understanding model for mobile manipulation.

关键词： visualization Training Measurement Feature extraction Standards Older adults Object recognition Generators Detectors Three-dimensional displays deep learning for visual perception deep learning methods mobile manipulation

来源：评论

学校读者我要写书评

暂无评论

BiCo-Fusion: Bidirectional Complementary LiDAR-Camera Fusion for Semantic- and Spatial-Aware 3D Object Detection

引用

IEEE ROBOTICS AND AUTOMATION LETTERS 2025年第2期10卷 1457-1464页

作者： Song, Yang Wang, Lin Hong Kong Univ Sci & Technol Guangzhou AI Thrust Guangzhou 511458 Guangdong Peoples R China Nanyang Technol Univ NTU Sch Elect & Elect Engn EEE Singapore 639798 Singapore

3D object detection is an important task that has been widely applied in autonomous driving. To perform this task, a new trend is to fuse multi-modal inputs, i.e., LiDAR and camera. Under such a trend, recent methods fuse these two modalities by unifying them in the same 3D space. However, during direct fusion in a unified space, the drawbacks of both modalities (LiDAR features struggle with detailed semantic information and the camera lacks accurate 3D spatial information) are also preserved, diluting semantic and spatial awareness of the final unified representation. To address the issue, this letter proposes a novel bidirectional complementary LiDAR-camera fusion framework, called BiCo-Fusion that can achieve robust semantic- and spatial-aware 3D object detection. The key insight is to fuse LiDAR and camera features in a bidirectional complementary way to enhance the semantic awareness of the LiDAR and the 3D spatial awareness of the camera. The enhanced features from both modalities are then adaptively fused to build a semantic- and spatial-aware unified representation. Specifically, we introduce Pre-Fusion consisting of a Voxel Enhancement Module (VEM) to enhance the semantic awareness of voxel features from 2D camera features and Image Enhancement Module (IEM) to enhance the 3D spatial awareness of camera features from 3D voxel features. We then introduce Unified Fusion (U-Fusion) to adaptively fuse the enhanced features from the last stage to build a unified representation. Extensive experiments demonstrate the superiority of our BiCo-Fusion against the prior arts.

关键词： Cameras Laser radar Three-dimensional displays Semantics Feature extraction Point cloud compression Fuses Object detection Proposals Robot vision systems deep learning for visual perception sensor fusion object detection segmentation and categorization

来源：评论

学校读者我要写书评

暂无评论

OMEGA: Efficient Occlusion-Aware Navigation for Air-Ground Robots in Dynamic Environments via State Space Model

引用

IEEE ROBOTICS AND AUTOMATION LETTERS 2025年第2期10卷 1066-1073页

作者： Wang, Junming Guan, Xiuxian Sun, Zekai Shen, Tianxiang Huang, Dong Liu, Fangming Cui, Heming Univ Hong Kong Hong Kong Peoples R China Huazhong Univ Sci & Technol Peng Cheng Lab Wuhan 430074 Peoples R China Shanghai AI Labolatory Shanghai 200232 Peoples R China

Air-ground robots (AGRs) are widely used in surveillance and disaster response due to their exceptional mobility and versatility (i.e., flying and driving). Current AGR navigation systems perform well in static occlusion-prone environments (e.g., indoors) by using 3D semantic occupancy networks to predict occlusions for complete local mapping and then computing Euclidean Signed Distance Field (ESDF) for path planning. However, these systems face challenges in dynamic scenes (e.g., crowds) due to limitations in perception networks' low prediction accuracy and path planners' high computation overhead. In this paper, we propose Omega, which contains OccMamba with an Efficient AGR-plAnner to address the above-mentioned problems. OccMamba adopts a novel architecture that separates semantic and occupancy prediction into independent branches, incorporating the mamba module to efficiently extract semantic and geometric features in 3D environments. This ensures the network can learn long-distance dependencies and improve prediction accuracy. Features are then combined within the Bird's Eye View (BEV) space to minimise computational overhead during feature fusion. The resulting semantic occupancy map is integrated into the local map, providing occlusion awareness of the dynamic environment. Our AGR-Planner utilizes this local map and employs Kinodynamic A* search and gradient-based trajectory optimization for ESDF-free and energy-efficient planning. Experiments demonstrate that OccMamba outperforms the state-of-the-art 3D semantic occupancy network with 25.0% mIoU. End-to-end navigation experiments in dynamic scenes verify OMEGA's efficiency, achieving a 96% average planning success rate.

关键词： Semantics Three-dimensional displays Navigation Accuracy Real-time systems Robots Planning Point cloud compression Energy efficiency Transformers deep learning for visual perception autonomous navigation 3D semantic occupancy prediction

来源：评论

学校读者我要写书评

暂无评论

DreamUp3D: Object-Centric Generative Models for Single-View 3D Scene Understanding and Real-to-Sim Transfer

引用

IEEE ROBOTICS AND AUTOMATION LETTERS 2024年第4期9卷 3291-3298页

作者： Wu, Yizhe Borde, Haitz Saez de Ocariz Collins, Jack Jones, Oiwi Parker Posner, Ingmar Oxford Robot Inst Appl AI Lab Oxford OX1 2JH England

3D scene understanding for robotic applications exhibits a unique set of requirements including real-time inference, object-centric latent representation learning, accurate 6D pose estimation and 3D reconstruction of objects. Current methods for scene understanding typically rely on a combination of trained models paired with either an explicit or learnt volumetric representation, all of which have their own drawbacks and limitations. We introduce DreamUp3D, a novel Object-Centric Generative Model (OCGM) designed explicitly to perform inference on a 3D scene informed only by a single RGB-D image. DreamUp3D is a self-supervised model, trained end-to-end, and is capable of segmenting objects, providing 3D object reconstructions, generating object-centric latent representations and accurate per-object 6D pose estimates. We compare DreamUp3D to baselines including NeRFs, pre-trained CLIP-features, ObSurf, and ObPose, in a range of tasks including 3D scene reconstruction, object-matching and object pose estimation. Our experiments show that our model outperforms all baselines by a significant margin in real-world scenarios displaying its applicability for 3D scene understanding tasks while meeting the strict demands exhibited in robotics applications.

关键词： Three-dimensional displays Shape Task analysis Solid modeling Pose estimation Point cloud compression Image reconstruction deep learning for visual perception RGB-D perception recognition

来源：评论

学校读者我要写书评

暂无评论

Semi-supervised generative adversarial network to estimate the depth map of underwater targets in acoustic images for 3D reconstruction

引用

MEASUREMENT SCIENCE AND TECHNOLOGY 2025年第3期36卷 035403-035403页

作者： Bai, Boming Yan, Weisheng Cui, Rongxin Northwestern Polytech Univ Sch Marine Sci & Technol Xian 710072 Peoples R China

Owing to the unique imaging formulation principle of acoustic cameras, certain dimensions of the target may be lost in acoustic images. Some studies have been conducted to retrieve these missing details from a single pseudo front depth that contains 3D information about the target. Nevertheless, the use of a single pseudo front depth may face ambiguity problems in recovering the 3D information of acoustic targets. Therefore, this paper presents a novel semi-supervised generative adversarial network (SGAN) for estimating the depth of two adjacent acoustic images at know relative sonar poses, which generates high-accuracy reconstruction results, while reducing the dependence on the labeled ground truth of acoustic targets. Specifically, we design a semi-supervised depth estimation framework that utilizes image warping for semi-supervised training, resulting in denser depth maps. Various datasets containing underwater targets and terrain were generated to simulate real underwater scenarios. Experiments with simulator data and real images verify the feasibility of using SGAN for depth estimation in 3D target reconstruction.

关键词： generative adversarial network (GAN) acoustic camera-based 3D reconstruction deep learning for visual perception

来源：评论

学校读者我要写书评

暂无评论

Towards Better Data Exploitation in Self-Supervised Monocular Depth Estimation

引用

IEEE ROBOTICS AND AUTOMATION LETTERS 2024年第1期9卷 763-770页

作者： Liu, Jinfeng Kong, Lingtong Yang, Jie Liu, Wei Shanghai Jiao Tong Univ Dept Automat Shanghai 200240 Peoples R China

Depth estimation plays an important role in robotic perception systems. The self-supervised monocular paradigm has gained significant attention since it can free training from the reliance on depth annotations. Despite recent advancements, existing self-supervised methods still underutilize the available training data, limiting their generalization ability. In this letter, we take two data augmentation techniques, namely Resizing-Cropping and Splitting-Permuting, to fully exploit the potential of training datasets. Specifically, the original image and the generated two augmented images are fed into the training pipeline simultaneously and we leverage them to conduct self-distillation. Additionally, we introduce the detail-enhanced DepthNet with an extra full-scale branch in the encoder and a grid decoder to enhance the restoration of fine details in depth maps. Experimental results demonstrate our method can achieve state-of-the-art performance on the KITTI and Cityscapes datasets. Moreover, our KITTI models also show superior generalization performance when transferring to Make3D, NYUv2 and Cityscapes datasets.

关键词： deep learning for visual perception deep learning methods visual learning

来源：评论

学校读者我要写书评

暂无评论

Co-Occ: Coupling Explicit Feature Fusion With Volume Rendering Regularization for Multi-Modal 3D Semantic Occupancy Prediction

引用

IEEE ROBOTICS AND AUTOMATION LETTERS 2024年第6期9卷 5687-5694页

作者： Pan, Jingyi Wang, Zipeng Wang, Lin Hong Kong Univ Sci & Technol AI Thrust Guangzhou 511458 Guangdong Peoples R China Hong Kong Univ Sci & Technol AI CMA Thrust Guangzhou 511458 Guangdong Peoples R China Hong Kong Univ Sci & Technol Dept CSE Hong Kong Peoples R China

3D semantic occupancy prediction is a pivotal task in the field of autonomous driving. Recent approaches have made great advances in 3D semantic occupancy predictions on a single modality. However, multi-modal semantic occupancy prediction approaches have encountered difficulties in dealing with the modality heterogeneity, modality misalignment, and insufficient modality interactions that arise during the fusion of different modalities data, which may result in the loss of important geometric and semantic information. This letter presents a novel multi-modal, i.e., LiDAR-camera 3D semantic occupancy prediction framework, dubbed Co-Occ, which couples explicit LiDAR-camera feature fusion with implicit volume rendering regularization. The key insight is that volume rendering in the feature space can proficiently bridge the gap between 3D LiDAR sweeps and 2D images while serving as a physical regularization to enhance LiDAR-camera fused volumetric representation. Specifically, we first propose a Geometric- and Semantic-aware Fusion (GSFusion) module to explicitly enhance LiDAR features by incorporating neighboring camera features through a K-nearest neighbors (KNN) search. Then, we employ volume rendering to project the fused feature back to the image planes for reconstructing color and depth maps. These maps are then supervised by input images from the camera and depth estimations derived from LiDAR, respectively. Extensive experiments on the popular nuScenes and SemanticKITTI benchmarks verify the effectiveness of our Co-Occ for 3D semantic occupancy prediction.

关键词： Laser radar Three-dimensional displays Semantics Cameras Rendering (computer graphics) Feature extraction Task analysis deep learning for visual perception sensor fusion semantic scene understanding

来源：评论

学校读者我要写书评

暂无评论

Collaborative Multi-Object Tracking With Conformal Uncertainty Propagation

引用

IEEE ROBOTICS AND AUTOMATION LETTERS 2024年第4期9卷 3323-3330页

作者： Su, Sanbao Han, Songyang Li, Yiming Zhang, Zhili Feng, Chen Ding, Caiwen Miao, Fei Univ Connecticut Dept Comp Sci & Engn Storrs CT 06268 USA Univ Connecticut Storrs CT 06268 USA NYU Tandon Sch Engn Brooklyn NY 11201 USA

Object detection and multiple object tracking (MOT) are essential components of self-driving systems. Accurate detection and uncertainty quantification are both critical for onboard modules, such as perception, prediction, and planning, to improve the safety and robustness of autonomous vehicles. Collaborative object detection (COD) has been proposed to improve detection accuracy and reduce uncertainty by leveraging the viewpoints of multiple agents. However, little attention has been paid to how to leverage the uncertainty quantification from COD to enhance MOT performance. In this letter, as the first attempt to address this challenge, we design an uncertainty propagation framework called MOT-CUP. Our framework first quantifies the uncertainty of COD through direct modeling and conformal prediction, and propagates this uncertainty information into the motion prediction and association steps. MOT-CUP is designed to work with different collaborative object detectors and baseline MOT algorithms. We evaluate MOT-CUP on V2X-Sim, a comprehensive collaborative perception dataset, and demonstrate a 2% improvement in accuracy and a 2.67 x reduction in uncertainty compared to the baselines, e.g. SORT and ByteTrack. In scenarios characterized by high occlusion levels, our MOT-CUP demonstrates a noteworthy 4.01% improvement in accuracy. MOT-CUP demonstrates the importance of uncertainty quantification in both COD and MOT, and provides the first attempt to improve the accuracy and reduce the uncertainty in MOT based on COD through uncertainty propagation.

关键词： Uncertainty Object detection Collaboration Kalman filters Prediction algorithms Predictive models Estimation Computer vision for transportation deep learning for visual perception multi-robot systems visual tracking

来源：评论

学校读者我要写书评

暂无评论

learning Rearrangement Manipulation via Scene Prediction in Point Cloud

引用

IEEE ROBOTICS AND AUTOMATION LETTERS 2024年第12期9卷 11090-11097页

作者： Ma, Anji Duan, Xingguang Beijing Inst Technol Sch Mechatron Engn Beijing 100081 Peoples R China

Predicting scene evolution conditioned on robotic actions is a vital technique in modeling robot manipulations. Previous studies have primarily focused on learning spatiotemporally continuous actions like Cartesian displacements, thus applying them to planar-pushing tasks. In this letter, we propose a scene prediction model that learns higher-level robot actions, such as grasping and pick-and-place, and is applicable to planning such actions in rearrangement manipulations. The model takes partially observed point clouds (e.g., from a single camera) and robot pick-and-place actions to predict point clouds of future scenes. The model directly learns scene prediction using point cloud observation and representation, without requiring prior knowledge of object properties like cad models, pose, or instance segmentation. We train the model only on a synthetic dataset acquired entirely automatically with minimal human intervention. Our experiments validate that our model can substantially learn robot grasping and pick-and-place actions and show that, when integrated into a sample-based planning framework, predicting scenes in point clouds outperforms the image-based baseline in grasping and rearrangement manipulation. Moreover, results show that our method can be directly transferred to real-world environments without fine-tuning and show promising performance on a collection of 18 household objects.

关键词： deep learning in grasping and manipulation manipulation planning deep learning for visual perception deep learning in grasping and manipulation manipulation planning

来源：评论

学校读者我要写书评

暂无评论

learning-To-Rank Approach for Identifying Everyday Objects Using a Physical-World Search Engine

引用

IEEE ROBOTICS AND AUTOMATION LETTERS 2024年第3期9卷 2088-2095页

作者： Kaneda, Kanta Nagashima, Shunya Korekata, Ryosuke Kambara, Motonari Sugiura, Komei Keio Univ Yokohama 2238522 Japan

Domestic service robots offer a solution to the increasing demand for daily care and support. A human-in-the-loop approach that combines automation and operator intervention is considered to be a realistic approach to their use in society. Therefore, we focus on the task of retrieving target objects from open-vocabulary user instructions in a human-in-the-loop setting, which we define as the learning-to-rank physical objects (LTRPO) task. For example, given the instruction "Please go to the dining room which has a round table. Pick up the bottle on it," the model is required to output a ranked list of target objects that the operator/user can select. In this letter, we propose MultiRankIt, which is a novel approach for the LTRPO task. MultiRankIt introduces the Crossmodal Noun Phrase Encoder to model the relationship between phrases that contain referring expressions and the target bounding box, and the Crossmodal Region Feature Encoder to model the relationship between the target object and multiple images of its surrounding contextual environment. Additionally, we built a new dataset for the LTRPO task that consists of instructions with complex referring expressions accompanied by real indoor environmental images that feature various target objects. We validated our model on the dataset and it outperformed the baseline method in terms of the mean reciprocal rank and recall@k. Furthermore, we conducted physical experiments in a setting where a domestic service robot retrieved everyday objects in a standardized domestic environment, based on users' instruction in a human-in-the-loop setting. The experimental results demonstrate that the success rate for object retrieval achieved 80%.

关键词： deep learning methods learning categories and concepts deep learning for visual perception

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：