检索结果-内蒙古大学图书馆

iManip: Skill-Incremental Learning for Robotic Manipulation

学校读者我要写书评

暂无评论

arXiv 2025年

作者： Zheng, Zexin Cai, Jia-Feng Wu, Xiao-Ming Wei, Yi-Lin Tang, Yu-Ming Zheng, Wei-Shi School of Computer Science and Engineering Sun Yat-sen University China Pengcheng Lab China Key Laboratory of Machine Intelligence and Advanced Computing Ministry of Education China

The development of a generalist agent with adaptive multiple manipulation skills has been a long-standing goal in the robotics community. In this paper, we explore a crucial task, skill-incremental learning, in robotic manipulation, which is to endow the robots with the ability to learn new manipulation skills based on the previous learned knowledge without re-training. First, we build a skill-incremental environment based on the RLBench benchmark, and explore how traditional incremental methods perform in this setting. We find that they suffer from severe catastrophic forgetting due to the previous methods on classification overlooking the characteristics of temporality and action complexity in robotic manipulation tasks. Towards this end, we propose an incremental Manipulation framework, termed iManip, to mitigate the above issues. We firstly design a temporal replay strategy to maintain the integrity of old skills when learning new skill. Moreover, we propose the extendable PerceiverIO, consisting of an action prompt with extendable weight to adapt to new action primitives in new skill. Extensive experiments show that our framework performs well in Skill-Incremental Learning. Codes of the skill-incremental environment with our framework will be open-source. Copyright © 2025, The Authors. All rights reserved.

关键词： Robot learning

Attention-based Interactive Disentangling Network for Instance-level Emotional Voice Conversion

学校读者我要写书评

暂无评论

arXiv 2023年

作者： Chen, Yun Yang, Lingxiao Chen, Qi Lai, Jian-Huang Xie, Xiaohua School of Computer Science and Engineering Sun Yat-Sen University China Guangdong Province Key Laboratory of Information Security Technology China Key Laboratory of Machine Intelligence and Advanced Computing Ministry of Education China

Emotional Voice Conversion aims to manipulate a speech according to a given emotion while preserving non-emotion components. Existing approaches cannot well express fine-grained emotional attributes. In this paper, we propose an Attention-based Interactive diseNtangling Network (AINN) that leverages instance-wise emotional knowledge for voice conversion. We introduce a two-stage pipeline to effectively train our network: Stage I utilizes inter-speech contrastive learning to model fine-grained emotion and intra-speech disentanglement learning to better separate emotion and content. In Stage II, we propose to regularize the conversion with a multi-view consistency mechanism. This technique helps us transfer fine-grained emotion and maintain speech content. Extensive experiments show that our AINN outperforms state-of-the-arts in both objective and subjective metrics. Copyright © 2023, The Authors. All rights reserved.

关键词： machine learning

Unsupervised 3D Face Reconstruction with Reprogramming Skip Connections

学校读者我要写书评

暂无评论

Unsupervised 3D Face Reconstruction with Reprogramming Skip ...

IEEE International Conference on Multimedia and Expo (ICME)

作者： Zhuoming Dong Huajun Zhou Jianhuang Lai School of Computer Science and Engineering Sun Yat-sen University China Guangdong Province Key Laboratory of Information Security Technology China Key Laboratory of Machine Intelligence and Advanced Computing Ministry of Education China

In unsupervised 3D face reconstruction, existing methods modeling the canonical face typically exclude the skip connections between encoder-decoder pairs. Consequently, they have difficulty capturing appearance details necessary for the task. However, directly applying original skip connections only induces these methods to degrade to a trivial 2D texture reconstruction algorithm. In this paper, we propose novel Reprogramming Skip Connections (RSCs), which escape from bringing about degradation and improve the 3D face reconstruction quality. Specifically, the proposed method filters out inappropriate information causing degradation by aggregating the features from the encoder in spatial dimensions into several prototypes. These prototypes preserving beneficial information are subsequently combined with the corresponding decoder features with the help of expansion masks. Further, we design the masks reconstruction consistency loss to improve the quality of the expansion masks. Our experiments verify the superiority of our method compared to other competitors.

关键词：

Texture-Guided Saliency Distilling for Unsupervised Salient Object Detection

学校读者我要写书评

暂无评论

Texture-Guided Saliency Distilling for Unsupervised Salient ...

Conference on Computer Vision and Pattern Recognition (CVPR)

作者： Huajun Zhou Bo Qiao Lingxiao Yang Jianhuang Lai Xiaohua Xie School of Computer Science and Engineering Sun Yat-sen University China Guangdong Province Key Laboratory of Information Security Technology China Ministry of Education Key Laboratory of Machine Intelligence and Advanced Computing China

Deep Learning-based Unsupervised Salient Object Detection (USOD) mainly relies on the noisy saliency pseudo labels that have been generated from traditional handcraft methods or pre-trained networks. To cope with the noisy labels problem, a class of methods focus on only easy samples with reliable labels but ignore valuable knowledge in hard samples. In this paper, we propose a novel USOD method to mine rich and accurate saliency knowledge from both easy and hard samples. First, we propose a Confidence-aware Saliency Distilling (CSD) strategy that scores samples conditioned on samples' confidences, which guides the model to distill saliency knowledge from easy samples to hard samples progressively. Second, we propose a Boundary-aware Texture Matching (BTM) strategy to refine the boundaries of noisy labels by matching the textures around the predicted boundaries. Extensive experiments on RGB, RGB-D, RGB-T, and video SOD benchmarks prove that our method achieves state-of-the-art USOD performance. Code is available at ***/moothes/A2S-v2.

关键词：

Few Shot Object Detection with Incompletely Annotated Samples

学校读者我要写书评

暂无评论

Few Shot Object Detection with Incompletely Annotated Sample...

International Joint Conference on Neural Networks (IJCNN)

作者： Bo Qiao Huajun Zhou Lingxiao Yang Xiaohua Xie School of Computer Science and Engineering Sun Yat-sen University China Guangdong Province Key Laboratory of Information Security Technology China Key Laboratory of Machine Intelligence and Advanced Computing Ministry of Education China

Few shot object detection aims to generalize the model to previously unseen classes with only a few training samples, which has been attached great attention due to its practicability in real scenes. Many existing methods hold the missed detection issue, mainly due to the problem of incompletely annotated samples. Specifically, only partial objects in a sample are labeled, resulting in unlabeled objects being used as the background for training. This problem is especially serious for few-shot learning. To solve this noisy label problem, we first propose a label calibration method based on confidence to correct potential incorrect labels, and then introduce the class center library to eliminate the negative impact of unlabeled objects. We conduct extensive experiments on PASCAL VOC and MS-COCO benchmarks, which proves the effectiveness of our approach and achieves the state-of-the-art results.

关键词：

Hierarchical Semantic Correspondence Networks for Video Paragraph Grounding

学校读者我要写书评

暂无评论

Hierarchical Semantic Correspondence Networks for Video Para...

Conference on Computer Vision and Pattern Recognition (CVPR)

作者： Chaolei Tan Zihang Lin Jian-Fang Hu Wei-Shi Zheng Jianhuang Lai School of Computer Science and Engineering Sun Yat-sen University China Guangdong Province Key Laboratory of Information Security Technology China Key Laboratory of Machine Intelligence and Advanced Computing Ministry of Education China

Video Paragraph Grounding (VPG) is an essential yet challenging task in vision-language understanding, which aims to jointly localize multiple events from an untrimmed video with a paragraph query description. One of the critical challenges in addressing this problem is to comprehend the complex semantic relations between visual and textual modalities. Previous methods focus on modeling the contextual information between the video and text from a single-level perspective (i.e., the sentence level), ignoring rich visual-textual correspondence relations at different semantic levels, e.g., the video-word and video-paragraph correspondence. To this end, we propose a novel Hierarchical Semantic Correspondence Network (HSCNet), which explores multi-level visual-textual correspondence by learning hierarchical semantic alignment and utilizes dense supervision by grounding diverse levels of queries. Specifically, we develop a hierarchical encoder that encodes the multi-modal inputs into semantics-aligned representations at different levels. To exploit the hierarchical semantic correspondence learned in the encoder for multi-level supervision, we further design a hierarchical decoder that progressively performs finer grounding for lower-level queries conditioned on higher-level semantics. Extensive experiments demonstrate the effectiveness of HSCNet and our method significantly outstrips the state-of-the-arts on two challenging benchmarks, i.e., ActivityNet-Captions and TACoS.

关键词：

LLMDet: Learning Strong Open-Vocabulary Object Detectors under the Supervision of Large Language Models

学校读者我要写书评

暂无评论

arXiv 2025年

作者： Fu, Shenghao Yang, Qize Mo, Qijie Yan, Junkai Wei, Xihan Meng, Jingke Xie, Xiaohua Zheng, Wei-Shi School of Computer Science and Engineering Sun Yat-sen University China Tongyi Lab Alibaba Group China Peng Cheng Laboratory China Key Laboratory of Machine Intelligence and Advanced Computing Ministry of Education China Guangdong Province Key Laboratory of Information Security Technology China China

Recent open-vocabulary detectors achieve promising performance with abundant region-level annotated data. In this work, we show that an open-vocabulary detector co-training with a large language model by generating image-level detailed captions for each image can further improve performance. To achieve the goal, we first collect a dataset, GroundingCap-1M, wherein each image is accompanied by associated grounding labels and an image-level detailed caption. With this dataset, we finetune an open-vocabulary detector with training objectives including a standard grounding loss and a caption generation loss. We take advantage of a large language model to generate both region-level short captions for each region of interest and image-level long captions for the whole image. Under the supervision of the large language model, the resulting detector, LLMDet, outperforms the baseline by a clear margin, enjoying superior open-vocabulary ability. Further, we show that the improved LLMDet can in turn build a stronger large multi-modal model, achieving mutual benefits. The code, model, and dataset is available at https://***/iSEE-laboratory/LLMDet. Copyright © 2025, The Authors. All rights reserved.

关键词： Object detection

SAUGE: Taming SAM for Uncertainty-Aligned Multi-Granularity Edge Detection 39

学校读者我要写书评

暂无评论

SAUGE: Taming SAM for Uncertainty-Aligned Multi-Granularity ...

39th Annual AAAI Conference on Artificial intelligence, AAAI 2025

作者： Liufu, Xing Tan, Chaolei Lin, Xiaotong Qi, Yonggang Li, Jinxuan Hu, Jian-Fang School of Computer Science and Engineering Sun Yat-sen University Guangzhou China Beijing University of Posts and Telecommunications Beijing China Key Laboratory of Machine Intelligence and Advanced Computing Ministry of Education Guangzhou China Guangdong Province Key Laboratory of Information Security Technology China

ISBN: (纸本)157735897X

Edge labels are typically at various granularity levels owing to the varying preferences of annotators, thus handling the subjectivity of per-pixel labels has been a focal point for edge detection. Previous methods often employ a simple voting strategy to diminish such label uncertainty or impose a strong assumption of labels with a pre-defined distribution, e.g., Gaussian. In this work, we unveil that the segment anything model (SAM) provides strong prior knowledge to model the uncertainty in edge labels. Our key insight is that the intermediate SAM features inherently correspond to object edges at various granularities, which reflects different edge options due to uncertainty. Therefore, we attempt to align uncertainty with granularity by regressing intermediate SAM features from different layers to object edges at multi-granularity levels. In doing so, the model can fully and explicitly explore diverse "uncertainties" in a data-driven fashion. Specifically, we inject a lightweight module (∼ 1.5% additional parameters) into the frozen SAM to progressively fuse and adapt its intermediate features to estimate edges from coarse to fine. It is crucial to normalize the granularity level of human edge labels to match their innate uncertainty. For this, we simply perform linear blending to the real edge labels at hand to create pseudo labels with varying granularities. Consequently, our uncertainty-aligned edge detector can flexibly produce edges at any desired granularity (including an optimal one). Thanks to SAM, our model uniquely demonstrates strong generalizability for cross-dataset edge detection. Extensive experimental results on BSDS500, Muticue and NYUDv2 validate our model’s superiority. Copyright © 2025, Association for the Advancement of Artificial intelligence (***). All rights reserved.

关键词：

ViSpeak: Visual Instruction Feedback in Streaming Videos

学校读者我要写书评

暂无评论

arXiv 2025年

作者： Fu, Shenghao Yang, Qize Li, Yuan-Ming Peng, Yi-Xing Lin, Kun-Yu Wei, Xihan Hu, Jian-Fang Xie, Xiaohua Zheng, Wei-Shi School of Computer Science and Engineering Sun Yat-sen University China Tongyi Lab Alibaba Group China Peng Cheng Laboratory China Key Laboratory of Machine Intelligence and Advanced Computing Ministry of Education China Guangdong Province Key Laboratory of Information Security Technology China China

Recent advances in Large Multi-modal Models (LMMs) are primarily focused on offline video understanding. Instead, streaming video understanding poses great challenges to recent models due to its time-sensitive, omni-modal and interactive characteristics. In this work, we aim to extend the streaming video understanding from a new perspective and propose a novel task named Visual Instruction Feedback in which models should be aware of visual contents and learn to extract instructions from them. For example, when users wave their hands to agents, agents should recognize the gesture and start conversations with welcome information. Thus, following instructions in visual modality greatly enhances user-agent interactions. To facilitate research, we define seven key subtasks highly relevant to visual modality and collect the ViSpeak-Instruct dataset for training and the ViSpeak-Bench for evaluation. Further, we propose the ViSpeak model, which is a SOTA streaming video understanding LMM with GPT-4o-level performance on various streaming video understanding benchmarks. After finetuning on our ViSpeak-Instruct dataset, ViSpeak is equipped with basic visual instruction feedback ability, serving as a solid baseline for future research. Copyright © 2025, The Authors. All rights reserved.

关键词： Video streaming