检索结果-内蒙古大学图书馆

iManip: Skill-Incremental Learning for Robotic Manipulation

学校读者我要写书评

暂无评论

arXiv 2025年

作者： Zheng, Zexin Cai, Jia-Feng Wu, Xiao-Ming Wei, Yi-Lin Tang, Yu-Ming Zheng, Wei-Shi School of Computer Science and Engineering Sun Yat-sen University China Pengcheng Lab China Key Laboratory of Machine Intelligence and Advanced Computing Ministry of Education China

The development of a generalist agent with adaptive multiple manipulation skills has been a long-standing goal in the robotics community. In this paper, we explore a crucial task, skill-incremental learning, in robotic manipulation, which is to endow the robots with the ability to learn new manipulation skills based on the previous learned knowledge without re-training. First, we build a skill-incremental environment based on the RLBench benchmark, and explore how traditional incremental methods perform in this setting. We find that they suffer from severe catastrophic forgetting due to the previous methods on classification overlooking the characteristics of temporality and action complexity in robotic manipulation tasks. Towards this end, we propose an incremental Manipulation framework, termed iManip, to mitigate the above issues. We firstly design a temporal replay strategy to maintain the integrity of old skills when learning new skill. Moreover, we propose the extendable PerceiverIO, consisting of an action prompt with extendable weight to adapt to new action primitives in new skill. Extensive experiments show that our framework performs well in Skill-Incremental Learning. Codes of the skill-incremental environment with our framework will be open-source. Copyright © 2025, The Authors. All rights reserved.

关键词： Robot learning

LLMDet: Learning Strong Open-Vocabulary Object Detectors under the Supervision of Large Language Models

学校读者我要写书评

暂无评论

arXiv 2025年

作者： Fu, Shenghao Yang, Qize Mo, Qijie Yan, Junkai Wei, Xihan Meng, Jingke Xie, Xiaohua Zheng, Wei-Shi School of Computer Science and Engineering Sun Yat-sen University China Tongyi Lab Alibaba Group China Peng Cheng Laboratory China Key Laboratory of Machine Intelligence and Advanced Computing Ministry of Education China Guangdong Province Key Laboratory of Information Security Technology China China

Recent open-vocabulary detectors achieve promising performance with abundant region-level annotated data. In this work, we show that an open-vocabulary detector co-training with a large language model by generating image-level detailed captions for each image can further improve performance. To achieve the goal, we first collect a dataset, GroundingCap-1M, wherein each image is accompanied by associated grounding lab.ls and an image-level detailed caption. With this dataset, we finetune an open-vocabulary detector with training objectives including a standard grounding loss and a caption generation loss. We take advantage of a large language model to generate both region-level short captions for each region of interest and image-level long captions for the whole image. Under the supervision of the large language model, the resulting detector, LLMDet, outperforms the baseline by a clear margin, enjoying superior open-vocabulary ability. Further, we show that the improved LLMDet can in turn build a stronger large multi-modal model, achieving mutual benefits. The code, model, and dataset is availab.e at https://***/iSEE-lab.ratory/LLMDet. Copyright © 2025, The Authors. All rights reserved.

关键词： Object detection

ViSpeak: Visual Instruction Feedback in Streaming Videos

学校读者我要写书评

暂无评论

arXiv 2025年

作者： Fu, Shenghao Yang, Qize Li, Yuan-Ming Peng, Yi-Xing Lin, Kun-Yu Wei, Xihan Hu, Jian-Fang Xie, Xiaohua Zheng, Wei-Shi School of Computer Science and Engineering Sun Yat-sen University China Tongyi Lab Alibaba Group China Peng Cheng Laboratory China Key Laboratory of Machine Intelligence and Advanced Computing Ministry of Education China Guangdong Province Key Laboratory of Information Security Technology China China

Recent advances in Large Multi-modal Models (LMMs) are primarily focused on offline video understanding. Instead, streaming video understanding poses great challenges to recent models due to its time-sensitive, omni-modal and interactive characteristics. In this work, we aim to extend the streaming video understanding from a new perspective and propose a novel task named Visual Instruction Feedback in which models should be aware of visual contents and learn to extract instructions from them. For example, when users wave their hands to agents, agents should recognize the gesture and start conversations with welcome information. Thus, following instructions in visual modality greatly enhances user-agent interactions. To facilitate research, we define seven key subtasks highly relevant to visual modality and collect the ViSpeak-Instruct dataset for training and the ViSpeak-Bench for evaluation. Further, we propose the ViSpeak model, which is a SOTA streaming video understanding LMM with GPT-4o-level performance on various streaming video understanding benchmarks. After finetuning on our ViSpeak-Instruct dataset, ViSpeak is equipped with basic visual instruction feedback ability, serving as a solid baseline for future research. Copyright © 2025, The Authors. All rights reserved.

关键词： Video streaming

A Hierarchical Semantic Distillation Framework for Open-Vocabulary Object Detection

学校读者我要写书评

暂无评论

arXiv 2025年

作者： Fu, Shenghao Yan, Junkai Yang, Qize Wei, Xihan Xie, Xiaohua Zheng, Wei-Shi School of Computer Science and Engineering Sun Yat-sen University Guangzhou510006 China Peng Cheng Laboratory China Tongyi Lab Alibaba Group China School of Computer Science and Engineering Sun Yat-sen University Guangdong Guangzhou China Guangdong Key Laboratory of Information Security Technology Sun Yat-sen University Guangdong Guangzhou China Key Laboratory of Machine Intelligence and Advanced Computing Sun Yat-sen University Ministry of Education Guangdong Guangzhou China

Open-vocabulary object detection (OVD) aims to detect objects beyond the training annotations, where detectors are usually aligned to a pre-trained vision-language model, e.g., CLIP, to inherit its generalizable recognition ability so that detectors can recognize new or novel objects. However, previous works directly align the feature space with CLIP and fail to learn the semantic knowledge effectively. In this work, we propose a hierarchical semantic distillation framework named HD-OVD to construct a comprehensive distillation process, which exploits generalizable knowledge from the CLIP model in three aspects. In the first hierarchy of HD-OVD, the detector learns fine-grained instance-wise semantics from the CLIP image encoder by modeling relations among single objects in the visual space. Besides, we introduce text space novel-class-aware classification to help the detector assimilate the highly generalizable classwise semantics from the CLIP text encoder, representing the second hierarchy. Lastly, abundant image-wise semantics containing multi-object and their contexts are also distilled by an image-wise contrastive distillation. Benefiting from the elab.rated semantic distillation in triple hierarchies, our HD-OVD inherits generalizable recognition ability from CLIP in instance, class, and image levels. Thus, we boost the novel AP on the OV-COCO dataset to 46.4% with a ResNet50 backbone, which outperforms others by a clear margin. We also conduct extensive ablation studies to analyze how each component works. Code is availab.e at https://***/iSEE-lab.ratory/HD-OVD. Copyright © 2025, The Authors. All rights reserved.

关键词： Semantics

SpikingMiniLM: energy-efficient spiking transformer for natural language understanding

学校读者我要写书评

暂无评论

Science China(Information Sciences) 2024年第10期67卷 115-128页

作者： Jiayu ZHANG Jiangrong SHEN Zeke WANG Qinghai GUO Rui YAN Gang PAN Huajin TANG College of Computer Science and Technology Zhejiang University The State Key Lab of Brain-Machine Intelligence Zhejiang University Collaborative Innovation Center of Artificial Intelligence Zhejiang University Advanced Computing and Storage Laboratory Huawei Technologies Co. Ltd. College of Computer Science and Technology Zhejiang University of Technology MOE Frontier Science Center for Brain Science and Brain-Machine Integration Zhejiang University

In the era of large-scale pretrained models, artificial neural networks（ANNs） have excelled in natural language understanding（NLU） tasks. However, their success often necessitates substantial computational resources and energy consumption. To address this, we explore the potential of spiking neural networks（SNNs） in NLU——a promising avenue with demonstrated advantages, including reduced power consumption and improved efficiency due to their event-driven characteristics. We propose the SpikingMiniLM,a novel spiking Transformer model tailored for natural language understanding. We first introduce a multi-step encoding method to convert text embeddings into spike trains. Subsequently, we redesign the attention mechanism and residual connections to make our model operate on the pure spike-based paradigm without any normalization technique. To facilitate stable and fast convergence, we propose a general parameter initialization method grounded in the stable firing rate principle. Furthermore, we apply an ANN-to-SNN knowledge distillation to overcome the challenges of pretraining SNNs. Our approach achieves a macro-average score of 75.5 on the dev sets of the GLUE benchmark, retaining 98% of the performance exhibited by the teacher model MiniLMv2. Our smaller model also achieves similar performance to BERTMINIwith fewer parameters and much lower energy consumption, underscoring its competitiveness and resource efficiency in NLU tasks.

关键词： spiking neural networks natural language understanding spiking Transformer spike-based attention multi-step encoding ANN-to-SNN distillation

High-throughput Sampling, Communicating and Training for Reinforcement Learning Systems 31

学校读者我要写书评

暂无评论

High-throughput Sampling, Communicating and Training for Rei...

31st IEEE/ACM International Symposium on Quality of Service, IWQoS 2023

作者： Zhao, Laiping Dai, Xinan Zhao, Zhixin Xin, Yusong Hu, Yitao Qian, Jun Yao, Jun Li, Keqiu College of Intelligence & Computing Tianjin University Tianjin Key Lab. of Advanced Networking Tianjin China Huawei Noahs Ark Lab Beijing China

ISBN: (纸本)9798350399738

Reinforcement Learning (RL) algorithms require large amounts of computational resources and time to train due to the simultaneous model training and real-time interactions with simulation environments. This results in an RL algorithm's training time potentially exceeding days to months. We present HRL, a comprehensive optimization system designed to improve the system throughput of RL algorithms. HRL addresses the challenges of discovering and identifying bottlenecks in the sampling, training, or communication stages and promptly improving their performance. We propose a group-parallel pipeline method to improve the sampling efficiency and apply data quantization and multi-learner training to resolve the network and learning bottlenecks. The HRL system has been fully implemented and integrated with XingTian. The results of the experiments show that the HRL system can improve the throughput by 18.6%-90.6%. © 2023 IEEE.

关键词： Pipelines

Embedding Dynamic Attributed Networks by Modeling the Evolution Processes 28

学校读者我要写书评

暂无评论

Embedding Dynamic Attributed Networks by Modeling the Evolut...

28th International Conference on Computational Linguistics, COLING 2020

作者： Xu, Zenan Ou, Zijing Su, Qinliang Yu, Jianxing Quan, Xiaojun Lin, Zhenkun School of Data and Computer Science Sun Yat-Sen University China Guangdong Key Laboratory of Big Data Analysis and Processing Guangzhou China Key Lab. of Machine Intelligence and Advanced Computing Ministry of Education China

ISBN: (纸本)9781952148279

Network embedding has recently emerged as a promising technique to embed nodes of a network into low-dimensional vectors. While fairly successful, most existing works focus on the embedding techniques for static networks. But in practice, there are many networks that are evolving over time and hence are dynamic, e.g., the social networks. To address this issue, a high-order spatio-temporal embedding model is developed to track the evolutions of dynamic networks. Specifically, an activeness-aware neighborhood embedding method is first proposed to extract the high-order neighborhood information at each given timestamp. Then, an embedding prediction framework is further developed to capture the temporal correlations, in which the attention mechanism is employed instead of recurrent neural networks (RNNs) for its efficiency in computing and flexibility in modeling. Extensive experiments are conducted on four real-world datasets from three different areas. It is shown that the proposed method outperforms all the baselines by a substantial margin for the tasks of dynamic link prediction and node classification, which demonstrates the effectiveness of the proposed methods on tracking the evolutions of dynamic networks. © 2020 COLING 2020 - 28th International Conference on Computational Linguistics, Proceedings of the Conference. All rights reserved.

关键词： Recurrent neural networks

Topology analysis system for vehicular Ad Hoc network 17

学校读者我要写书评

暂无评论

Topology analysis system for vehicular Ad Hoc network

17th International Conference on Parallel and Distributed computing, Applications and Technologies, PDCAT 2016

作者： Dong, Baihong Deng, Jian Wu, Weigang Meng, Tianyu 1. School of Data and Computer Science Sun Yat-Sen Univ. China Guangdong Province Key Lab. of Big Data Analysis and Processing China Key Laboratory of Machine Intelligence and Advanced Computing Ministry of Education Guangzhou China

ISBN: (纸本)9781509050819

With the development of the technology, Vehicular Ad-hoc Network is developing rapidly and continuously. And many new algorithms in VANET were put forward. For example, many people apply the Named Data Networks or Software Defined Networking6 to the VANET. However, few researches are looking into the analysis of the topology of the VANET, among most of which are based on simple situations and calculation of the track of vehicles. And therefore, these analysis are inadequate for analyzing the topology under complex situations. This paper is going to put forward a new method for analyzing the change of the topology of VANET, which is capable to quantitatively analyze the change under different and complex situations. In the real experiment, when compared with traditional analysis methods, this new method can analyze more precisely the impact of some possible wireless communication problem on the change of the topology, for example, the impact of hidden node on the vehicular wireless communication and whether the number or the density of vehicles makes the topology of VANET more stable. And this new method can provide a numerical result instead of purely qualitative analysis. © 2016 IEEE.

关键词： Topology

A dimension-decreasing particle swarm optimization method for portfolio optimization 15

学校读者我要写书评

暂无评论

A dimension-decreasing particle swarm optimization method fo...

17th Genetic and Evolutionary Computation Conference, GECCO 2015

作者： Wang, Jia-Bin Chen, Wei-Neng Zhang, Jun Lin, Ying School of Advanced Computing Sun Yat-sen University Guangzhou China Key Lab. Machine Intelligence and Advanced Computing Ministry of Education China Collaborative Innovation Center of High Performance Computing China Engineering Research Center of Supercomputing Engineering Software Ministry of Education China Department of Psychology Sun Yat-sen University Guangzhou China

ISBN: (纸本)9781450334884

Portfolio optimization problems are challenging as they contain different kinds of constrains and their complexity becomes very high when the number of assets grows. In this paper, we develop a dimension-decreasing particle swarm optimization (DDPSO) for solving multi-constrained portfolio optimization problems. DDPSO improves the efficiency of PSO for solving portfolio optimization problems with a lot of asset and it can easily handle the cardinality constraint in portfolio optimization. To improve search diversity, the dimension-decreasing method is coupled with the comprehensive learning particle swarm optimization (CLPSO) algorithm. The proposed method is tested on benchmark problems from the OR library. Experimental results show that the proposed algorithm performs well. Copyright is held by the owner/author(s).

关键词： Particle swarm optimization (PSO)