检索结果-内蒙古大学图书馆

Proceedings of the 38th International Conference on Neural Information Processing Systems

作者： Ziqi Yang Zhaopeng Peng Zihui Wang Jianzhong Qi Chaochao Chen Weike Pan Chenglu Wen Cheng Wang Xiaoliang Fan Fujian Key Laboratory of Sensing and Computing for Smart Cities Xiamen University China and Key Laboratory of Multimedia Trusted Perception and Efficient Computing Ministry of Education of China Xiamen University China School of Computing and Information Systems The University of Melbourne Australia College of Computer Science and Technology Zhejiang University Hangzhou China College of Computer Science and Software Engineering Shenzhen University Shenzhen China

ISBN: (纸本)9798331314385

Cross-domain recommendation (CDR) offers a promising solution to the data sparsity problem by enabling knowledge transfer between source and target domains. However, many recent CDR models overlook crucial issues such as privacy as well as the risk of negative transfer (which negatively impact model performance), especially in multi-domain settings. To address these challenges, we propose FedGCDR, a novel federated graph learning framework that securely and effectively leverages positive knowledge from multiple source domains. First, we design a positive knowledge transfer module that ensures privacy during inter-domain knowledge transmission. This module employs differential privacy-based knowledge extraction combined with a feature mapping mechanism, transforming source domain embeddings from federated graph attention networks into reliable domain knowledge. Second, we design a knowledge activation module to filter out potential harmful or conflicting knowledge from source domains, addressing the issues of negative transfer. This module enhances target domain training by expanding the graph of the target domain to generate reliable domain attentions and fine-tunes the target model for improved negative knowledge filtering and more accurate predictions. We conduct extensive experiments on 16 popular domains of the Amazon dataset, demonstrating that FedGCDR significantly outperforms state-of-the-art methods. We open source the code at https://***/LafinHana/FedGCDR.

关键词：

来源：评论

学校读者我要写书评

暂无评论

DiffAgent: Fast and Accurate Text-to-Image API Selection with Large Language Model

arXiv

引用

arXiv 2024年

作者： Zhao, Lirui Yang, Yue Zhang, Kaipeng Shao, Wenqi Zhang, Yuxin Qiao, Yu Luo, Ping Ji, Rongrong Key Laboratory of Multimedia Trusted Perception and Efficient Computing Ministry of Education of China Xiamen University China OpenGVLab Shanghai AI Laboratory China The University of Hong Kong Hong Kong Institute of Artificial Intelligence Xiamen University China Shanghai Jiao Tong University China

Text-to-image (T2I) generative models have attracted significant attention and found extensive applications within and beyond academic research. For example, the Civitai community, a platform for T2I innovation, currently hosts an impressive array of 74,492 distinct models. However, this diversity presents a formidable challenge in selecting the most appropriate model and parameters, a process that typically requires numerous trials. Drawing inspiration from the tool usage research of large language models (LLMs), we introduce DiffAgent, an LLM agent designed to screen the accurate selection in seconds via API calls. DiffAgent leverages a novel two-stage training framework, SFTA, enabling it to accurately align T2I API responses with user input in accordance with human preferences. To train and evaluate DiffAgent’s capabilities, we present DABench, a comprehensive dataset encompassing an extensive range of T2I APIs from the community. Our evaluations reveal that DiffAgent not only excels in identifying the appropriate T2I API but also underscores the effectiveness of the SFTA training framework. Codes are available at https://***/OpenGVLab/DiffAgent. Copyright © 2024, The Authors. All rights reserved.

关键词： Computational linguistics

来源：评论

学校读者我要写书评

暂无评论

Cheap and Quick: efficient Vision-Language Instruction Tuning for Large Language Models

arXiv

引用

arXiv 2023年

作者： Luo, Gen Zhou, Yiyi Ren, Tianhe Chen, Shengxin Sun, Xiaoshuai Ji, Rongrong Key Laboratory of Multimedia Trusted Perception and Efficient Computing Ministry of Education of China School of Informatics Xiamen University 361005 China Institute of Artificial Intelligence Xiamen University 361005 China Peng Cheng Laboratory Shenzhen518000 China

Recently, growing interest has been aroused in extending the multimodal capability of large language models (LLMs), e.g., vision-language (VL) learning, which is regarded as the next milestone of artificial general intelligence. However, existing solutions are prohibitively expensive, which not only need to optimize excessive parameters, but also require another large-scale pre-training before VL instruction tuning. In this paper, we propose a novel and affordable solution for the effective VL adaption of LLMs, called Mixture-of-Modality Adaptation (MMA). Instead of using large neural networks to connect the image encoder and LLM, MMA adopts lightweight modules, i.e., adapters, to bridge the gap between LLMs and VL tasks, which also enables the joint optimization of the image and language models. Meanwhile, MMA is also equipped with a routing algorithm to help LLMs achieve an automatic shift between single- and multi-modal instructions without compromising their ability of natural language understanding. To validate MMA, we apply it to a recent LLM called LLaMA and term this formed large vision-language instructed model as LaVIN. To validate MMA and LaVIN, we conduct extensive experiments under two setups, namely multimodal science question answering and multimodal dialogue. The experimental results not only demonstrate the competitive performance and the superior training efficiency of LaVIN than existing multimodal LLMs, but also confirm its great potential as a general-purpose chatbot. More importantly, the actual expenditure of LaVIN is extremely cheap, e.g., only 1.4 training hours with 3.8M trainable parameters, greatly confirming the effectiveness of MMA. Our project is released at https://***/lavin. © 2023, CC BY-NC-SA.

关键词： Computational linguistics

来源：评论

学校读者我要写书评

暂无评论

Aligning and Prompting Everything All at Once for Universal Visual perception

arXiv

引用

arXiv 2023年

作者： Shen, Yunhang Fu, Chaoyou Chen, Peixian Zhang, Mengdan Li, Ke Sun, Xing Wu, Yunsheng Lin, Shaohui Ji, Rongrong Tencent Youtu Lab China School of Computer Science and Technology East China Normal University Shanghai China Key Laboratory of Multimedia Trusted Perception and Efficient Computing Ministry of Education of China Xiamen University China

Vision foundation models have been explored recently to build general-purpose vision systems. However, predominant paradigms, driven by casting instance-level tasks as an object-word alignment, bring heavy cross-modality interaction, which is not effective in prompting object detection and visual grounding. Another line of work that focuses on pixel-level tasks often encounters a large annotation gap of things and stuff, and suffers from mutual interference between foreground-object and background-class segmentation. In stark contrast to the prevailing methods, we present APE, a universal visual perception model for aligning and prompting everything all at once in an image to perform diverse tasks, i.e., detection, segmentation, and grounding, as an instance-level sentence-object matching paradigm. Specifically, APE advances the convergence of detection and grounding by reformulating language-guided grounding as open-vocabulary detection, which efficiently scales up model prompting to thousands of category vocabularies and region descriptions while maintaining the effectiveness of cross-modality fusion. To bridge the granularity gap of different pixel-level tasks, APE equalizes semantic and panoptic segmentation to proxy instance learning by considering any isolated regions as individual instances. APE aligns vision and language representation on broad data with natural and challenging characteristics all at once without task-specific fine-tuning. The extensive experiments on over 160 datasets demonstrate that, with only one-suit of weights, APE outperforms (or is on par with) the state-of-the-art models, proving that an effective yet universal perception for anything aligning and prompting is indeed feasible. Codes and trained models are released at https://***/shenyunhang/APE. © 2023, CC BY.

关键词： Pixels

来源：评论

学校读者我要写书评

暂无评论

Adapting Pre-trained Language Models to Vision-Language Tasksvia Dynamic Visual Prompting

Adapting Pre-trained Language Models to Vision-Language Task...

引用

International Joint Conference on Neural Networks (IJCNN)

作者： Shubin Huang Qiong Wu Yiyi Zhou Ministry of Education of China Key Laboratory of Multimedia Trusted Perception and Efficient Computing Xiamen University P.R. China School of Information Xiamen University P.R. China Institute of Artificial Intelligence Xiamen University P.R. China

ISBN: (数字)9798350359312

ISBN: (纸本)9798350359329

Pre-trained language models (PLMs) have played an increasing role in vision-language (VL) learning, but they usually require a deep multi-modal branch for VL reasoning, resulting in excessive computation and memory overhead. Recently, visual prompting is a feasible way to adapt PLMs to VL tasks, but we notice that the use of all visual tokens will greatly execrate the already high computation, and the token placement is also vital to performance. Based on these observations, we propose a novel transfer learning approach for PLMs in this paper, termed Dynamic Visual Prompting (DVP). Concretely, DVP first deploys a cross-attention module to obtain text-related and compact visual prompt tokens, thereby greatly reducing the input length of PLMs. To obtain the optimal placement, we also equip DVP with a reinforcement-learning based search algorithm, which can automatically merge DVP with PLMs for different VL tasks via a very short search process. In addition, we also combine DVP with the recently popular adapter approach to keep the most parameters of PLMs intact during adaption, which also help PLMs achieve a quick shift between single- and multi-modal tasks. We apply DVP to two representative PLMs, namely BERT and T5, and a recent large language model called LLaMA. Extensive experiments are conducted on a set of VL reasoning benchmarks including VQA2.0, GQA, SNLI-VE and ScienceQA. The experimental results not only show the merits of DVP in performance and efficiency, e.g. +2.28% accuracy and -80% FLOPs on VQA2.0, but also confirm its superiority in adapting pre-trained language models to VL tasks. Our code is released at https://***/hsb1357173526/Dynamic_Visual_Prompting.

关键词： Visualization Adaptation models Computational modeling Heuristic algorithms Transfer learning Redundancy Neural networks

来源：评论

学校读者我要写书评

暂无评论

Weakly Supervised Open-Vocabulary Object Detection

arXiv

引用

arXiv 2023年

作者： Lin, Jianghang Shen, Yunhang Wang, Bingquan Lin, Shaohui Li, Ke Cao, Liujuan Key Laboratory of Multimedia Trusted Perception and Efficient Computing Ministry of Education of China Xiamen University China Tencent Youtu Lab China School of Computer Science and Technology East China Normal University Shanghai China

Despite weakly supervised object detection (WSOD) being a promising step toward evading strong instance-level annotations, its capability is confined to closed-set categories within a single training dataset. In this paper, we propose a novel weakly supervised open-vocabulary object detection framework, namely WSOVOD1, to extend traditional WSOD to detect novel concepts and utilize diverse datasets with only image-level annotations. To achieve this, we explore three vital strategies, including dataset-level feature adaptation, image-level salient object localization, and region-level vision-language alignment. First, we perform data-aware feature extraction to produce an input-conditional coefficient, which is leveraged into dataset attribute prototypes to identify dataset bias and help achieve cross-dataset generalization. Second, a customized location-oriented weakly supervised region proposal network is proposed to utilize high-level semantic layouts from the category-agnostic segment anything model to distinguish object boundaries. Lastly, we introduce a proposal-concept synchronized multiple-instance network, i.e., object mining and refinement with visual-semantic alignment, to discover objects matched to the text embeddings of concepts. Extensive experiments on Pascal VOC and MS COCO demonstrate that the proposed WSOVOD achieves new state-of-the-art compared with previous WSOD methods in both close-set object localization and detection tasks. Meanwhile, WSOVOD enables cross-dataset and open-vocabulary learning to achieve on-par or even better performance than well-established fully-supervised open-vocabulary object detection (FSOVOD). © 2023, CC BY.

关键词： Object detection

来源：评论

学校读者我要写书评

暂无评论

Deep Instruction Tuning for Segment Anything Model

arXiv

引用

arXiv 2024年

作者： Huang, Xiaorui Luo, Gen Zhu, Chaoyang Tong, Bo Zhou, Yiyi Sun, Xiaoshuai Ji, Rongrong Key Laboratory of Multimedia Trusted Perception and Efficient Computing Ministry of Education of China Xiamen University 361005 China Institute of Artificial Intelligence Xiamen University 361005 China The Department of Computer Science and Engineering The Hong Kong University of Science and Technology Kowloon Hong Kong

Recently, Segment Anything Model (SAM) has become a research hotspot in the fields of multimedia and computer vision, which exhibits powerful yet versatile capabilities on various (un) conditional image segmentation tasks. Although SAM can support different types of segmentation prompts, we note that, compared to point- and box-guided segmentations, it performs much worse on text-instructed tasks, e.g., referring image segmentation (RIS). In this paper, we argue that deep text instruction tuning is key to mitigate such shortcoming caused by the shallow fusion scheme in its default light-weight mask decoder. To address this issue, we propose two simple yet effective deep instruction tuning (DIT) methods for SAM, one is end-to-end and the other is layer-wise. With minimal modifications, DITs can directly transform the image encoder of SAM as a stand-alone vision-language learner in contrast to building another deep fusion branch, maximizing the benefit of its superior segmentation capability. Extensive experiments on three highly competitive benchmark datasets of RIS show that a simple end-to-end DIT can improve SAM by a large margin, while the layer-wise DIT can further boost the performance to state-of-the-art with much less data and training expenditures. Our code is released at: https://***/wysnzzzz/DIT. Copyright © 2024, The Authors. All rights reserved.

关键词： Image segmentation

来源：评论

学校读者我要写书评

暂无评论

Diverse Embedding Expansion Network and Low-Light Cross-Modality Benchmark for Visible-Infrared Person Re-identification

Diverse Embedding Expansion Network and Low-Light Cross-Moda...

引用

Conference on Computer Vision and Pattern Recognition (CVPR)

作者： Yukang Zhang Hanzi Wang Fujian Key Laboratory of Sensing and Computing for Smart City School of Informatics Xiamen University P.R. China Key Laboratory of Multimedia Trusted Perception and Efficient Computing Ministry of Education of China Xiamen University P.R. China Shanghai Artificial Intelligence Laboratory Shanghai China

For the visible-infrared person re-identification (VIReID) task, one of the major challenges is the modality gaps between visible (VIS) and infrared (IR) images. However, the training samples are usually limited, while the modality gaps are too large, which leads that the existing methods cannot effectively mine diverse cross-modality clues. To handle this limitation, we propose a novel augmentation network in the embedding space, called diverse embedding expansion network (DEEN). The proposed DEEN can effectively generate diverse embeddings to learn the informative feature representations and reduce the modality discrepancy between the VIS and IR images. Moreover, the VIReID model may be seriously affected by drastic illumination changes, while all the existing VIReID datasets are captured under sufficient illumination without significant light changes. Thus, we provide a low-light cross-modality (LLCM) dataset, which contains 46,767 bounding boxes of 1,064 identities captured by 9 RGB/IR cameras. Extensive experiments on the SYSU-MM01, RegDB and LLCM datasets show the superiority of the proposed DEEN over several other state-of-the-art methods. The code and dataset are released at: https://***/ZYK100/LLCM

关键词：

来源：评论

学校读者我要写书评

暂无评论

Optg: Optimizing Gradient-Driven Criteria in Network Sparsity

SSRN

引用

SSRN 2024年

作者： Zhang, Yuxin Lin, Mingbao Chen, Mengzhao Chao, Fei Tian, Yonghong Ji, Rongrong Key Laboratory of Multimedia Trusted Perception and Efficient Computing Ministry of Education of China Xiamen University China Tecent Youtu Lab China Department of Computer Science and Technology Peking University China Peng Cheng Laboratory China Department of Artificial Intelligence School of Informatics Xiamen University China

Network sparsity receives popularity mostly due to its capability to reduce the complexity of the network. Extensive studies excavate gradient-driven sparsity. Typically, these methods are constructed upon the premise of weight independence, which however, is contrary to the fact that weights are mutually influenced. Thus, their performance remains to be improved. In this paper, we propose to optimize gradient-driven sparsity (OptG) by solving this independence paradox. Our motive comes from the recent advances in supermask training which shows that high-performing sparse subnetworks can be located by simply updating mask values without modifying any weight. We prove that supermask training is to accumulate the criteria of gradientdriven sparsity for both removed and preserved weights, and it can partly solve the independence paradox. Consequently, OptG integrates supermask training into gradient-driven sparsity, and a novel supermask optimizer is further proposed to comprehensively mitigate the independence paradox. Experiments show that OptG can well surpass many existing state-of-the-art competitors, especially at ultra-high sparsity levels. Our project is available at \url{https://***/zyxxmu/OptG}. © 2024, The Authors. All rights reserved.

关键词： Image classification

来源：评论

学校读者我要写书评

暂无评论

Federated Graph Learning for Cross-Domain Recommendation

arXiv

引用

arXiv 2024年

作者： Yang, Ziqi Peng, Zhaopeng Wang, Zihui Qi, Jianzhong Chen, Chaochao Pan, Weike Wen, Chenglu Wang, Cheng Fan, Xiaoliang Fujian Key Laboratory of Sensing and Computing for Smart Cities Xiamen University China Key Laboratory of Multimedia Trusted Perception and Efficient Computing Ministry of Education of China Xiamen University China School of Computing and Information Systems The University of Melbourne Australia College of Computer Science and Technology Zhejiang University Hangzhou China College of Computer Science and Software Engineering Shenzhen University Shenzhen China

Cross-domain recommendation (CDR) offers a promising solution to the data sparsity problem by enabling knowledge transfer across source and target domains. However, many recent CDR models overlook crucial issues such as privacy as well as the risk of negative transfer (which negatively impact model performance), especially in multi-domain settings. To address these challenges, we propose FedGCDR, a novel federated graph learning framework that securely and effectively leverages positive knowledge from multiple source domains. First, we design a positive knowledge transfer module that ensures privacy during inter-domain knowledge transmission. This module employs differential privacy-based knowledge extraction combined with a feature mapping mechanism, transforming source domain embeddings from federated graph attention networks into reliable domain knowledge. Second, we design a knowledge activation module to filter out potential harmful or conflicting knowledge from source domains, addressing the issues of negative transfer. This module enhances target domain training by expanding the graph of the target domain to generate reliable domain attentions and fine-tunes the target model for improved negative knowledge filtering and more accurate predictions. We conduct extensive experiments on 16 popular domains of the Amazon dataset, demonstrating that FedGCDR significantly outperforms state-of-the-art methods. © 2024, CC BY.

关键词： Federated learning

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：