检索结果-内蒙古大学图书馆

38th Conference on Neural Information Processing Systems, NeurIPS 2024

作者： Wu, Mingrui Cai, Xinyue Ji, Jiayi Li, Jiale Huang, Oucheng Luo, Gen Fei, Hao Jiang, Guannan Sun, Xiaoshuai Ji, Rongrong Key Laboratory of Multimedia Trusted Perception and Efficient Computing Ministry of Education of China Xiamen University 361005 China National University of Singapore Singapore CATL Canada

In this work, we propose a training-free method to inject visual prompts into Multimodal Large Language Models (MLLMs) through learnable latent variable optimization. We observe that attention, as the core module of MLLMs, connects text prompt tokens and visual tokens, ultimately determining the final results. Our approach involves adjusting visual tokens from the MLP output during inference, controlling the attention response to ensure text prompt tokens attend to visual tokens in referring regions. We optimize a learnable latent variable based on an energy function, enhancing the strength of referring regions in the attention map. This enables detailed region description and reasoning without the need for substantial training costs or model retraining. Our method offers a promising direction for integrating referring abilities into MLLMs, and supports referring with box, mask, scribble and point. The results demonstrate that our method exhibits out-of-domain generalization and interpretability. Code: https://***/mrwu-mac/ControlMLLM. © 2024 Neural information processing systems foundation. All rights reserved.

关键词：

来源：评论

学校读者我要写书评

暂无评论

Adaptive Selection based Referring Image Segmentation 24

Adaptive Selection based Referring Image Segmentation

引用

32nd ACM International Conference on multimedia, MM 2024

作者： Yue, Pengfei Lin, Jianghang Zhang, Shengchuan Hu, Jie Lu, Yilin Niu, Hongwei Ding, Haixin Zhang, Yan Jiang, Guannan Cao, Liujuan Ji, Rongrong Key Laboratory of Multimedia Trusted Perception and Efficient Computing Ministry of Education of China Xiamen China Contemporary Amperex Technology Co. Limited Ningde China

ISBN: (纸本)9798400706868

Referring image segmentation (RIS) aims to segment a particular region based on a specific expression. Existing one-stage methods have explored various fusion strategies, yet they encounter two significant issues. Primarily, most methods rely on manually selected visual features from the visual encoder layers. Moreover, the direct fusion of word-level features into coarse aligned features disrupts the established vision-language alignment. In this paper, we introduce an innovative framework for RIS that seeks to overcome these challenges with adaptive alignment of vision and language features, termed the Adaptive Selection with Dual Alignment (ASDA). ASDA innovates in two aspects. Firstly, we design an Adaptive Feature Selection and Fusion (AFSF) module to dynamically select visual features focusing on different regions related to various descriptions. AFSF is equipped with scale-wise feature aggregator to provide hierarchically coarse features that preserve crucial low-level details. Secondly, a Word Guided Dual-Branch Aligner (WGDA) is leveraged to integrate coarse features with linguistic cues by word-guided attention, which effectively addresses the common issue of vision-language misalignment. Extensive experimental results demonstrate that our ASDA framework surpasses state-of-the-art methods on RefCOCO, RefCOCO+ and G-Ref benchmark. © 2024 ACM.

关键词： Image segmentation

来源：评论

学校读者我要写书评

暂无评论

I2EBench: A Comprehensive Benchmark for Instruction-based Image Editing 38

I2EBench: A Comprehensive Benchmark for Instruction-based Im...

引用

38th Conference on Neural Information Processing Systems, NeurIPS 2024

作者： Ma, Yiwei Ji, Jiayi Ye, Ke Lin, Weihuang Wang, Zhibin Zheng, Yonghan Zhou, Qiang Sun, Xiaoshuai Ji, Rongrong Key Laboratory of Multimedia Trusted Perception and Efficient Computing Ministry of Education of China Xiamen University 361005 China Inf Tech Company Hangzhou310000 China

Significant progress has been made in the field of Instruction-based Image Editing (IIE). However, evaluating these models poses a significant challenge. A crucial requirement in this field is the establishment of a comprehensive evaluation benchmark for accurately assessing editing results and providing valuable insights for its further development. In response to this need, we propose I2EBench, a comprehensive benchmark designed to automatically evaluate the quality of edited images produced by IIE models from multiple dimensions. I2EBench consists of 2,000+ images for editing, along with 4,000+ corresponding original and diverse instructions. It offers three distinctive characteristics: 1) Comprehensive Evaluation Dimensions: I2EBench comprises 16 evaluation dimensions that cover both high-level and low-level aspects, providing a comprehensive assessment of each IIE model. 2) Human perception Alignment: To ensure the alignment of our benchmark with human perception, we conducted an extensive user study for each evaluation dimension. 3) Valuable Research Insights: By analyzing the advantages and disadvantages of existing IIE models across the 16 dimensions, we offer valuable research insights to guide future development in the field. We will open-source I2EBench, including all instructions, input images, human annotations, edited images from all evaluated methods, and a simple script for evaluating the results from new IIE models. The code, dataset and generated images from all IIE models are provided in github: https://***/cocoshe/I2EBench. © 2024 Neural information processing systems foundation. All rights reserved.

关键词：

来源：评论

学校读者我要写书评

暂无评论

P2SAM: Probabilistically Prompted SAMs Are efficient Segmentator for Ambiguous Medical Images 24

P2SAM: Probabilistically Prompted SAMs Are Efficient Segment...

引用

32nd ACM International Conference on multimedia, MM 2024

作者： Huang, Yuzhi Li, Chenxin Lin, Zixu Liu, Hengyu Xu, Haote Liu, Yifan Huang, Yue Ding, Xinghao Tu, Xiaotong Yuan, Yixuan School of Informatics Xiamen University China The Chinese University of Hong Kong Hong Kong Key Laboratory of Multimedia Trusted Perception and Efficient Computing Xiamen University China

ISBN: (纸本)9798400706868

Generating diverse plausible outputs from a single input is crucial for addressing visual ambiguities, exemplified in medical imaging where experts may provide varying semantic segmentation annotations for the same *** methods handles ambiguous segmentation relying on probabilistic modeling and extensive multi-output annotated data while often struggles with limited ambiguously labeled datasets common in real-world *** surmount the challenge, we propose P2SAM, a novel framework that leverages the Segment Anything Model (SAM)'s prior knowledge for ambiguous object segmentation. By transforming SAM's sensitivity to prompts into an advantage, we introduce a prior probabilistic space for *** results show that P2SAM significantly enhances medical segmentation precision and diversity using minimal ambiguously annotated samples. Benchmarking against state-of-the-art methods demonstrates superior performance with just 5.5% of the training data (+12% Dmax). This approach marks a significant advancement towards deploying probabilistic models in data-limited real-world scenarios. © 2024 ACM.

关键词： Semantic Segmentation

来源：评论

学校读者我要写书评

暂无评论

Semi-supervised Visible-Infrared Person Re-identification via Modality Unification and Confidence Guidance 24

Semi-supervised Visible-Infrared Person Re-identification vi...

引用

32nd ACM International Conference on multimedia, MM 2024

作者： Zheng, Xiying Zhang, Yukang Lu, Yang Wang, Hanzi Fujian Key Laboratory of Sensing and Computing for Smart City School of Informatics Xiamen University Xiamen China Key Laboratory of Multimedia Trusted Perception and Efficient Computing Ministry of Education of China Xiamen University Xiamen China

ISBN: (纸本)9798400706868

Semi-supervised visible-infrared person re-identification (SSVI-ReID) aims to match pedestrian images of the same identity from different modalities (visible and infrared) while only annotating visible images, which is highly related to multimedia and multi-modal processing. Existing works primarily focus on assigning accurate pseudo-labels to infrared images, but overlook the two key challenges: erroneous pseudo-labels and large modality discrepancy. To alleviate these issues, this paper proposes a novel Modality-Unified and Confidence-Guided (MUCG) semi-supervised learning method. Specifically, we first propose a Dynamic Intermediate Modality Generation (DIMG) module, which transfers knowledge from labeled visible images to unlabeled infrared images, enhancing the pseudo-label quality and bridging the modality discrepancy. Meanwhile, we propose a Weighted Identification Loss (WIL) that can reduce the model's dependence on erroneous labels by using confidence weighting. Moreover, an effective Modality Consistency Loss (MCL) is proposed to narrow the distribution of visible and infrared features, further narrowing the modality discrepancy and enabling the learning of modality-unified features. Extensive experiments show that the proposed MUCG has significant advantages in improving the performance of the SSVI-ReID task, surpassing the current state-of-the-art methods by a significant margin. © 2024 ACM.

关键词： Self-supervised learning

来源：评论

学校读者我要写书评

暂无评论

DiffusionFake: Enhancing Generalization in Deepfake Detection via Guided Stable Diffusion 38

DiffusionFake: Enhancing Generalization in Deepfake Detectio...

引用

38th Conference on Neural Information Processing Systems, NeurIPS 2024

作者： Sun, Ke Chen, Shen Yao, Taiping Liu, Hong Sun, Xiaoshuai Ding, Shouhong Ji, Rongrong Key Laboratory of Multimedia Trusted Perception and Efficient Computing Ministry of Education of China Xiamen University 361005 China Youtu Lab Tencent China Osaka University Japan

The rapid progress of Deepfake technology has made face swapping highly realistic, raising concerns about the malicious use of fabricated facial content. Existing methods often struggle to generalize to unseen domains due to the diverse nature of facial manipulations. In this paper, we revisit the generation process and identify a universal principle: Deepfake images inherently contain information from both source and target identities, while genuine faces maintain a consistent identity. Building upon this insight, we introduce DiffusionFake, a novel plug-and-play framework that reverses the generative process of face forgeries to enhance the generalization of detection models. DiffusionFake achieves this by injecting the features extracted by the detection model into a frozen pre-trained Stable Diffusion model, compelling it to reconstruct the corresponding target and source images. This guided reconstruction process constrains the detection network to capture the source and target related features to facilitate the reconstruction, thereby learning rich and disentangled representations that are more resilient to unseen forgeries. Extensive experiments demonstrate that DiffusionFake significantly improves cross-domain generalization of various detector architectures without introducing additional parameters during inference. Code are available in https://***/skJack/***. © 2024 Neural information processing systems foundation. All rights reserved.

关键词：

来源：评论

学校读者我要写书评

暂无评论

GLATrack: Global and Local Awareness for Open-Vocabulary Multiple Object Tracking 24

GLATrack: Global and Local Awareness for Open-Vocabulary Mul...

引用

32nd ACM International Conference on multimedia, MM 2024

作者： Li, Guangyao Jian, Yajun Yan, Yan Wang, Hanzi Key Laboratory of Multimedia Trusted Perception and Efficient Computing Ministry of Education of China Xiamen University Xiamen China Fujian Key Laboratory of Sensing and Computing for Smart City School of Informatics Xiamen University Xiamen China

ISBN: (纸本)9798400706868

Open-vocabulary multi-object tracking (MOT) aims to track arbitrary objects encountered in the real world beyond the training set. However, recent methods rely solely on instance-level detection and association of novel objects, which may not consider the valuable fine-grained semantic representations of the targets within key and reference frames. In this paper, we propose a Global and Local Awareness open-vocabulary MOT method (GLATrack), which learns to tackle the task of real-world MOT from both global and instance-level perspectives. Specifically, we introduce a region-aware feature enhancement module to refine global knowledge for complementing local target information, which enhances semantic representation and bridges the distribution gap between the image feature map and the pooled regional features. We propose a bidirectional semantic complementarity strategy to mitigate semantic misalignment arising from missing target information in key frames, which dynamically selects valuable information within reference frames to enrich object representation during the knowledge distillation process. Furthermore, we introduce an appearance richness measurement module to provide appropriate representations for targets with different appearances. The proposed method gains an improvement of 6.9% in TETA and 5.6% in mAP on the large-scale TAO benchmark. © 2024 ACM.

关键词： Semantics

来源：评论

学校读者我要写书评

暂无评论

SAM as the Guide: Mastering Pseudo-Label Refinement in Semi-Supervised Referring Expression Segmentation 41

SAM as the Guide: Mastering Pseudo-Label Refinement in Semi-...

引用

41st International Conference on Machine Learning, ICML 2024

作者： Yang, Danni Ji, Jiayi Ma, Yiwei Guo, Tianyu Wang, Haowei Sun, Xiaoshuai Ji, Rongrong Key Laboratory of Multimedia Trusted Perception and Efficient Computing Ministry of Education of China School of Informatics Xiamen University 361005 China Youtu Lab Tencent Shanghai China

In this paper, we introduce SemiRES, a semi-supervised framework that effectively leverages a combination of labeled and unlabeled data to perform RES. A significant hurdle in applying semi-supervised techniques to RES is the prevalence of noisy pseudo-labels, particularly at the boundaries of objects. SemiRES incorporates the Segment Anything Model (SAM), renowned for its precise boundary demarcation, to improve the accuracy of these pseudo-labels. Within SemiRES, we offer two alternative matching strategies: IoU-based Optimal Matching (IOM) and Composite Parts Integration (CPI). These strategies are designed to extract the most accurate masks from SAM's output, thus guiding the training of the student model with enhanced precision. In instances where a precise mask cannot be matched from the available candidates, we develop the Pixel-Wise Adjustment (PWA) strategy, guiding the student model's training directly by the pseudo-labels. Extensive experiments on three RES benchmarks-RefCOCO, RefCOCO+, and G-Ref reveal its superior performance compared to fully supervised methods. Remarkably, with only 1% labeled data, our SemiRES outperforms the supervised baseline by a large margin, e.g. +18.64% gains on RefCOCO val set. The project code is available at https://***/nini0919/SemiRES. Copyright 2024 by the author(s)

关键词： Labeled data

来源：评论

学校读者我要写书评

暂无评论

MMAPS: End-to-End Multi-Grained Multi-Modal Attribute-Aware Product Summarization 30

MMAPS: End-to-End Multi-Grained Multi-Modal Attribute-Aware ...

引用

Joint 30th International Conference on Computational Linguistics and 14th International Conference on Language Resources and Evaluation, LREC-COLING 2024

作者： Chen, Tao Lin, Ze Li, Hui Ji, Jiayi Zhou, Yiyi Li, Guanbin Ji, Rongrong Key Laboratory of Multimedia Trusted Perception and Efficient Computing Ministry of Education of China Xiamen University China School of Computer Science and Engineering Sun Yat-sen University China

ISBN: (纸本)9782493814104

Given the long textual product information and the product image, Multi-modal Product Summarization (MPS) aims to increase customers' desire to purchase by highlighting product characteristics with a short textual summary. Existing MPS methods can produce promising results. Nevertheless, they still 1) lack end-to-end product summarization, 2) lack multi-grained multi-modal modeling, and 3) lack multi-modal attribute modeling. To improve MPS, we propose an end-to-end multi-grained multi-modal attribute-aware product summarization method (MMAPS) for generating high-quality product summaries in e-commerce. MMAPS jointly models product attributes and generates product summaries. We design several multi-grained multi-modal tasks to better guide the multi-modal learning of MMAPS. Furthermore, we model product attributes based on both text and image modalities so that multi-modal product characteristics can be manifested in the generated summaries. Extensive experiments on a real large-scale Chinese e-commence dataset demonstrate that our model outperforms state-of-the-art product summarization methods w.r.t. several summarization metrics. Our code is publicly available at: https://***/KDEGroup/MMAPS. © 2024 ELRA Language Resource Association: CC BY-NC 4.0.

关键词： Large datasets

来源：评论

学校读者我要写书评

暂无评论

Learning Based Interference Coordination for Maritime Communications

引用

China Communications 2025年第4期22卷 356-374页

作者： Liu Chuhuan Xiao Liang Chen Yifan Li Siyao Yang Helin Lyu Zefang Department of Information and Communication Engineering Xiamen UniversityXiamen 361005China Key Laboratory of Multimedia Trusted Perception and Efficient Computing Ministry of Education of ChinaXiamen UniversityXiamen 361005China

With the boom in maritime activities,the need for highly reliable maritime communication is becoming urgent,which is an important component of 5G/6G communication ***,the bandwidth reuse characteristic of 5G/6G networks will inevitably lead to severe interference,resulting in degradation in the communication performance of maritime *** this paper,we propose a safe deep reinforcement learning based interference coordination scheme to jointly optimize the power control and bandwidth allocation in maritime communication systems,and exploit the quality-of-service requirements of users as the risk value references to evaluate the communication *** particular,this scheme designs a deep neural network to select the communication policies through the evaluation network and update the parameters using the target network,which improves the communication performance and speeds up the convergence ***,the Nash equilibrium of the interference coordination game and the computational complexity of the proposed scheme are *** and experimental results verify the performance gain of the proposed scheme compared with benchmarks.

关键词： bandwidth allocation interference coordination maritime communication power control rein-forcement learning

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：