检索结果-内蒙古大学图书馆

Local Manifold Learning for No-Reference Image Quality Assessment

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Gao, Timin Pan, Wensheng Zhang, Yan Zhao, Sicheng Zhang, Shengchuan Zheng, Xiawu Li, Ke Cao, Liujuan Ji, Rongrong Key Laboratory of Multimedia Trusted Perception and Efficient Computing Ministry of Education of China Xiamen University China Harbin Institute of Technology China Tencent Youtu Lab

Contrastive learning has considerably advanced the field of Image Quality Assessment (IQA), emerging as a widely adopted technique. The core mechanism of contrastive learning involves minimizing the distance between quality-similar (positive) examples while maximizing the distance between quality-dissimilar (negative) examples. Despite its successes, current contrastive learning methods often neglect the importance of preserving the local manifold structure. This oversight can result in a high degree of similarity among hard examples within the feature space, thereby impeding effective differentiation and assessment. To address this issue, we propose an innovative framework that integrates local manifold learning with contrastive learning for No-Reference Image Quality Assessment (NR-IQA). Our method begins by sampling multiple crops from a given image, identifying the most visually salient crop. This crop is then used to cluster other crops from the same image as the positive class, while crops from different images are treated as negative classes to increase inter-class distance. Uniquely, our approach also considers non-saliency crops from the same image as intra-class negative classes to preserve their distinctiveness. Additionally, we employ a mutual learning framework, which further enhances the model's ability to adaptively learn and identify visual saliency regions. Our approach demonstrates a better performance compared to state-of-the-art methods in 7 standard datasets, achieving PLCC values of 0.942 (compared to 0.908 in TID2013) and 0.914 (compared to 0.894 in LIVEC). Copyright © 2024, The Authors. All rights reserved.

关键词： Crops

Clover: Towards A Unified Video-Language Alignment and Fusion Model

学校读者我要写书评

暂无评论

Clover: Towards A Unified Video-Language Alignment and Fusio...

Conference on Computer Vision and Pattern Recognition (CVPR)

作者： Jingjia Huang Yinan Li Jiashi Feng Xinglong Wu Xiaoshuai Sun Rongrong Ji Key Laboratory of Multimedia Trusted Perception and Efficient Computing Ministry of Education of China Xiamen University China ByteDance Inc China

Building a universal Video-Language model for solving various video understanding tasks (e.g., text-video retrieval, video question answering) is an open challenge to the machine learning field. Towards this goal, most recent works build the model by stacking uni-modal and cross-modal feature encoders and train it with pair-wise contrastive pre-text tasks. Though offering attractive generality, the resulted models have to compromise between efficiency and performance. They mostly adopt different architectures to deal with different downstream tasks. We find this is because the pair-wise training cannot well align and fuse features from different modalities. We then introduce Clover-a Correlated Video-Language pre-training method-towards a universal Video-Language modelfor solving multiple video understanding tasks with neither performance nor efficiency compromise. It improves cross-modal feature alignment and fusion via a novel tri-modal alignment pre-training task. Additionally, we propose to enhance the tri-modal alignment via incorporating learning from semantic masked samples and a new pair-wise ranking loss. Clover establishes new state-of-the-arts on multiple downstream tasks, including three retrieval tasks for both zero-shot and fine-tuning settings, and eight video question answering tasks. Codes and pre-trained models will be released at https://***/LeeYN-43/Clover.

关键词：

Boosting the Cross-Architecture Generalization of Dataset Distillation through an Empirical Study

学校读者我要写书评

暂无评论

arXiv 2023年

作者： Zhao, Lirui Zhang, Yuxin Chao, Fei Ji, Rongrong Key Laboratory of Multimedia Trusted Perception and Efficient Computing Ministry of Education of China School of Informatics Xiamen University Xiamen China

The poor cross-architecture generalization of dataset distillation greatly weakens its practical significance. This paper attempts to mitigate this issue through an empirical study, which suggests that the synthetic datasets undergo an inductive bias towards the distillation model. Therefore, the evaluation model is strictly confined to having similar architectures of the distillation model. We propose a novel method of EvaLuation with distillation Feature (ELF), which utilizes features from intermediate layers of the distillation model for the cross-architecture evaluation. In this manner, the evaluation model learns from bias-free knowledge therefore its architecture becomes unfettered while retaining performance. By performing extensive experiments, we successfully prove that ELF can well enhance the cross-architecture generalization of current DD methods. Code of this project is at https://***/Lirui-Zhao/ELF. © 2023, CC BY.

关键词： Distillation

Boosting CLIP Adaptation for Image Quality Assessment via Meta-Prompt Learning and Gradient Regularization

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Li, Xudong Huang, Zihao Hu, Runze Zhang, Yan Cao, Liujuan Ji, Rongrong Key Laboratory of Multimedia Trusted Perception and Efficient Computing Ministry of Education of China Xiamen University China School of Information and Electronics Beijing Institute of Technology China

Image Quality Assessment (IQA) remains an unresolved challenge in the field of computer vision, due to complex distortion conditions, diverse image content, and limited data availability. The existing Blind IQA (BIQA) methods heavily rely on extensive human annotations to train models, which is both labor-intensive and costly due to the demanding nature of creating IQA datasets. To mitigate the dependence on labeled samples, this paper introduces a novel Gradient-Regulated Meta-Prompt IQA Framework (GRMP-IQA). This framework aims to fast adapt the powerful visual-language pre-trained model, CLIP, to downstream IQA tasks, significantly improving accuracy in scenarios with limited data. Specifically, the GRMP-IQA comprises two key modules: Meta-Prompt Pre-training Module and Quality-Aware Gradient Regularization. The Meta Prompt Pre-training Module leverages a meta-learning paradigm to pre-train soft prompts with shared meta-knowledge across different distortions, enabling rapid adaptation to various IQA tasks. On the other hand, the Quality-Aware Gradient Regularization is designed to adjust the update gradients during fine-tuning, focusing the model's attention on quality-relevant features and preventing overfitting to semantic information. Extensive experiments on five standard BIQA datasets demonstrate the superior performance to the state-of-the-art BIQA methods under limited data setting, i.e., achieving SRCC values of 0.836 (↑ 7.6% vs. 0.760 on LIVEC) and 0.853 (↑ 4.1% vs. 0.812 on KonIQ). Notably, utilizing just 20% of the training data, our GRMP-IQA outperforms most existing fully supervised BIQA methods. Copyright © 2024, The Authors. All rights reserved.

关键词： Visual languages

AccDiffusion: An Accurate Method for Higher-Resolution Image Generation

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Lin, Zhihang Lin, Mingbao Zhao, Meng Ji, Rongrong Key Laboratory of Multimedia Trusted Perception and Efficient Computing Ministry of Education of China Xiamen University China Skywork AI Singapore Tencent Youtu Lab China

This paper attempts to address the object repetition issue in patch-wise higher-resolution image generation. We propose AccDiffusion, an accurate method for patch-wise higher-resolution image generation without training. An in-depth analysis in this paper reveals an identical text prompt for different patches causes repeated object generation, while no prompt compromises the image details. Therefore, our AccDiffusion, for the first time, proposes to decouple the vanilla image-content-aware prompt into a set of patch-content-aware prompts, each of which serves as a more precise description of an image patch. Besides, AccDiffusion also introduces dilated sampling with window interaction for better global consistency in higher-resolution image generation. Experimental comparison with existing methods demonstrates that our AccDiffusion effectively addresses the issue of repeated object generation and leads to better performance in higher-resolution image generation. Copyright © 2024, The Authors. All rights reserved.

关键词：

Depth-Guided Semi-Supervised Instance Segmentation

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Chen, Xin Hu, Jie Zheng, Xiewu Lin, Jianghang Cao, Liujuan Ji, Rongrong Key Laboratory of Multimedia Trusted Perception and Efficient Computing Ministry of Education of China Xiamen University China Contemporary Amperex Technology Co. Limited China

Semi-Supervised Instance Segmentation (SSIS) aims to leverage an amount of unlabeled data during training. Previous frameworks primarily utilized the RGB information of unlabeled images to generate pseudo-labels. However, such a mechanism often introduces unstable noise, as a single instance can display multiple RGB values. To overcome this limitation, we introduce a Depth-Guided (DG) SSIS framework. This framework uses depth maps extracted from input images, which represent individual instances with closely associated distance values, offering precise contours for distinct instances. Unlike RGB data, depth maps provide a unique perspective, making their integration into the SSIS process complex. To this end, we propose Depth Feature Fusion, which integrates features extracted from depth estimation. This integration allows the model to understand depth information better and ensure its effective utilization. Additionally, to manage the variability of depth images during training, we introduce the Depth Controller. This component enables adaptive adjustments of the depth map, enhancing convergence speed and dynamically balancing the loss weights between RGB and depth maps. Extensive experiments conducted on the COCO and Cityscapes datasets validate the efficacy of our proposed method. Our approach establishes a new benchmark for SSIS, outperforming previous methods. Specifically, our DG achieves 22.29%, 31.47%, and 35.14% mAP for 1%, 5%, and 10% labeled data on the COCO dataset, respectively. Copyright © 2024, The Authors. All rights reserved.

关键词： Machine learning

ControlMLLM: Training-Free Visual Prompt Learning for Multimodal Large Language Models

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Wu, Mingrui Cai, Xinyue Ji, Jiayi Li, Jiale Huang, Oucheng Luo, Gen Fei, Hao Jiang, Guannan Sun, Xiaoshuai Ji, Rongrong Key Laboratory of Multimedia Trusted Perception and Efficient Computing Ministry of Education of China Xiamen University 361005 China National University of Singapore Singapore CATL China

In this work, we propose a training-free method to inject visual prompts into Multimodal Large Language Models (MLLMs) through test time optimization of a learnable latent variable. We observe that attention, as the core module of MLLMs, connects text prompt tokens and visual tokens, ultimately determining the final results. Our approach involves adjusting visual tokens from the MLP output at test time, controlling the attention response to ensure text prompt tokens attend to visual tokens in referring regions. We optimize a learnable latent variable based on an energy function, enhancing the strength of referring regions in the attention map. This enables detailed region description and reasoning without the need for substantial training costs or model retraining. Our method offers a promising direction for integrating referring abilities into MLLMs, and supports referring with box, mask, scribble and point. The results demonstrate that our method exhibits out-of-domain generalization and interpretability. Code: https://***/mrwu-mac/ControlMLLM. © 2024, CC BY.

关键词： Visual languages

Integrating Global Context Contrast and Local Sensitivity for Blind Image Quality Assessment 41

学校读者我要写书评

暂无评论

Integrating Global Context Contrast and Local Sensitivity fo...

41st International Conference on Machine Learning, ICML 2024

作者： Li, Xudong Hu, Runze Zheng, Jingyuan Zhang, Yan Zhang, Shengchuan Zheng, Xiawu Li, Ke Shen, Yunhang Liu, Yutao Dai, Pingyang Ji, Rongrong Key Laboratory of Multimedia Trusted Perception and Efficient Computing Ministry of Education of China Xiamen University 361005 China School of Information and Electronics Beijing Institute of Technology Beijing100080 China School of Medicine Xiamen University China Tencent Youtu Lab China School of Computer Science and Technology Ocean University of China China

Blind Image Quality Assessment (BIQA) mirrors subjective made by human observers. Generally, humans favor comparing relative qualities over predicting absolute qualities directly. However, current BIQA models focus on mining the "local" context, i.e., the relationship between information among individual images and the absolute quality of the image, ignoring the "global" context of the relative quality contrast among different images in the training data. In this paper, we present the Perceptual Context and Sensitivity BIQA (CSIQA), a novel contrastive learning paradigm that seamlessly integrates "global" and "local" perspectives into the BIQA. Specifically, the CSIQA comprises two primary components: 1) A Quality Context Contrastive Learning module, which is equipped with different contrastive learning strategies to effectively capture potential quality correlations in the global context of the dataset. 2) A Quality-aware Mask Attention Module, which employs the random mask to ensure the consistency with visual local sensitivity, thereby improving the model's perception of local distortions. Extensive experiments on eight standard BIQA datasets demonstrate the superior performance to the state-of-the-art BIQA methods. Copyright 2024 by the author(s)

关键词： Contrastive Learning

UniDSeg: Unified Cross-Domain 3D Semantic Segmentation via Visual Foundation Models Prior 38

学校读者我要写书评

暂无评论

UniDSeg: Unified Cross-Domain 3D Semantic Segmentation via V...

38th Conference on Neural Information Processing Systems, NeurIPS 2024

作者： Wu, Yao Xing, Mingwei Zhang, Yachao Luo, Xiaotong Xie, Yuan Qu, Yanyun School of Informatics Xiamen University China Institute of Artificial Intelligence Xiamen University China Key Laboratory of Multimedia Trusted Perception and Efficient Computing Ministry of Education of China Xiamen University China School of Computer Science and Technology East China Normal University China Chongqing Institute of East China Normal University China

3D semantic segmentation using an adapting model trained from a source domain with or without accessing unlabeled target-domain data is the fundamental task in computer vision, containing domain adaptation and domain generalization. The essence of simultaneously solving cross-domain tasks is to enhance the generalizability of the encoder. In light of this, we propose a groundbreaking universal method with the help of off-the-shelf Visual Foundation Models (VFMs) to boost the adaptability and generalizability of cross-domain 3D semantic segmentation, dubbed UniDSeg. Our method explores the VFMs prior and how to harness them, aiming to inherit the recognition ability of VFMs. Specifically, this method introduces layer-wise learnable blocks to the VFMs, which hinges on alternately learning two representations during training: (i) Learning visual prompt. The 3D-to-2D transitional prior and task-shared knowledge is captured from the prompt space, and then (ii) Learning deep query. Spatial Tunability is constructed to the representation of distinct instances driven by prompts in the query space. Integrating these representations into a cross-modal learning framework, UniDSeg efficiently mitigates the domain gap between 2D and 3D modalities, achieving unified cross-domain 3D semantic segmentation. Extensive experiments demonstrate the effectiveness of our method across widely recognized tasks and datasets, all achieving superior performance over state-of-the-art methods. Remarkably, UniDSeg achieves 57.5%/54.4% mIoU on "A2D2/sKITTI" for domain adaptive/generalized tasks. Code is available at https://***/Barcaaaa/UniDSeg. © 2024 Neural information processing systems foundation. All rights reserved.

关键词：