检索结果-内蒙古大学图书馆

Adaptive Feature Selection for No-Reference Image Quality Assessment by Mitigating Semantic Noise Sensitivity

学校读者我要写书评

暂无评论

arXiv 2023年

作者： Li, Xudong Gao, Timin Hu, Runze Zhang, Yan Zhang, Shengchuan Zheng, Xiawu Zheng, Jingyuan Shen, Yunhang Li, Ke Liu, Yutao Dai, Pingyang Ji, Rongrong Key Laboratory of Multimedia Trusted Perception and Efficient Computing Ministry of Education of China Xiamen University 361005 China School of Information and Electronics Beijing Institute of Technology China School of Medicine Xiamen University China Tencent Youtu Lab China School of Computer Science and Technology Ocean University of China China

The current state-of-the-art No-Reference Image Quality Assessment (NR-IQA) methods typically rely on feature extraction from upstream semantic backbone networks, assuming that all extracted features are relevant. However, we make a key observation that not all features are beneficial, and some may even be harmful, necessitating careful selection. Empirically, we find that many image pairs with small feature spatial distances can have vastly different quality scores, indicating that the extracted features may contain a significant amount of quality-irrelevant noise. To address this issue, we propose a Quality-Aware Feature Matching IQA Metric (QFM-IQM) that employs an adversarial perspective to remove harmful semantic noise features from the upstream task. Specifically, QFM-IQM enhances the semantic noise distinguish capabilities by matching image pairs with similar quality scores but varying semantic features as adversarial semantic noise and adaptively adjusting the upstream task’s features by reducing sensitivity to adversarial noise perturbation. Furthermore, we utilize a distillation framework to expand the dataset and improve the model’s generalization ability. Our approach achieves superior performance to the state-of-the-art NR-IQA methods on eight standard IQA datasets. Copyright © 2023, The Authors. All rights reserved.

关键词： Semantics

Speculative Decoding Reimagined for Multimodal Large Language Models

学校读者我要写书评

暂无评论

arXiv 2025年

作者： Lin, Luxi Lin, Zhihang Zeng, Zhanpeng Ji, Rongrong Key Laboratory of Multimedia Trusted Perception and Efficient Computing Ministry of Education of China Xiamen University 361005 China Shanghai Innovation Institute China

This paper introduces Multimodal Speculative Decoding (MSD) to accelerate Multimodal Large Language Models (MLLMs) inference. Speculative decoding has been shown to accelerate Large Language Models (LLMs) without sacrificing accuracy. However, current speculative decoding methods for MLLMs fail to achieve the same speedup as they do for LLMs. To address this, we reimagine speculative decoding specifically for MLLMs. Our analysis of MLLM characteristics reveals two key design principles for MSD: (1) Text and visual tokens have fundamentally different characteristics and need to be processed separately during drafting. (2) Both language modeling ability and visual perception capability are crucial for the draft model. For the first principle, MSD decouples text and visual tokens in the draft model, allowing each to be handled based on its own characteristics. For the second principle, MSD uses a two-stage training strategy: In stage one, the draft model is trained on text-only instruction-tuning datasets to improve its language modeling ability. In stage two, MSD gradually introduces multimodal data to enhance the visual perception capability of the draft model. Experiments show that MSD boosts inference speed by up to 2.29× for LLaVA-1.5-7B and up to 2.46× for LLaVA-1.5-13B on multimodal benchmarks, demonstrating its effectiveness. Our code is available at https://***/LynLucy/MSD. Copyright © 2025, The Authors. All rights reserved.

关键词：

Joint Trajectory Optimization and Power Control for Cognitive UAV-Assisted Secure Communications

学校读者我要写书评

暂无评论

Joint Trajectory Optimization and Power Control for Cognitiv...

IEEE Conference on Global Communications (GLOBECOM)

作者： Can Wang Helin Yang Liang Xiao Huabing Lu Zehui Xiong Department of Information & Communication Engineering Xiamen University Xiamen China Key Laboratory of Multimedia Trusted Perception and Efficient Computing Xiamen University Xiamen China School of Information & Communication Engineering Dalian University of Technology Dalian China Pillar of Information System Technology & Design Singapore University of Technology and Design Singapore

Cognitive unmanned aerial vehicle (UAV) communication systems combine benefits of both the cognitive radio and UAV, which improves the spectral efficiency and communication coverage area. However, high-quality air-ground channel links maybe more vulnerable to potential eavesdropping or jamming attacks. Thus, this paper proposes a secure transmission approach assisted by deploying a cooperative UAV to transmit artificial noise to jam an active eavesdropper, in order to maximize the system secrecy rate under the quality of service (QoS) requirement of a primary device. Specifically, we jointly optimize the flight trajectory and transmission power of the cooperative jammer to maximize the system's secrecy rate under strict constraints. To achieve this, we convert the non-convex problem into an approximately convex problem using the block coordinate descent algorithm and successive convex approximation method. Simulation results show that compared to existing algorithms, the proposed algorithm in this study can significantly improve the system's secrecy rate.

关键词：

Dynamic Sparse No Training: Training-Free Fine-tuning for Sparse LLMs

学校读者我要写书评

暂无评论

arXiv 2023年

作者： Zhang, Yuxin Zhao, Lirui Lin, Mingbao Sun, Yunyun Yao, Yiwu Han, Xingjia Tanner, Jared Liu, Shiwei Ji, Rongrong Key Laboratory of Multimedia Trusted Perception and Efficient Computing Ministry of Education of China Xiamen University China Tencent Youtu Lab China Huawei Technologies China University of Oxford United Kingdom University of Texas Austin United States Eindhoven University of Technology Netherlands Institute of Artificial Intelligence Xiamen University China

The ever-increasing large language models (LLMs), though opening a potential path for the upcoming artificial general intelligence, sadly drops a daunting obstacle on the way towards their on-device deployment. As one of the most well-established pre-LLMs approaches in reducing model complexity, network pruning appears to lag behind in the era of LLMs, due mostly to its costly fine-tuning (or re-training) necessity under the massive volumes of model parameter and training data. To close this industry-academia gap, we introduce Dynamic Sparse No Training (DS⃠T1), a training-free fine-tuning approach that slightly updates sparse LLMs without the expensive backpropagation and any weight updates. Inspired by the Dynamic Sparse Training, DST minimizes the reconstruction error between the dense and sparse LLMs, in the fashion of performing iterative weight pruning-and-growing on top of sparse LLMs. To accomplish this purpose, DS⊘T particularly takes into account the anticipated reduction in reconstruction error for pruning and growing, as well as the variance w.r.t. different input data for growing each weight. This practice can be executed efficiently in linear time since its obviates the need of backpropagation for fine-tuning LLMs. Extensive experiments on LLaMA-V1/V2, Vicuna, and OPT across various benchmarks demonstrate the effectiveness of DS⊘T in enhancing the performance of sparse LLMs, especially at high sparsity levels. For instance, DS⊘T is able to outperform the state-of-the-art Wanda by 26.79 perplexity at 70% sparsity with LLaMA-7B. Our paper offers fresh insights into how to fine-tune sparse LLMs in an efficient training-free manner and open new venues to scale the great potential of sparsity to LLMs. Codes are available at https://***/zyxxmu/DSnoT. © 2023, CC BY.

关键词： Backpropagation

Less is More: Learning Reference Knowledge Using No-Reference Image Quality Assessment

学校读者我要写书评

暂无评论

arXiv 2023年

作者： Li, Xudong Zheng, Jingyuan Zheng, Xiawu Hu, Runze Zhang, Enwei Gao, Yuting Shen, Yunhang Li, Ke Liu, Yutao Dai, Pingyang Zhang, Yan Ji, Rongrong Key Laboratory of Multimedia Trusted Perception and Efficient Computing Ministry of Education of China Xiamen University China School of Medicine Xiamen University China Peng Cheng Laboratory Shenzhen China School of Information and Electronics Beijing Institute of Technology China Tencent Youtu Lab China School of Computer Science and Technology Ocean University of China China

Image Quality Assessment (IQA) with reference images have achieved great success by imitating the human vision system, in which the image quality is effectively assessed by comparing the query image with its pristine reference image. However, for the images in the wild, it is quite difficult to access accurate reference images. We argue that it is possible to learn reference knowledge under the No-Reference Image Quality Assessment (NR-IQA) setting, which is effective and efficient empirically. Concretely, by innovatively introducing a novel feature distillation method in IQA, we propose a new framework to learn comparative knowledge from non-aligned reference images. And then, to achieve fast convergence and avoid overfitting, we further propose an inductive bias regularization. Such a framework not only solves the congenital defects of NR-IQA but also improves the feature extraction framework, enabling it to express more abundant quality information. Surprisingly, our method utilizes less input while obtaining a more significant improvement compared to the teacher models. Extensive experiments on 8 standard NR-IQA datasets demonstrate the superior performance to the state-of-the-art NR-IQA methods, i.e., achieving the PLCC values of 0.917 (↑ 3.3% vs. 0.884 in LIVEC) and 0.686 (↑ 2.5% vs. 0.661 in LIVEFB). Copyright © 2023, The Authors. All rights reserved.

关键词： Distillation

Spatial Re-parameterization for N:M Sparsity

学校读者我要写书评

暂无评论

arXiv 2023年

作者： Zhang, Yuxin Xu, Mingliang Tian, Yonghong Ji, Rongrong Key Laboratory of Multimedia Trusted Perception and Efficient Computing Ministry of Education of China Xiamen University Xiamen China School of Computer and Artificial Intelligence Zhengzhou University Zhengzhou China Engineering Research Center of Intelligent Swarm Systems Zhengzhou China School of Electronics Engineering and Computer Science Peking University Beijing China

This paper presents a Spatial Re-parameterization (SpRe) method for the N:M sparsity. SpRe stems from an observation regarding the restricted variety in spatial sparsity presented in N:M sparsity compared with unstructured sparsity. Particularly, N:M sparsity exhibits a fixed sparsity rate within the spatial domains due to its distinctive pattern that mandates N non-zero components among M successive weights in the input channel dimension of convolution filters. On the contrary, we observe that conventional unstructured sparsity displays a substantial divergence in sparsity across the spatial domains, which we experimentally verify to be very crucial for its robust performance retention compared with N:M sparsity. Therefore, SpRe employs the spatial-sparsity distribution of unstructured sparsity by assigning an extra branch in conjunction with the original N:M branch at training time, which allows the N:M sparse network to sustain a similar distribution of spatial sparsity with unstructured sparsity. During inference, the extra branch can be further re-parameterized into the main N:M branch, without exerting any distortion on the sparse pattern or additional computation costs. SpRe has achieved a commendable feat by matching the performance of N:M sparsity methods with state-of-the-art unstructured sparsity methods across various benchmarks. Our project is available at https://***/zyxxmu/SpRE. © 2023, CC BY.

关键词： Convolution

Oracle Bone Inscriptions Multi-modal Dataset

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Li, Bang Luo, Donghao Liang, Yujie Yang, Jing Ding, Zengmao Peng, Xu Jiang, Boyuan Han, Shengwei Sui, Dan Qin, Peichao Wu, Pian Wang, Chaoyang Qi, Yun Jin, Taisong Wang, Chengjie Huang, Xiaoming Shu, Zhan Ji, Rongrong Liu, Yongge Wu, Yunsheng Key Laboratory of Oracle Bone Inscriptions Information Processing Ministry of Education of China Anyang Normal University China Youtu Lab Tencent China Key Laboratory of Multimedia Trusted Perception and Effcient Computing Ministry of Education of China Xiamen University China Digital Culture Lab Tencent SSV Sustainable Social Value Dept University of Cambridge United Kingdom

Oracle bone inscriptions(OBI) is the earliest developed writing system in China, bearing invaluable written exemplifications of early Shang history and paleography. However, the task of deciphering OBI, in the current climate of the scholarship, can prove extremely challenging. Out of the 4,500 oracle bone characters excavated, only a third have been successfully identified. Therefore, leveraging the advantages of advanced AI technology to assist in the decipherment of OBI is a highly essential research topic. However, fully utilizing AI's capabilities in these matters is reliant on having a comprehensive and high-quality annotated OBI dataset at hand whereas most existing datasets are only annotated in just a single or a few dimensions, limiting the value of their potential application. For instance, the Oracle-MNIST dataset only offers 30k images classified into 10 categories. Therefore, this paper proposes an Oracle Bone Inscriptions Multi-modal Dataset(OBIMD), which includes annotation information for 10,077 pieces of oracle bones. Each piece has two modalities: pixel-level aligned rubbings and facsimiles. The dataset annotates the detection boxes, character categories, transcriptions, corresponding inscription groups, and reading sequences in the groups of each oracle bone character, providing a comprehensive and high-quality level of annotations. This dataset can be used for a variety of AI-related research tasks relevant to the field of OBI, such as OBI Character Detection and Recognition, Rubbing Denoising, Character Matching, Character Generation, Reading Sequence Prediction, Missing Characters Completion task and so on. We believe that the creation and publication of a dataset like this will help significantly advance the application of AI algorithms in the field of OBI research. Copyright © 2024, The Authors. All rights reserved.

关键词：

Training-Free Large Model Priors for Multiple-in-One Image Restoration

学校读者我要写书评

暂无评论

arXiv 2024年

作者： He, Xuanhua Li, Lang Wang, Yingying Zheng, Hui Cao, Ke Yan, keyu Li, Rui Xie, Chengjun Zhang, Jie Zhou, Man Hefei Institutes of Physical Science Chinese Academy of Sciences Hefei230031 China University of Science and Technology of China Hefei230026 China Key Laboratory of Multimedia Trusted Perception and Efficient Computing Xiamen University Xiamen361102 China University of Science and Technology of China China The Intelligent Agriculture Engineering Laboratory of Anhui Province Institute of Intelligent Machines Hefei Institutes of Physical Science Chinese Academy of Sciences Hefei230031 China

Image restoration aims to reconstruct the latent clear images from their degraded versions. Despite the notable achievement, existing methods predominantly focus on handling specific degradation types and thus require specialized models, impeding real-world applications in dynamic degradation scenarios. To address this issue, we propose Large Model Driven Image Restoration framework (LMDIR), a novel multiple-in-one image restoration paradigm that leverages the generic priors from large multi-modal language models (MMLMs) and the pretrained diffusion models. In detail, LMDIR integrates three key prior knowledges: 1) global degradation knowledge from MMLMs, 2) scene-aware contextual descriptions generated by MMLMs, and 3) fine-grained high-quality reference images synthesized by diffusion models guided by MMLM descriptions. Standing on above priors, our architecture comprises a query-based prompt encoder, degradation-aware transformer block injecting global degradation knowledge, content-aware transformer block incorporating scene description, and reference-based transformer block incorporating fine-grained image priors. This design facilitates single-stage training paradigm to address various degradations while supporting both automatic and user-guided restoration. Extensive experiments demonstrate that our designed method outperforms state-of-the-art competitors on multiple evaluation benchmarks. Copyright © 2024, The Authors. All rights reserved.

关键词： Image reconstruction

Collaborative Inference for MEC Services Based on Multimodal Deep Neural Network

学校读者我要写书评

暂无评论

Collaborative Inference for MEC Services Based on Multimodal...

IEEE International Conference on Communications in China (ICCC)

作者： Yu Liu Chuxuan Wang Yilin Xiao Zefang Lv Liang Xiao Xiangyang Ji Dept. of Institute of Artificial Intelligence Xiamen University Xiamen China Ministry of Education of China Key Laboratory of Multimedia Trusted Perception and Efficient Computing Xiamen University China Shenzhen Institute of Artificial Intelligence and Robotics for Society Shenzhen China School of Informatics Xiamen University Xiamen China Department of Automation Tsinghua University Beijing China

Collaborative inference can effectively ease the computational burden imposed on mobile devices by deep neural networks (DNNs), such as employing multimodal DNNs to achieve a comprehensive understanding of the environment. Although reinforcement learning (RL)-based collaborative inference adapts to dynamic environments and enhances inference performance for DNNs, especially those with chain-structured architectures, the policy selection for multimodal DNNs must account for multiple computation-intensive feature encoders. This paper proposes a hierarchical reinforcement learning (RL)-based collaborative inference scheme for multimodal DNNs that concurrently determines the partition point and identifies the appropriate DNN-based feature encoder for each modality within the mobile edge computing system. The hierarchical structure designed for policy selection has alleviated the exponential growth of the modalities. Moreover, the mobile device maintains a confidence score based on probabilities of predefined classes to assess the risk of inference policies that results in low inference accuracy and thus avoids exploring the risk policy. Simulation results demonstrate that our proposed scheme surpasses the Neurosurgeon benchmark in reducing inference latency and energy consumption, all while maintaining a high level of confidence score without significant degradation.

关键词：