检索结果-内蒙古大学图书馆

M-RAT: a Multi-grained Retrieval Augmentation Transformer for image Captioning 17th

学校读者我要写书评

暂无评论

M-RAT: a Multi-grained Retrieval Augmentation Transformer f...

17th Asian Conference on Computer Vision, ACCV 2024

作者： Song, Jiayan Pan, Renjie Zhou, Jun Yang, Hua Institute of Image Communication and Network Engineering Shanghai Jiao Tong University Shanghai200240 China Shanghai Key Lab of Digital Media Processing and Transmission Shanghai200240 China

ISBN: (纸本)9789819609079

Current encoder-decoder methods for image captioning mai-nly consist of an object detection module (two-stage), or rely on big models with large-scale datasets to improve the effectiveness, which leads to increasing computation costs and cannot introduce new external knowledge. In this paper, we propose a novel end-to-end method Multi-grained Retrieval Augmentation Transformer (M-RAT) that innovatively fuses retrieved text derived from a changeable datastore with input visual feature through a Multi-modal Aligned Encoder, and introduce a specialized attention mechanism, Multi-MSA, to exploit both local and global interactions for delicate fine-grained details. Additionally, we enhance the decoder generation ability by employing low-level and high-level fused embeddings. Experiments demonstrate that M-RAT achieves comparable performance to state-of-the-art baselines with remarkable accuracy and details, as well as showing excellent domain adaptability for novel objects. © The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd. 2025.

关键词： Decoding

Full-Reference and No-Reference Quality Assessment for Video Frame Interpolation

学校读者我要写书评

暂无评论

IEEE Transactions on Circuits and Systems for Video Technology 2025年

作者： Han, Jinliang Min, Xiongkuo Jia, Jun Gao, Yixuan Liu, Xiaohong Zhai, Guangtao Shanghai Jiao Tong University Institute of Image Communication and Network Engineering Shanghai200240 China

Video frame interpolation (VFI) synthesizes new frames from original video frames to produce high frame-rate videos and enhance their visual appeal. The quality of these interpolated frames significantly affects the perceptual experience of the synthesized video. Recent research in VFI has increasingly focused on perceptual quality of the interpolated frames and the overall video. However, most existing quality metrics do not align well with human perceptual experiences and often suffer from unnatural artifacts in the interpolated frames. Consequently, there is an urgent need for VFI video quality assessment (VFIVQA) methods to assess the quality of the synthesized videos. In this paper, we propose both a full-reference (FR) method and a no-reference (NR) method for VFIVQA. The FR method employs two feature extraction blocks to measure continuous frame changes, extracting flow features with short temporal spans and motion features with long temporal spans. By calculating multilevel similarities in the temporal dimension of 3D convolutional neural networks and fusing these similarity features, the quality score of the VFI video is obtained from the quality regression network. Since the flow feature extraction block does not utilize the reference VFI video, the proposed NR method consists solely of this feature block. Extensive validation on several VFIVQA datasets demonstrates that the proposed methods outperform state-of-the-art FR and NR methods. © 1991-2012 IEEE.

关键词： Feature extraction

3DGCQA: A Quality Assessment Database for 3D AI-Generated Contents

学校读者我要写书评

暂无评论

3DGCQA: A Quality Assessment Database for 3D AI-Generated Co...

International Conference on Acoustics, Speech, and Signal Processing (ICASSP)

作者： Yingjie Zhou Zicheng Zhang Farong Wen Jun Jia Yanwei Jiang Xiaohong Liu Xiongkuo Min Guangtao Zhai Institute of Image Communication and Network Engineering Shanghai Jiao Tong University China

ISBN: (数字)9798350368741

ISBN: (纸本)9798350368758

Although 3D generated content (3DGC) offers advantages in reducing production costs and accelerating design timelines, its quality often falls short when compared to 3D professionally generated content. Common quality issues frequently affect 3DGC, highlighting the importance of timely and effective quality assessment. Such evaluations not only ensure a higher standard of 3DGCs for end-users but also provide critical insights for advancing generative technologies. To address existing gaps in this domain, this paper introduces a novel 3DGC quality assessment dataset, 3DGCQA, built using 7 representative Text-to-3D generation methods. During the dataset’s construction, 50 fixed prompts are utilized to generate contents across all methods, resulting in the creation of 313 textured meshes that constitute the 3DGCQA dataset. The visualization intuitively reveals the presence of 6 common distortion categories in the generated 3DGCs. To further explore the quality of the 3DGCs, subjective quality assessment is conducted by evaluators, whose ratings reveal significant variation in quality across different generation methods. Additionally, several objective quality assessment algorithms are tested on the 3DGCQA dataset. The results expose limitations in the performance of existing algorithms and underscore the need for developing more specialized quality assessment methods. To provide a valuable resource for future research and development in 3D content generation and quality assessment, the dataset has been open-sourced in https://***/zyj-2000/3DGCQA.

关键词： Three-dimensional displays Generative AI Databases Signal processing algorithms Production Data structures Quality assessment Speech processing Standards Research and development

Rate-Aware Learned Speech Compression

学校读者我要写书评

暂无评论

arXiv 2025年

作者： Xu, Jun Cheng, Zhengxue Chi, Guangchuan Liu, Yuhan Hu, Yuelin Song, Li Institute of Image Communication and Network Engineering Shanghai Jiao Tong University China

The rapid rise of real-time communication and large language models has significantly increased the importance of speech compression. Deep learning-based neural speech codecs have outperformed traditional signal-level speech codecs in terms of rate-distortion (RD) performance. Typically, these neural codecs employ an encoder–quantizer–decoder architecture, where audio is first converted into latent code feature representations and then into discrete tokens. However, this architecture exhibits insufficient RD performance due to two main drawbacks: (1) the inadequate performance of the quantizer, challenging training processes, and issues such as codebook collapse;(2) the limited representational capacity of the encoder and decoder, making it difficult to meet feature representation requirements across various bitrates. In this paper, we propose a rate-aware learned speech compression scheme that replaces the quantizer with an advanced channel-wise entropy model to improve RD performance, simplify training, and avoid codebook collapse. We employ multi-scale convolution and linear attention mixture blocks to enhance the representational capacity and flexibility of the encoder and decoder. Experimental results demonstrate that the proposed method achieves state-of-the-art RD performance, obtaining 53.51% BD-Rate bitrate saving in average, and achieves 0.26 BD-VisQol and 0.44 BD-PESQ gains. Copyright © 2025, The Authors. All rights reserved.

关键词： Decoding

IllusionBench: A Large-scale and Comprehensive Benchmark for Visual Illusion Understanding in Vision-Language Models

学校读者我要写书评

暂无评论

arXiv 2025年

作者： Zhang, Yiming Zhang, Zicheng Wei, Xinyi Liu, Xiaohong Zhai, Guangtao Min, Xiongkuo Institute of Image Communication and Network Engineering Shanghai Jiao Tong University China

Current Visual Language Models (VLMs) show impressive image understanding but struggle with visual illusions, especially in real-world scenarios. Existing benchmarks focus on classical cognitive illusions, which have been learned by state-of-the-art (SOTA) VLMs, revealing issues such as hallucinations and limited perceptual abilities. To address this gap, we introduce IllusionBench, a comprehensive visual illusion dataset that encompasses not only classic cognitive illusions but also real-world scene illusions. This dataset features 1,051 images, 5,548 question-answer pairs, and 1,051 golden text descriptions that address the presence, causes, and content of the illusions. We evaluate ten SOTA VLMs on this dataset using true-or-false, multiple-choice, and open-ended tasks. In addition to real-world illusions, we design trap illusions that resemble classical patterns but differ in reality, highlighting hallucination issues in SOTA models. The top-performing model, GPT-4o, achieves 80.59% accuracy on true-or-false tasks and 76.75% on multiple-choice questions, but still lags behind human performance. In the semantic description task, GPT-4o’s hallucinations on classical illusions result in low scores for trap illusions, even falling behind some open-source models. IllusionBench is, to the best of our knowledge, the largest and most comprehensive benchmark for visual illusions in VLMs to date. © 2025, CC BY.

关键词： Visual languages

Energy-Efficient VR 360 Video Streaming in the IRS-aided Rate-Splitting Multiple Access network

学校读者我要写书评

暂无评论

IEEE Transactions on communications 2025年

作者： Teng, Long Wu, Qingqing Duan, Huiyu Min, Xiongkuo Zhai, Guangtao Shanghai Jiao Tong University Institute of Image Communication and Network Engineering Shanghai200240 China Shanghai Jiao Tong University Department of Electronic Engineering Shanghai200240 China

Maximizing energy efficiency in VR 360 video transmission is essential for advancing VR applications. Motivated by this goal, we conduct a comprehensive study that integrates the characteristics of VR 360 video with beamforming strategies and intelligent reflecting surface (IRS) shifting techniques in the rate-splitting multiple access (RSMA) network. In the IRS-aided RSMA network, we propose a stable energy-efficient transmission (SEET) scheme aimed at minimizing the number of transmitted VR video chunks. The SEET scheme constructs a stable pre-transmission and playback flow, ensuring seamless and continuous display of the upcoming content without latency. We also propose a mixed-format-based chunk (MFC) method that simultaneously pre-transmits both 2D and 3D chunk frames to each user, further enhancing energy efficiency. We utilize an alternating optimization method to divide the original energy-efficient problem into three subproblems. To tackle the non-convex and NP-hard beamforming subproblem, we utilize the first-order Taylor expansion and then obtain the approximate transmission rates of common messages and private messages regarding the quadratic form of beamforming vectors. We then utilize quadratically constrained programming, fractional programming, and linear programming to obtain the near-optimal solutions for beamforming vectors, IRS phase shifts, and RSMA parameters, respectively. The final numerical results affirm that the proposed SEET scheme can notably minimize the beamforming power of the base station. Through the SEET scheme, the MFC method with the approximation method exhibits superior energy efficiency, outperforming existing transmission methods in terms of both energy utility and consumption by HMDs. © 1972-2012 IEEE.

关键词： Beamforming

Towards All-in-One Medical image Re-Identification

学校读者我要写书评

暂无评论

arXiv 2025年

作者： Tian, Yuan Ji, Kaiyuan Zhang, Rongzhao Jiang, Yankai Li, Chunyi Wang, Xiaosong Zhai, Guangtao Shanghai AI Laboratory China School of Communication and Electronic Engineering East China Normal University China Institute of Image Communication and Network Engineering Shanghai Jiao Tong Unversity China

Medical image re-identification (MedReID) is under-explored so far, despite its critical applications in personalized healthcare and privacy protection. In this paper, we introduce a thorough benchmark and a unified model for this problem. First, to handle various medical modalities, we propose a novel Continuous Modality-based Parameter Adapter (ComPA). ComPA condenses medical content into a continuous modality representation and dynamically adjusts the modality-agnostic model with modality-specific parameters at runtime. This allows a single model to adaptively learn and process diverse modality data. Furthermore, we integrate medical priors into our model by aligning it with a bag of pre-trained medical foundation models, in terms of the differential features. Compared to single-image feature, modeling the inter-image difference better fits the re-identification problem, which involves discriminating multiple images. We evaluate the proposed model against 25 foundation models and 8 large multimodal language models across 11 image datasets, demonstrating consistently superior performance. Additionally, we deploy the proposed MedReID technique to two real-world applications, i.e., history-augmented personalized diagnosis and medical privacy protection. Codes and model is available at https://***/tianyuan168326/All-inOne-MedReID-Pytorch. Copyright © 2025, The Authors. All rights reserved.

关键词： Medical imaging

AGAV-Rater: Adapting Large Multimodal Model for AI-Generated Audio-Visual Quality Assessment

学校读者我要写书评

暂无评论

arXiv 2025年

作者： Cao, Yuqin Min, Xiongkuo Gao, Yixuan Sun, Wei Zhai, Guangtao Institute of Image Communication and Network Engineering Shanghai Key Laboratory of Digital Media Processing and Transmissions Shanghai Jiao Tong University Shanghai China

Many video-to-audio (VTA) methods have been proposed for dubbing silent AI-generated videos. An efficient quality assessment method for AI-generated audio-visual content (AGAV) is crucial for ensuring audio-visual quality. Existing audio-visual quality assessment methods struggle with unique distortions in AGAVs, such as unrealistic and inconsistent elements. To address this, we introduce AGAVQA, the first large-scale AGAV quality assessment dataset, comprising 3, 382 AGAVs from 16 VTA methods. AGAVQA includes two subsets: AGAVQA-MOS, which provides multi-dimensional scores for audio quality, content consistency, and overall quality, and AGAVQA-Pair, designed for optimal AGAV pair selection. We further propose AGAV-Rater, a LMM-based model that can score AGAVs, as well as audio and music generated from text, across multiple dimensions, and selects the best AGAV generated by VTA methods to present to the user. AGAV-Rater achieves state-of-the-art performance on AGAVQA, Text-to-Audio, and Text-to-Music datasets. Subjective tests also confirm that AGAV-Rater enhances VTA performance and user experience. The project page is available at https://***. Copyright © 2025, The Authors. All rights reserved.

关键词： Subjective testing

SSP-IR: Semantic and Structure Priors for Diffusion-based Realistic image Restoration

学校读者我要写书评

暂无评论

IEEE Transactions on Circuits and Systems for Video Technology 2025年

作者： Zhang, Yuhong Zhang, Hengsheng Cheng, Zhengxue Xie, Rong Song, Li Zhang, Wenjun Shanghai Jiao Tong University Institute of Image Communication and Network Engineering Shanghai China Shanghai Jiao Tong University MoE Key Lab of Artificial Intelligence AI Institute Shanghai China

Realistic image restoration is a crucial task in computer vision, and diffusion-based models for image restoration have garnered significant attention due to their ability to produce realistic results. Restoration can be seen as a controllable generation conditioning on priors. However, due to the severity of image degradation, existing diffusion-based restoration methods cannot fully exploit priors from low-quality images and still have many challenges in perceptual quality, semantic fidelity, and structure accuracy. Based on the challenges, we introduce a novel image restoration method, SSP-IR. Our approach aims to fully exploit semantic and structure priors from low-quality images to guide the diffusion model in generating semantically faithful and structurally accurate natural restoration results. Specifically, we integrate the visual comprehension capabilities of Multimodal Large Language Models (explicit) and the visual representations of the original image (implicit) to acquire accurate semantic prior. To extract degradation-independent structure prior, we introduce a Processor with RGB and FFT constraints to extract structure prior from the low-quality images, guiding the diffusion model and preventing the generation of unreasonable artifacts. Lastly, we employ a multi-level attention mechanism to integrate the acquired semantic and structure priors. The qualitative and quantitative results demonstrate that our method outperforms other state-of-the-art methods overall on both synthetic and real-world datasets. Our project page is https://***/projects/SSP-IR. © 1991-2012 IEEE.

关键词： image denoising