检索结果-内蒙古大学图书馆

SVFR: A Unified Framework for Generalized Video Face Restoration

学校读者我要写书评

暂无评论

arXiv 2025年

作者： Wang, Zhiyao Chen, Xu Xu, Chengming Zhu, Junwei Hu, Xiaobin Zhang, Jiangning Wang, Chengjie Liu, Yuqi Zhou, Yiyi Ji, Rongrong Key Laboratory of Multimedia Trusted Perception and Efficient Computing Ministry of Education of China Xiamen University China Tencent Youtu Lab China

Face Restoration (FR) is a crucial area within image and video processing, focusing on reconstructing high-quality portraits from degraded inputs. Despite advancements in image FR, video FR remains relatively under-explored, primarily due to challenges related to temporal consistency, motion artifacts, and the limited availability of high-quality video data. Moreover, traditional face restoration typically prioritizes enhancing resolution and may not give as much consideration to related tasks such as facial colorization and inpainting. In this paper, we propose a novel approach for the Generalized Video Face Restoration (GVFR) task, which integrates video BFR, inpainting, and colorization tasks that we empirically show to benefit each other. We present a unified framework, termed as stable video face restoration (SVFR), which leverages the generative and motion priors of Stable Video Diffusion (SVD) and incorporates task-specific information through a unified face restoration framework. A learnable task embedding is introduced to enhance task identification. Meanwhile, a novel Unified Latent Regularization (ULR) is employed to encourage the shared feature representation learning among different subtasks. To further enhance the restoration quality and temporal stability, we introduce the facial prior learning and the self-referred refinement as auxiliary strategies used for both training and inference. The proposed framework effectively combines the complementary strengths of these tasks, enhancing temporal coherence and achieving superior restoration quality. This work advances the state-of-the-art in video FR and establishes a new paradigm for generalized video face restoration. Code and video demo are available at https://***/wangzhiyaoo/***. Copyright © 2025, The Authors. All rights reserved.

关键词：

X-Oscar: A Progressive Framework for High-quality Text-guided 3D Animatable Avatar Generation

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Ma, Yiwei Lin, Zhekai Ji, Jiayi Fan, Yijun Sun, Xiaoshuai Ji, Rongrong Key Laboratory of Multimedia Trusted Perception and Efficient Computing Ministry of Education of China School of Informatics Xiamen University 361005 China

Recent advancements in automatic 3D avatar generation guided by text have made significant progress. However, existing methods have limitations such as oversaturation and low-quality output. To address these challenges, we propose X-Oscar, a progressive framework for generating high-quality animatable avatars from text prompts. It follows a sequential "Geometry→Texture→Animation" paradigm, simplifying optimization through step-by-step generation. To tackle oversaturation, we introduce Adaptive Variational Parameter (AVP), representing avatars as an adaptive distribution during training. Additionally, we present Avatar-aware Score Distillation Sampling (ASDS), a novel technique that incorporates avatar-aware noise into rendered images for improved generation quality during optimization. Extensive evaluations confirm the superiority of X-Oscar over existing text-to-3D and text-to-avatar approaches. Our anonymous project page: https://***/Projects/X-Oscar/. Copyright © 2024, The Authors. All rights reserved.

关键词： Distillation

ME-FAS: Multimodal Text Enhancement for Cross-Domain Face Anti-Spoofing

学校读者我要写书评

暂无评论

IEEE Transactions on Information Forensics and Security 2025年 20卷 5451-5464页

作者： Cai, Lvpan Wang, Haowei Ji, Jiayi Sun, Xiaoshuai Cao, Liujuan Ji, Rongrong Ministry of Education of China Xiamen University Key Laboratory of Multimedia Trusted Perception and Efficient Computing 361005 China Youtu Lab Tencent Shanghai China National University of Singapore Singapore

The focus of Face Anti-Spoofing (FAS) is shifting toward improving generalization performance in unseen scenarios. Traditional methods employing adversarial learning and meta learning aim to extract or decouple generalizable features to address these challenges. However, enhancing performance solely through facial features remains challenging without additional informative inputs. To address this, Vision-Language Models (VLMs) with robust generalization capabilities have recently been introduced to FAS. Despite their potential, these VLMs typically adopt a late alignment strategy, relying only on encoder output features for modality alignment, which largely neglects mutual guidance between modalities. To bridge this gap, inspired by recent advancements in prompt learning, we employ learnable prompts and masking as intermediaries to enhance interaction between text and visual modalities, enabling the extraction of more generalizable features. Specifically, we propose ME-FAS, a Modality-Enhanced cross-domain FAS model integrating Prompt Fusion Transfer (PFT) and Text-guided Image Masking (TIM). PFT facilitates the integration of text features with visual information, improving domain adaptability in alignment with the textual context. Meanwhile, TIM leverages text features to mask image patches, directing visual features toward critical generalizable facial information, such as the eyes and mouth. Comprehensive evaluations across multiple benchmarks and various visualizations demonstrate significant performance gains, validating the effectiveness of our proposed approach. © 2005-2012 IEEE.

关键词： Domain Generalization Face Anti-Spoofing Vision and Language

Federated Graph Learning for Cross-Domain Recommendation 38

学校读者我要写书评

暂无评论

Federated Graph Learning for Cross-Domain Recommendation

38th Conference on Neural Information Processing Systems, NeurIPS 2024

作者： Yang, Ziqi Peng, Zhaopeng Wang, Zihui Qi, Jianzhong Chen, Chaochao Pan, Weike Wen, Chenglu Wang, Cheng Fan, Xiaoliang Fujian Key Laboratory of Sensing and Computing for Smart Cities Xiamen University China Key Laboratory of Multimedia Trusted Perception and Efficient Computing Ministry of Education of China Xiamen University China School of Computing and Information Systems The University of Melbourne Australia College of Computer Science and Technology Zhejiang University Hangzhou China College of Computer Science and Software Engineering Shenzhen University Shenzhen China

Cross-domain recommendation (CDR) offers a promising solution to the data sparsity problem by enabling knowledge transfer between source and target domains. However, many recent CDR models overlook crucial issues such as privacy as well as the risk of negative transfer (which negatively impact model performance), especially in multi-domain settings. To address these challenges, we propose FedGCDR, a novel federated graph learning framework that securely and effectively leverages positive knowledge from multiple source domains. First, we design a positive knowledge transfer module that ensures privacy during inter-domain knowledge transmission. This module employs differential privacy-based knowledge extraction combined with a feature mapping mechanism, transforming source domain embeddings from federated graph attention networks into reliable domain knowledge. Second, we design a knowledge activation module to filter out potential harmful or conflicting knowledge from source domains, addressing the issues of negative transfer. This module enhances target domain training by expanding the graph of the target domain to generate reliable domain attentions and fine-tunes the target model for improved negative knowledge filtering and more accurate predictions. We conduct extensive experiments on 16 popular domains of the Amazon dataset, demonstrating that FedGCDR significantly outperforms state-of-the-art methods. We open source the code at https://***/LafinHana/FedGCDR. © 2024 Neural information processing systems foundation. All rights reserved.

关键词：

Camouflaged Object Detection Via Dual-Branch Fusion and Dual Self-Similarity Constraints

学校读者我要写书评

暂无评论

SSRN

SSRN 2024年

作者： Yang, Haozhe Zhu, Yuan Sun, Ke Ding, Haoyang Lin, Xianming Key Laboratory of Multimedia Trusted Perception and Efficient Computing Ministry of Education of China School of Informatics Xiamen University 361005 China

Camouflaged Object Detection (COD) remains a challenging task due to the inherent difficulty in distinguishing concealed objects from their surroundings, primarily owing to their high visual similarity. Current research efforts have been insufficient in distinguishing the similarity differences between camouflaged objects and their corresponding backgrounds, leading to suboptimal performance, particularly when locating small-scale objects. To solve this issue, we introduce a novel Dual-branch Fusion and Dual Self-similarity Network (DSNet) comprising three modules. The first, a dual-branch fusion, mimics human observation of camouflaged objects and extracts multi-angle information. Dual-branch features are then decoded using a symmetric joint decoder module with channel interaction via multi-stage inter-group interaction. Inspired by natural organisms' self-similarity, the self-similarity constraint module employs global and mutual constraints to identify subtle foreground-background differences. DSNet shows superior performance in experiments, and the constraint module can enhance other models as a plug-and-play component, further boosting overall performance. © 2024, The Authors. All rights reserved.

关键词： Object detection

Rotated Multi-Scale Interaction Network for Referring Remote Sensing Image Segmentation

学校读者我要写书评

暂无评论

Rotated Multi-Scale Interaction Network for Referring Remote...

Conference on Computer Vision and Pattern Recognition (CVPR)

作者： Sihan Liu Yiwei Ma Xiaoqing Zhang Haowei Wang Jiayi Ji Xiaoshuai Sun Rongrong Ji Key Laboratory of Multimedia Trusted Perception and Efficient Computing Ministry of Education of China Xiamen University P.R. China

ISBN: (数字)9798350353006

ISBN: (纸本)9798350353013

Referring Remote Sensing Image Segmentation (RRSIS) is a new challenge that combines computer vision and natu-ral language processing. Traditional Referring Image Seg-mentation (RIS) approaches have been impeded by the com-plex spatial scales and orientations found in aerial imagery, leading to suboptimal segmentation results. To address these challenges, we introduce the Rotated Multi-Scale In-teraction Network (RMSIN), an innovative approach de-signed for the unique demands of RRSIS. RMSIN incorpo-rates an Intra-scale Interaction Module (IIM) to effectively address the fine-grained detail required at multiple scales and a Cross-scale Interaction Module (CIM) for integrating these details coherently across the network. Furthermore, RMSIN employs an Adaptive Rotated Convolution (ARC) to account for the diverse orientations of objects, a novel contribution that significantly enhances segmentation accu-racy. To assess the efficacy of RMSIN, we have curated an expansive dataset comprising 17,402 image-caption-mask triplets, which is unparalleled in terms of scale and vari-ety. This dataset not only presents the model with a wide range of spatial and rotational scenarios but also estab-lishes a stringent benchmark for the RRSIS task, ensuring a rigorous evaluation of performance. Experimental eval-uations demonstrate the exceptional performance of RM-SIN, surpassing existing state-of-the-art models by a signif-icant margin. Datasets and code are available at https://***/Lsan2401/RMSIN.

关键词： Image segmentation Computer vision Codes Navigation Convolution Benchmark testing Solids

DiagnoCare: A Diagnostic Analysis System for Electronic Medical Records Combining Medical Knowledge Graph and Large Language Model

学校读者我要写书评

暂无评论

DiagnoCare: A Diagnostic Analysis System for Electronic Medi...

Smart World Congress (SWC), IEEE

作者： Shiwang Huang Guofeng Luo Jiaru Wang Longbiao Chen Fujian Key Laboratory of Sensing and Computing for Smart Cities Xiamen University China Key Laboratory of Multimedia Trusted Perception and Efficient Computing Xiamen University Xiamen China

ISBN: (数字)9798331520861

ISBN: (纸本)9798331520878

With the rapid development of healthcare technologies and the increasing demand for intelligent diagnostic systems, we present a novel approach that combines medical knowledge graphs and Large Language Models (LLMs) for enhanced Electronic Medical Record (EMR) analysis and condition warning. Our system integrates structured medical knowledge with advanced text analysis capabilities to accurately interpret and process complex EMR data. By leveraging LLMs and prompt learning, the system provides personalized examination suggestions, diagnostic analyses, and disease warnings. Experimental results on real-world datasets, including the MIMIC III database and anonymized hospital records from Fujian Province, China, demonstrate the system’s efficacy in delivering precise and actionable insights. Case studies further illustrate the system’s practical value and usability, highlighting its potential to significantly improve clinical decision-making and patient outcomes. This research offers substantial implications for the advancement of intelligent EMR systems and personalized healthcare.

关键词： Electric potential Text analysis Hospitals Large language models MIMICs Decision making Knowledge graphs Electronic medical records Medical diagnostic imaging Diseases

BOOSTING MULTIMODAL LARGE LANGUAGE MODELS WITH VISUAL TOKENS WITHDRAWAL FOR RAPID INFERENCE

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Lin, Zhihang Lin, Mingbao Lin, Luxi Ji, Rongrong Key Laboratory of Multimedia Trusted Perception and Efficient Computing Ministry of Education of China Xiamen University China Skywork AI Singapore

Multimodal large language models (MLLMs) demand considerable computations for inference due to the extensive parameters and the additional input tokens needed for visual information representation. Herein, we introduce Visual Tokens Withdrawal (VTW), a plug-and-play module to boost MLLMs for rapid inference. Our approach is inspired by two intriguing phenomena we have observed: (1) the attention sink phenomenon that is prevalent in LLMs also persists in MLLMs, suggesting that initial tokens and nearest tokens receive the majority of attention, while middle vision tokens garner minimal attention in deep layers;(2) the presence of information migration, which implies that visual information is transferred to subsequent text tokens within the first few layers of MLLMs. As per our findings, we conclude that vision tokens are unnecessary in the deep layers of MLLMs. Thus, we strategically withdraw them at a certain layer, enabling only text tokens to engage in subsequent layers. To pinpoint the ideal layer for VTW, we initially analyze a limited set of tiny datasets and choose the first layer that meets the Kullback-Leibler divergence criterion. Our VTW approach can cut computational overhead by over 40% across diverse multimodal tasks while maintaining performance. Our code is released at https://***/lzhxmu/VTW. Copyright © 2024, The Authors. All rights reserved.

关键词： Visual BASIC

Code Membership Inference for Detecting Unauthorized Data Use in Code Pre-trained Language Models

学校读者我要写书评

暂无评论

arXiv 2023年

作者： Zhang, Sheng Li, Hui Ji, Rongrong Key Laboratory of Multimedia Trusted Perception and Efficient Computing Ministry of Education of China Xiamen University China

Code pre-trained language models (CPLMs) have received great attention since they can benefit various tasks that facilitate software development and maintenance. However, CPLMs are trained on massive open-source code, raising concerns about potential data infringement. This paper launches the study of detecting unauthorized code use in CPLMs, i.e., Code Membership Inference (CMI) task. We design a framework BUZZER for different settings of CMI. BUZZER deploys several inference techniques, including signal extraction from pre-training tasks, hard-to-learn sample calibration and weighted inference, to identify code membership status accurately. Extensive experiments show that CMI can be achieved with high accuracy using BUZZER. Hence, BUZZER can serve as a CMI tool and help protect intellectual property rights. The implementation of BUZZER is available at: https://***/KDEGroup/Buzzer. Copyright © 2023, The Authors. All rights reserved.

关键词： Open source software