检索结果-内蒙古大学图书馆

High-Fidelity and High-Efficiency Talking Portrait Synthesis With Detail-Aware Neural Radiance Fields

IEEE Transactions on Visualization and computer Graphics 2024年 PP卷 PP-PP页

作者： Wang, Muyu Zhao, Sanyuan Dong, Xingping Shen, Jianbing Wuhan University School of Computer Science National Engineering Research Center for Multimedia Software Institute of Artificial Intelligence Hubei Key Laboratory of Multimedia and Network Communication Engineering Wuhan430072 China Beijing Institute of Technology School of Computer Science Beijing China University of Macau State Key Laboratory of Internet of Things for Smart City Department of Computer and Information Science China

In this paper, we propose a novel rendering framework based on neural radiance fields (NeRF) named HH-NeRF that can generate high-resolution audio-driven talking portrait videos with high fidelity and fast rendering. Specifically, our framework includes a detail-aware NeRF module and an efficient conditional super-resolution module. Firstly, a detail-aware NeRF is proposed to efficiently generate a high-fidelity low-resolution talking head, by using the encoded volume density estimation and audio-eye-aware color calculation. This module can capture natural eye blinks and high-frequency details, and maintain a similar rendering time as previous fast methods. Secondly, we present an efficient conditional super-resolution module on the dynamic scene to directly generate the high-resolution portrait with our low-resolution head. Incorporated with the prior information, such as depth map and audio features, our new proposed efficient conditional super resolution module can adopt a lightweight network to efficiently generate realistic and distinct high-resolution videos. Extensive experiments demonstrate that our method can generate more distinct and fidelity talking portraits on high resolution (900 × 900) videos compared to state-of-the-art methods. Our code is available at https://***/muyuWang/HHNeRF. © 1995-2012 IEEE.

关键词： Rendering (computer graphics)

来源：评论

学校读者我要写书评

暂无评论

Socially Beneficial Metaverse: Framework, Technologies, Applications, and Challenges

arXiv

引用

arXiv 2023年

作者： Xu, Xiaolong Zhou, Xuanhong Bilal, Muhammad Zeadally, Sherali Crowcroft, Jon Qi, Lianyong Xue, Shengjun Nanjing University of Information Science and Technology Nanjing China School of Software Nanjing University of Information Science and Technology Nanjing China School of Computing and Communications Lancaster University Lancaster United Kingdom College of Communication and Information University of Kentucky Lexington United States Department of Computer Science and Technology University of Cambridge Cambridge United Kingdom State Key Laboratory for Novel Software Technology Nanjing University China School of Computer Science and Technology Silicon Lake College Suzhou China

In recent years, the maturation of emerging technologies such as Virtual Reality, Digital Twin,s and Blockchain has accelerated the realization of the metaverse. As a virtual world independent of the real world, the metaverse will provide users with a variety of virtual activities which bring great convenience to society. In addition, the metaverse can facilitate digital twins, which offers transformative possibilities for the industry. Thus, the metaverse has attracted the attention of the industry, and a huge amount of capital is about to be invested. However, the development of the metaverse is still in its infancy and little research has been undertaken so far. We describe the development of the metaverse. Next, we introduce the architecture of the socially beneficial metaverse (SB-Metaverse) and we focus on the technologies that support the operation of SB-Metaverse. In addition, we also present the applications of SB-Metaverse. Finally, we discuss several challenges faced by SB-Metaverse which must be addressed in the *** Codes 68U01, 68M11, 68U35 © 2023, CC BY.

关键词： Blockchain

来源：评论

学校读者我要写书评

暂无评论

Anomize: Better Open Vocabulary Video Anomaly Detection

arXiv

引用

arXiv 2025年

作者： Li, Fei Liu, Wenxuan Chen, Jingjing Zhang, Ruixu Wang, Yuran Zhong, Xian Wang, Zheng National Engineering Research Center for Multimedia Software School of Computer Science Wuhan University China Shanghai Key Lab of Intelligent Information Processing School of Computer Science Fudan University China State Key Laboratory for Multimedia Information Processing School of Computer Science Peking University China Hubei Key Laboratory of Transportation Internet of Things Wuhan University of Technology China

Open Vocabulary Video Anomaly Detection (OVVAD) seeks to detect and classify both base and novel anomalies. However, existing methods face two specific challenges related to novel anomalies. The first challenge is detection ambiguity, where the model struggles to assign accurate anomaly scores to unfamiliar anomalies. The second challenge is categorization confusion, where novel anomalies are often misclassified as visually similar base instances. To address these challenges, we explore supplementary information from multiple sources to mitigate detection ambiguity by leveraging multiple levels of visual data alongside matching textual information. Furthermore, we propose incorporating label relations to guide the encoding of new labels, thereby improving alignment between novel videos and their corresponding labels, which helps reduce categorization confusion. The resulting Anomize framework effectively tackles these issues, achieving superior performance on UCF-CRIME and XD-VIOLENCE datasets, demonstrating its effectiveness in OVVAD. Copyright © 2025, The Authors. All rights reserved.

关键词：

来源：评论

学校读者我要写书评

暂无评论

On the Effectiveness of Distillation in Mitigating Backdoors in Pre-trained Encoder

arXiv

引用

arXiv 2024年

作者： Han, Tingxu Huang, Shenghan Ding, Ziqi Sun, Weisong Feng, Yebo Fang, Chunrong Li, Jun Qian, Hanwei Wu, Cong Zhang, Quanjun Liu, Yang Chen, Zhenyu The State Key Laboratory for Novel Software Technology Nanjing University Jiangsu210093 China The School of Computer Science University of New South Wales NSW2052 Australia The School of Computer Science and Engineering Nanyang Technological University Nanyang639798 Singapore The Department of Computer Science The Director of the Network & Security Research Laboratory The University of Oregon United States

With the development of deep learning, self-supervised learning (SSL) has become particularly popular. SSL pre-trains an encoder (e.g., an image encoder) to learn generic features from a large amount of unlabeled data, releases it on a third-party platform, and allows any downstream task (e.g., an image classification task) to further fine-tune the encoder with task-specific data to derive the model for the task. However, despite the tremendous performance improvements it brings, SSL faces serious security risks. An adversary can poison pre-training datasets used to pre-train an encoder or publish a poisoned encoder directly to third-party platforms, which will also poison the model for any downstream task that uses these encoders. In this paper, we study a defense against poisoned encoders in SSL called distillation, which is a defense used in supervised learning originally. Distillation aims to distill knowledge from a given model (a.k.a the teacher net) and transfer it to another (a.k.a the student net). Now, we use it to distill benign knowledge from poisoned pre-trained encoders and transfer it to a new encoder, resulting in a clean pre-trained encoder. In particular, we conduct an empirical study on the effectiveness and performance of distillation against poisoned encoders. Using two state-of-the-art backdoor attacks against pre-trained image encoders and four commonly used image classification datasets, our experimental results show that distillation can reduce attack success rate from 80.87% to 27.51% while suffering a 6.35% loss in accuracy. Moreover, we investigate the impact of three core components of distillation on performance: Teacher net, student net, and distillation loss. By comparing 4 different teacher nets, 3 student nets, and 6 distillation losses, we find that fine-tuned teacher nets, warm-up-training-based student nets, and attention-based distillation loss perform best, respectively. Copyright © 2024, The Authors. All rights reserved.

关键词： Distillation

来源：评论

学校读者我要写书评

暂无评论

Adaptive Perception for Unified Visual Multi-modal Object Tracking

IEEE Transactions on Artificial Intelligence

引用

IEEE Transactions on Artificial Intelligence 2025年

作者： Hu, Xiantao Zhong, Bineng Liang, Qihua Shi, Liangtao Mo, Zhiyi Tai, Ying Yang, Jian Guangxi Normal University Guangxi Key Lab of Multi-Source Information Mining Guilin541004 China Guangxi Normal University Security Guilin541004 China Guangxi Normal University Intelligent Technology Guilin541004 China Guangxi Normal University Ministry of Education Guilin541004 China Wuzhou University School of Data Science Wuzhou543002 China Wuzhou University Software Engineering Wuzhou543002 China Nanjing University State Key Laboratory for Novel Software Technology Suzhou215163 China Nanjing University of Science and Technology School of Computer Science Nanjing210094 China Nanjing University of Science and Technology Engineering Nanjing210094 China

Recently, many multi-modal trackers prioritize RGB as the dominant modality, treating other modalities as auxiliary, and fine-tuning separately various multi-modal tasks. This imbalance in modality dependence limits the ability of methods to dynamically utilize complementary information from each modality in complex scenarios, making it challenging to fully perceive the advantages of multi-modal. As a result, a unified parameter model often underperforms in various multi-modal tracking tasks. To address this issue, we propose APTrack, a novel unified tracker designed for multi-modal adaptive perception. Unlike previous methods, APTrack explores a unified representation through an equal modeling strategy. This strategy allows the model to dynamically adapt to various modalities and tasks without requiring additional fine-tuning between different tasks. Moreover, our tracker integrates an adaptive modality interaction (AMI) module that efficiently bridges cross-modality interactions by generating learnable tokens. Experiments conducted on five diverse multi-modal datasets (RGBT234, LasHeR, VisEvent, DepthTrack, and VOT-RGBD2022) demonstrate that APTrack not only surpasses existing state-of-the-art unified multi-modal trackers but also outperforms trackers designed for specific multi-modal tasks. © 2020 IEEE.

关键词： Multicarrier modulation

来源：评论

学校读者我要写书评

暂无评论

Grrln: Gated Recurrent Residual Learning Networks for Code Clone Detection

SSRN

引用

SSRN 2022年

作者： Zhang, Xiangping Xiao, Qiaoxiang Hu, Haize Liu, Jianxun Hunan Provincial Key Lab. for Services Computing and Novel Software Technology Hunan University of Science and Technology Hunan Xiangtan411100 China School of Computer Science and Technology Hunan University of Science and Technology Hunan Xiangtan411100 China

Code clone detection is a critical problem in software development and maintenance domains. It aims to identify functionally identical or similar code fragments within an application. Existing works formulate the code clone detection task as a binary classification problem which is determine the code pairs are clone or not with a pre-defined threshold. In reality, there are various types of code clone subject to the degree of how a pair of code fragments are similar to each other. To investigate the effect of different code clone detection manner on the clone detection result, we propose Gated Recurrent Residual Learning Networks (GRRLN), a novel neural network model for code clone detection. To train GRRLN, we first represent each code fragment as a statement-level tree sequence derived from the whole AST. Then, a gated recurrent neural network with residual connections is adopted to fully extract semantics of all individual statement trees together with their dependency relationships across the input statement sequence. Finally, the output representations of code fragments by GRRLN are used for similarity calculation and clone detection. We evaluate GRRLN using two real-world datasets for code clone detection and clone type classification. Experiments show that GRRLN achieves generally better performance than state-of-the-art methods and meanwhile needs significant less time and memory costs. © 2022, The Authors. All rights reserved.

关键词： Recurrent neural networks

来源：评论

学校读者我要写书评

暂无评论

MOL-AE: Auto-encoder based molecular representation learning with 3D cloze test objective 24

MOL-AE: Auto-encoder based molecular representation learning...

引用

Proceedings of the 41st International Conference on Machine Learning

作者： Junwei Yang Kangjie Zheng Siyu Long Zaiqing Nie Ming Zhang Xinyu Dai Wei-Ying Ma Hao Zhou School of Computer Science National Key Laboratory for Multimedia Information Processing Peking University-Anker Embodied AI Lab Peking University School of Artificial Intelligence National Key Laboratory for Novel Software Technology Nanjing University Institute for AI Industry Research (AIR) Tsinghua University and PharMolix Inc. Institute for AI Industry Research (AIR) Tsinghua University

3D molecular representation learning has gained tremendous interest and achieved promising performance in various downstream tasks. A series of recent approaches follow a prevalent framework: an encoder-only model coupled with a coordinate denoising objective. However, through a series of analytical experiments, we prove that the encoderonly model with coordinate denoising objective exhibits inconsistency between pre-training and downstream objectives, as well as issues with disrupted atomic identifiers. To address these two issues, we propose MOL-AE for molecular representation learning, an auto-encoder model using positional encoding as atomic identifiers. We also propose a new training objective named 3D Cloze Test to make the model learn better atom spatial relationships from real molecular substructures. Empirical results demonstrate that MOL-AE achieves a large margin performance gain compared to the current state-of-the-art 3D molecular modeling approach. The source codes of MOL-AE are publicly available at https://***/yjwtheonly/MolAE.

关键词：

来源：评论

学校读者我要写书评

暂无评论

Skinned Motion Retargeting with Dense Geometric Interaction Perception

arXiv

引用

arXiv 2024年

作者： Ye, Zijie Liu, Jia-Wei Jia, Jia Sun, Shikun Shou, Mike Zheng Department of Computer Science and Technology BNRist Tsinghua University China Key Laboratory of Pervasive Computing Ministry of Education China Show Lab National University of Singapore Singapore

Capturing and maintaining geometric interactions among different body parts is crucial for successful motion retargeting in skinned characters. Existing approaches often overlook body geometries or add a geometry correction stage after skeletal motion retargeting. This results in conflicts between skeleton interaction and geometry correction, leading to issues such as jittery, interpenetration, and contact mismatches. To address these challenges, we introduce a new retargeting framework, MeshRet, which directly models the dense geometric interactions in motion retargeting. Initially, we establish dense mesh correspondences between characters using semantically consistent sensors (SCS), effective across diverse mesh topologies. Subsequently, we develop a novel spatio-temporal representation called the dense mesh interaction (DMI) field. This field, a collection of interacting SCS feature vectors, skillfully captures both contact and non-contact interactions between body geometries. By aligning the DMI field during retargeting, MeshRet not only preserves motion semantics but also prevents self-interpenetration and ensures contact preservation. Extensive experiments on the public Mixamo dataset and our newly-collected ScanRet dataset demonstrate that MeshRet achieves state-of-the-art performance. Code available at https://***/abcyzj/MeshRet. Copyright © 2024, The Authors. All rights reserved.

关键词： Mesh generation

来源：评论

学校读者我要写书评

暂无评论

Knowledge Base Question Answering: A Semantic Parsing Perspective

arXiv

引用

arXiv 2022年

作者： Gu, Yu Pahuja, Vardaan Cheng, Gong Su, Yu Department of Computer Science and Engineering The Ohio State University United States State Key Laboratory for Novel Software Technology Nanjing University China

Recent advances in deep learning have greatly propelled the research on semantic parsing. Improvement has since been made in many downstream tasks, including natural language interface to web APIs, text-to-SQL generation, among others. However, despite the close connection shared with these tasks, research on question answering over knowledge bases (KBQA) has comparatively been progressing slowly. We identify and attribute this to two unique challenges of KBQA, schema-level complexity and fact-level complexity. In this survey, we situate KBQA in the broader literature of semantic parsing and give a comprehensive account of how existing KBQA approaches attempt to address the unique challenges. Regardless of the unique challenges, we argue that we can still take much inspiration from the literature of semantic parsing, which has been overlooked by existing research on KBQA. Based on our discussion, we can better understand the bottleneck of current KBQA research and shed light on promising directions for KBQA to keep up with the literature of semantic parsing, particularly in the era of pre-trained language models. © 2022, CC BY.

关键词： Deep learning

来源：评论

学校读者我要写书评

暂无评论

Dmnet: Difference minimization network for semi-supervised segmentation in medical images 23rd

Dmnet: Difference minimization network for semi-supervised s...

引用

23rd International Conference on Medical Image Computing and computer-Assisted Intervention, MICCAI 2020

作者： Fang, Kang Li, Wu-Jun National Key Laboratory for Novel Software Technology Department of Computer Science and Technology Nanjing University National Institute of Healthcare Data Science at Nanjing University Nanjing China

ISBN: (纸本)9783030597092

Semantic segmentation is an important task in medical image analysis. In general, training models with high performance needs a large amount of labeled data. However, collecting labeled data is typically difficult, especially for medical images. Several semi-supervised methods have been proposed to use unlabeled data to facilitate learning. Most of these methods use a self-training framework, in which the model cannot be well trained if the pseudo masks predicted by the model itself are of low quality. Co-training is another widely used semi-supervised method in medical image segmentation. It uses two models and makes them learn from each other. All these methods are not end-to-end. In this paper, we propose a novel end-to-end approach, called difference minimization network(DMNet), for semi-supervised semantic segmentation. To use unlabeled data, DMNet adopts two decoder branches and minimizes the difference between soft masks generated by the two decoders. In this manner, each decoder can learn under the supervision of the other decoder, thus they can be improved at the same time. Also, to make the model generalize better, we force the model to generate low-entropy masks on unlabeled data so the decision boundary of model lies in low-density regions. Meanwhile, adversarial training strategy is adopted to learn a discriminator which can encourage the model to generate more accurate masks. Experiments on a kidney tumor dataset and a brain tumor dataset show that our method can outperform the baselines, including both supervised and semi-supervised ones, to achieve the best performance. © Springer Nature Switzerland AG 2020.

关键词： Semantic Segmentation

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：