检索结果-内蒙古大学图书馆

AVTENet: A human-Cognition-Inspired Audio-Visual Transformer-Based Ensemble Network for Video Deepfake Detection

IEEE Transactions on Cognitive and Developmental Systems 2025年

作者： Hashmi, Ammarah Shahzad, Sahibzada Adil Lin, Chia-Wen Tsao, Yu Wang, Hsin-Min Taiwan International Graduate Program Academia Sinica Social Networks and Human-Centered Computing Program Taipei11529 Taiwan National Tsing Hua University Institute of Information Systems and Applications Hsinchu30013 Taiwan National Chengchi University Department of Computer Science Taipei11605 Taiwan National Tsing Hua University Department of Electrical Engineering Institute of Communications Engineering Hsinchu300044 Taiwan Academia Sinica Research Center for Information Technology Innovation Taipei11529 Taiwan

The recent proliferation of hyper-realistic deepfake videos has drawn attention to the threat of audio and visual forgeries. Most previous studies on detecting artificial intelligence-generated fake videos only utilize visual modality or audio modality. While some methods exploit audio and visual modalities to detect forged videos, they have not been comprehensively evaluated on multimodal datasets of deepfake videos involving acoustic and visual manipulations, and are mostly based on convolutional neural networks with low detection accuracy. Considering that human cognition instinctively integrates multisensory information including audio and visual cues to perceive and interpret content and the success of transformer in various fields, this study introduces the audio-visual transformer-based ensemble network (AVTENet). This innovative framework tackles the complexities of deepfake technology by integrating both acoustic and visual manipulations to enhance the accuracy of video forgery detection. Specifically, the proposed model integrates several purely transformer-based variants that capture video, audio, and audio-visual salient cues to reach a consensus in prediction. For evaluation, we use the recently released benchmark multimodal audio-video FakeAVCeleb dataset. For a detailed analysis, we evaluate AVTENet, its variants, and several existing methods on multiple test sets of the FakeAVCeleb dataset. Experimental results show that the proposed model outperforms all existing methods and achieves state-of-the-art performance on Testset-I and Testset-II of the FakeAVCeleb dataset. We also compare AVTENet against humans in detecting video forgery. The results show that AVTENet significantly outperforms humans. © 2016 IEEE.

关键词： audio-visual AVTENet Deepfake detection

来源：评论

学校读者我要写书评

暂无评论

How Good is ChatGPT at Audiovisual Deepfake Detection: A Comparative Study of ChatGPT, AI Models and human Perception

arXiv

引用

arXiv 2024年

作者： Shahzad, Sahibzada Adil Hashmi, Ammarah Peng, Yan-Tsung Tsao, Yu Wang, Hsin-Min The Social Networks and Human-Centered Computing Program Taiwan International Graduate Program Academia Sinica Taipei Taiwan The Department of Computer Science National Chengchi University Taipei Taiwan The Institute of Information Systems and Applications National Tsing Hua University Hsinchu Taiwan The Research Center for Information Technology Innovation Academia Sinica Taipei Taiwan The Institute of Information Science Academia Sinica Taiwan

Multimodal deepfakes involving audiovisual manipulations are a growing threat because they are difficult to detect with the naked eye or using unimodal deep learning-based forgery detection methods. Audiovisual forensic models, while more capable than unimodal models, require large training datasets and are computationally expensive for training and inference. Furthermore, these models lack interpretability and often do not generalize well to unseen manipulations. In this study, we examine the detection capabilities of a large language model (LLM) (i.e., ChatGPT) to identify and account for any possible visual and auditory artifacts and manipulations in audiovisual deepfake content. Extensive experiments are conducted on videos from a benchmark multimodal deepfake dataset to evaluate the detection performance of ChatGPT and compare it with the detection capabilities of state-of-the-art multimodal forensic models and humans. Experimental results demonstrate the importance of domain knowledge and prompt engineering for video forgery detection tasks using LLMs. Unlike approaches based on end-to-end learning, ChatGPT can account for spatial and spatiotemporal artifacts and inconsistencies that may exist within or across modalities. Additionally, we discuss the limitations of ChatGPT for multimedia forensic tasks. © 2024, CC BY.

关键词： Audiovisual

来源：评论

学校读者我要写书评

暂无评论

Zema Dataset: A Comprehensive Study of Yaredawi Zema with a Focus on Horologium Chants

arXiv

引用

arXiv 2024年

作者： Muluneh, Mequanent Argaw Peng, Yan-Tsung Degife, Worku Abebe Tadesse, Nigussie Abate Demeku, Aknachew Mebreku Su, Li Institute of Information Science Academia Sinica Taipei Taiwan Department of Information System University of Gondar Gondar Ethiopia Department of Computer Science National Chengchi University Taipei Taiwan Social Networks and Human-Centered Computing Program Taiwan International Graduate Program Taiwan Department of Information Management National Taiwan University of Science and Technology Taipei Taiwan Department of Material Science and Engineering National Taiwan University of Science and Technology Taipei Taiwan

Computational music research plays a critical role in advancing music production, distribution, and understanding across various musical styles in the world. Despite the immense cultural and religious significance, the Ethiopian Orthodox Tewahedo Church (EOTC) chants are relatively underrepresented in computational music research. This paper contributes to this field by introducing a new dataset specifically tailored for the analysis of EOTC chants, also known as Yaredawi Zema. This work provides a comprehensive overview of a 10-hour dataset, 369 instances, creation, and curation process, including rigorous quality assurance measures. Our dataset has a detailed word-level temporal boundary and reading tone annotation along with the corresponding chanting mode label of audios. Moreover, we have also identified the chanting options associated with multiple chanting notations in the manuscript by annotating them accordingly. Our goal in making this dataset available to the public1 is to encourage more research and study of EOTC chants, including lyrics transcription, lyric-to-audio alignment, and music generation tasks. Such research work will advance knowledge and efforts to preserve this distinctive liturgical music, a priceless cultural artifact for the Ethiopian people. © 2024, CC BY.

关键词： Music

来源：评论

学校读者我要写书评

暂无评论

Towards Adversarial Robustness for Multi-Mode Data through Metric Learning

引用

Sensors 2023年第13期23卷 6173页

作者： Khan, Sarwar Chen, Jun-Cheng Liao, Wen-Hung Chen, Chu-Song Research Center for Information Technology Innovation Academia Sinica Taipei11529 Taiwan Social Networks Human-Centered Computing Taiwan International Graduate Program Academia Sinica Taipei11529 Taiwan Department of Computer Science National Chengchi University Taipei11605 Taiwan Department of Computer Science and Information Engineering National Taiwan University Taipei106319 Taiwan

Adversarial attacks have become one of the most serious security issues in widely used deep neural networks. Even though real-world datasets usually have large intra-variations or multiple modes, most adversarial defense methods, such as adversarial training, which is currently one of the most effective defense methods, mainly focus on the single-mode setting and thus fail to capture the full data representation to defend against adversarial attacks. To confront this challenge, we propose a novel multi-prototype metric learning regularization for adversarial training which can effectively enhance the defense capability of adversarial training by preventing the latent representation of the adversarial example changing a lot from its clean one. With extensive experiments on CIFAR10, CIFAR100, MNIST, and Tiny ImageNet, the evaluation results show the proposed method improves the performance of different state-of-the-art adversarial training methods without additional computational cost. Furthermore, besides Tiny ImageNet, in the multi-prototype CIFAR10 and CIFAR100 where we reorganize the whole datasets of CIFAR10 and CIFAR100 into two and ten classes, respectively, the proposed method outperforms the state-of-the-art approach by 2.22% and 1.65%, respectively. Furthermore, the proposed multi-prototype method also outperforms its single-prototype version and other commonly used deep metric learning approaches as regularization for adversarial training and thus further demonstrates its effectiveness. © 2023 by the authors.

关键词： Deep neural networks

来源：评论

学校读者我要写书评

暂无评论

CapST: An Enhanced and Lightweight Method for Deepfake Video Classification

arXiv

引用

arXiv 2023年

作者： Ahmad, Wasim Peng, Yan-Tsung Chang, Yuan-Hao Ganfure, Gaddisa Olani Khan, Sarwar Shahzad, Sahibzada Adil Institute of Information Science Academia Sinica Taipei11529 Taiwan Department of Computer Science National Chengchi University Taipei116 Taiwan Social Networks Human-Centered Computing Taiwan International Graduate Program Academia Sinica Taipei11529 Taiwan Dire Dawa University Ethiopia Research Center for Information Technology Innovation Academia Sinica Taipei11529 Taiwan

The proliferation of deepfake videos, synthetic media produced through advanced Artificial Intelligence techniques, has raised significant concerns across various sectors, encompassing realms such as politics, entertainment, and security. In response, this research introduces an innovative and streamlined model designed to classify deepfake videos generated by five distinct encoders adeptly. Our approach not only achieves state-of-the-art performance but also optimizes computational resources. At its core, our solution employs part of a VGG19_bn as a backbone to efficiently extract features, a strategy proven effective in image-related tasks. We integrate a Capsule Network coupled with a Spatial-Temporal attention mechanism to bolster the model’s classification capabilities while conserving resources. This combination captures intricate hierarchies among features, facilitating robust identification of deepfake attributes. Delving into the intricacies of our innovation, we introduce an existing video-level fusion technique that artfully capitalizes on temporal attention mechanisms. This mechanism serves to handle concatenated feature vectors, capitalizing on the intrinsic temporal dependencies embedded within deepfake videos. By aggregating insights across frames, our model gains a holistic comprehension of video content, resulting in more precise predictions. Experimental results on an extensive benchmark dataset of deepfake videos (DFDM) showcase the efficacy of our proposed method. Notably, our approach achieves up to a 4% improvement in accurately categorizing deepfake videos compared to baseline models, all while demanding fewer computational resources. Copyright © 2023, The Authors. All rights reserved.

关键词： Routing algorithms

来源：评论

学校读者我要写书评

暂无评论

Learning Adaptation and Generalization from human-Inspired Meta-Reinforcement Learning Using Bayesian Knowledge and Analysis

Learning Adaptation and Generalization from Human-Inspired M...

引用

Artificial Intelligence and Knowledge Engineering (AIKE)

作者： Joshua Ho Chien-Min Wang Chung-Ta King Yi-Hsin You Chi-Wei Feng Yen-Min Chen Bo-Yi Kuo Academia Sinica Institute of Information Science Taipei Taiwan Social Networks and Human-Centered Computing Program Taiwan International Graduate Program Institute of Information Systems and Applications National Tsing Hua University Hsinchu Taiwan Center of Intelligent Healthcare National Taiwan University Hospital Taipei Taiwan Department of Computer Science National Tsing Hua University Hsinchu Taiwan Department of Computer Science National Taiwan University Taipei Taiwan

Over the last decades, there has been growing interest in research in multiple and interdisciplinary fields of human-AI computing. In particular, approaches integrating the intersecting design with reinforcement learning (RL) have received more attention. However, the current research on RL may need to consider its enhancement from a human-inspired approach further. In the present work, we focus on enabling a meta-reinforcement learning (meta-RL) agent to achieve adaptation and generalization according to modeling Markov decision processes using Bayesian knowledge and analysis. By introducing a novel framework called human-inspired meta-RL (HMRL), we incorporate the agent performing resilient actions to leverage the dynamic dense reward based on the knowledge and prediction of a Bayesian analysis. The proposed framework can make the agent learn generalization and prevent the agent from failing catastrophically. The experimental results show that our approach helps the agent reduce computational costs with learning adaptation. Finally, we conclude and anticipate that integrating human-inspired meta-RL can enable learning more formulations relating to robustness and scalability, leading to promising directions and more complex AI goals in the future.

关键词：

来源：评论

学校读者我要写书评

暂无评论

AV-Lip-Sync+: Leveraging AV-HuBERT to Exploit Multimodal Inconsistency for Video Deepfake Detection

arXiv

引用

arXiv 2023年

作者： Shahzad, Sahibzada Adil Hashmi, Ammarah Peng, Yan-Tsung Tsao, Yu Wang, Hsin-Min The Social Networks and Human-Centered Computing Program Taiwan International Graduate Program Academia Sinica Taipei11529 Taiwan The Department of Computer Science National Chengchi University Taipei11605 Taiwan The Institute of Information Systems and Applications National Tsing Hua University Hsinchu30013 Taiwan The Research Center for Information Technology Innovation Academia Sinica Taipei11529 Taiwan The Institute of Information Science Academia Sinica Taipei11529 Taiwan

Multimodal manipulations (also known as audiovisual deepfakes) make it difficult for unimodal deepfake detectors to detect forgeries in multimedia content. To avoid the spread of false propaganda and fake news, timely detection is crucial. The damage to either modality (i.e., visual or audio) can only be discovered through multi-modal models that can exploit both pieces of information simultaneously. Previous methods mainly adopt uni-modal video forensics and use supervised pre-training for forgery detection. This study proposes a new method based on a multi-modal self-supervised-learning (SSL) feature extractor to exploit inconsistency between audio and visual modalities for multi-modal video forgery detection. We use the transformer-based SSL pre-trained Audio-Visual HuBERT (AV-HuBERT) model as a visual and acoustic feature extractor and a multi-scale temporal convolutional neural network to capture the temporal correlation between the audio and visual modalities. Since AV-HuBERT only extracts visual features from the lip region, we also adopt another transformer-based video model to exploit facial features and capture spatial and temporal artifacts caused during the deepfake generation process. Experimental results show that our model outperforms all existing models and achieves new state-of-the-art performance on the FakeAVCeleb and DeepfakeTIMIT datasets. © 2023, CC BY.

关键词： Audio acoustics

来源：评论

学校读者我要写书评

暂无评论

HAAQI-Net: A Non-intrusive Neural Music Audio Quality Assessment Model for Hearing Aids

arXiv

引用

arXiv 2024年

作者： Wisnu, Dyah A.M.G. Rini, Stefano Zezario, Ryandhimas E. Wang, Hsin-Min Tsao, Yu Taiwan International Graduate Program—Social Network and Human Centered Computing Institute of Information Science Academia Sinica Taipei Taiwan College of Informatics National Chengchi University Taipei Taiwan Institute of Communications Engineering National Yang Ming Chiao Tung University Hsinchu Taiwan Research Center for Information Technology Innovation Academia Sinica Taipei Taiwan Institute of Information Science Academia Sinica Taipei Taiwan

This paper introduces HAAQI-Net, a non-intrusive deep learning-based music audio quality assessment model for hearing aid users. Unlike traditional methods like the Hearing Aid Audio Quality Index (HAAQI) that require intrusive reference signal comparisons, HAAQI-Net offers a more accessible and computationally efficient alternative. By utilizing a Bidirectional Long Short-Term Memory (BLSTM) architecture with attention mechanisms and features extracted from the pre-trained BEATs model, it can predict HAAQI scores directly from music audio clips and hearing loss patterns. Experimental results demonstrate HAAQI-Net’s effectiveness, achieving a Linear Correlation Coefficient (LCC) of 0.9368, a Spearman’s Rank Correlation Coefficient (SRCC) of 0.9486, and a Mean Squared Error (MSE) of 0.0064 and inference time significantly reduces from 62.52 to 2.54 seconds. To address computational overhead, a knowledge distillation strategy was applied, reducing parameters by 75.85% and inference time by 96.46%, while maintaining strong performance (LCC: 0.9071, SRCC: 0.9307, MSE: 0.0091). To expand its capabilities, HAAQI-Net was adapted to predict subjective human scores like the Mean Opinion Score (MOS) through fine-tuning. This adaptation significantly improved prediction accuracy, validated through statistical analysis. Furthermore, the robustness of HAAQI-Net was evaluated under varying Sound Pressure Level (SPL) conditions, revealing optimal performance at a reference SPL of 65 dB, with accuracy gradually decreasing as SPL deviated from this point. The advancements in subjective score prediction, SPL robustness, and computational efficiency position HAAQI-Net as a scalable solution for music audio quality assessment in hearing aid applications, contributing to efficient and accurate models in audio signal processing and hearing aid technology. Copyright © 2024, The Authors. All rights reserved.

关键词： Mean square error

来源：评论

学校读者我要写书评

暂无评论

Unmasking Illusions: Understanding human Perception of Audiovisual Deepfakes

arXiv

引用

arXiv 2024年

作者： Hashmi, Ammarah Shahzad, Sahibzada Adil Lin, Chia-Wen Tsao, Yu Wang, Hsin-Min Social Networks and Human-Centered Computing Program Taiwan International Graduate Program Institute of Information Science Academia Sinica Taipei11529 Taiwan Institute of Information Systems and Applications National Tsing Hua University Hsinchu300044 Taiwan Department of Computer Science National Chengchi University Taipei11529 Taiwan Department of Electrical Engineering Institute of Communications Engineering National Tsing Hua University Hsinchu300044 Taiwan Research Center for Information Technology Innovation Academia Sinica Taipei11529 Taiwan Institute of Information Science Academia Sinica Taipei11529 Taiwan

The emergence of contemporary deepfakes has attracted significant attention in machine learning research, as artificial intelligence (AI) generated synthetic media increases the incidence of misinterpretation and is difficult to distinguish from genuine content. Currently, machine learning techniques have been extensively studied for automatically detecting deepfakes. However, human perception has been less explored. Malicious deepfakes could ultimately cause public and social problems. Can we humans correctly perceive the authenticity of the content of the videos we watch? The answer is obviously uncertain;therefore, this paper aims to evaluate the human ability to discern deepfake videos through a subjective study. We present our findings by comparing human observers to five state-of-the-art deepfake detection models. To this end, we used gamification concepts to provide 110 participants (55 native English speakers and 55 non-native English speakers) with a web-based platform where they could access a series of 40 videos (20 real and 20 fake) to determine their authenticity. Each participant performed the experiment twice with the same 40 videos in different random orders. The videos are manually selected from the FakeAVCeleb dataset. We found that all AI models performed significantly better than humans when evaluated on the same 40 videos. The study also reveals that while deception is not impossible, humans tend to overestimate their detection capabilities. Our experimental results may help benchmark human versus machine performance, advance forensics analysis, and enable adaptive countermeasures. © 2024, CC BY.

关键词： Audiovisual

来源：评论

学校读者我要写书评

暂无评论

Understanding Audiovisual Deepfake Detection: Techniques, Challenges, human Factors and Perceptual Insights

arXiv

引用

arXiv 2024年

作者： Hashmi, Ammarah Shahzad, Sahibzada Adil Lin, Chia-Wen Tsao, Yu Wang, Hsin-Min The Social Networks and Human-Centered Computing Program Taiwan International Graduate Program Institute of Information Science Academia Sinica Taipei11529 Taiwan The Institute of Information Systems and Applications National Tsing Hua University Hsinchu300044 Taiwan The Department of Computer Science National Chengchi University Taipei11529 Taiwan The Department of Electrical Engineering The Institute of Communications Engineering National Tsing Hua University Hsinchu300044 Taiwan The Research Center for Information Technology Innovation Academia Sinica Taipei11529 Taiwan The Institute of Information Science Academia Sinica Taipei11529 Taiwan

Deep Learning has been successfully applied in diverse fields, and its impact on deepfake detection is no exception. Deepfakes are fake yet realistic synthetic content that can be used deceitfully for political impersonation, phishing, slandering, or spreading misinformation. Despite extensive research on unimodal deepfake detection, identifying complex deepfakes through joint analysis of audio and visual streams remains relatively unexplored. To fill this gap, this survey first provides an overview of audiovisual deepfake generation techniques, applications, and their consequences, and then provides a comprehensive review of state-of-the-art methods that combine audio and visual modalities to enhance detection accuracy, summarizing and critically analyzing their strengths and limitations. Furthermore, we discuss existing open source datasets for a deeper understanding, which can contribute to the research community and provide necessary information to beginners who want to analyze deep learning-based audiovisual methods for video forensics. By bridging the gap between unimodal and multimodal approaches, this paper aims to improve the effectiveness of deepfake detection strategies and guide future research in cybersecurity and media integrity. © 2024, CC BY.

关键词： Audiovisual

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：