检索结果-内蒙古大学图书馆

CapST: Leveraging Capsule networks and Temporal Attention for Accurate Model Attribution in Deep-fake Videos

ACM Transactions on Multimedia computing, Communications and Applications 2025年第4期21卷

作者： Ahmad, Wasim Peng, Yan-Tsung Chang, Yuan-Hao Ganfure, Gaddisa Olani Khan, Sarwar Social Networks and Human-Centered Computing Taiwan International Graduate Program Institute of Information Science Academia Sinica Taipei Taiwan Department of Computer Science National Chengchi University Taipei Taiwan Institute of Information Science Academia Sinica Taipei Taiwan Department of Computer Science Dire Dawa University Dire Dawa Ethiopia Social Networks and Human-Centered Computing Taiwan International Graduate Program Research Center for Information Technology Innovation Academia Sinica Taipei Taiwan

Deep-fake videos, generated through AI face-swapping techniques, have garnered considerable attention due to their potential for impactful impersonation attacks. While existing research primarily distinguishes real from fake videos, attributing a deep-fake to its specific generation model or encoder is crucial for forensic investigation, enabling precise source tracing and tailored countermeasures. This approach not only enhances detection accuracy by leveraging unique model-specific artifacts but also provides insights essential for developing proactive defenses against evolving deep-fake techniques. Addressing this gap, this article investigates the model attribution problem for deep-fake videos using two datasets-Deepfakes from Different Models (DFDM) and GANGen-Detection, which comprise deep-fake videos and images generated by GAN models. We select only fake images from the GANGen-Detection dataset to align with the DFDM dataset, which specifies the goal of this study, focusing on model attribution rather than real/fake classification. This study formulates deep-fake model attribution as a multiclass classification task, introducing a novel Capsule-Spatial-Temporal (CapST) model that effectively integrates a modified VGG19 (utilizing only the first 26 out of 52 layers) for feature extraction, combined with Capsule networks and a Spatio-Temporal attention mechanism. The Capsule module captures intricate feature hierarchies, enabling robust identification of deep-fake attributes, while a video-level fusion technique leverages temporal attention mechanisms to process concatenated feature vectors and capture temporal dependencies in deep-fake videos. By aggregating insights across frames, our model achieves a comprehensive understanding of video content, resulting in more precise predictions. Experimental results on the DFDM and GANGen-Detection datasets demonstrate the efficacy of CapST, achieving substantial improvements in accurately categorizing deep-fake videos

关键词： Computer forensics

来源：评论

学校读者我要写书评

暂无评论

AVTENet: A human-Cognition-Inspired Audio-Visual Transformer-Based Ensemble Network for Video Deepfake Detection

引用

IEEE Transactions on Cognitive and Developmental Systems 2025年

作者： Hashmi, Ammarah Shahzad, Sahibzada Adil Lin, Chia-Wen Tsao, Yu Wang, Hsin-Min Taiwan International Graduate Program Academia Sinica Social Networks and Human-Centered Computing Program Taipei11529 Taiwan National Tsing Hua University Institute of Information Systems and Applications Hsinchu30013 Taiwan National Chengchi University Department of Computer Science Taipei11605 Taiwan National Tsing Hua University Department of Electrical Engineering Institute of Communications Engineering Hsinchu300044 Taiwan Academia Sinica Research Center for Information Technology Innovation Taipei11529 Taiwan

The recent proliferation of hyper-realistic deepfake videos has drawn attention to the threat of audio and visual forgeries. Most previous studies on detecting artificial intelligence-generated fake videos only utilize visual modality or audio modality. While some methods exploit audio and visual modalities to detect forged videos, they have not been comprehensively evaluated on multimodal datasets of deepfake videos involving acoustic and visual manipulations, and are mostly based on convolutional neural networks with low detection accuracy. Considering that human cognition instinctively integrates multisensory information including audio and visual cues to perceive and interpret content and the success of transformer in various fields, this study introduces the audio-visual transformer-based ensemble network (AVTENet). This innovative framework tackles the complexities of deepfake technology by integrating both acoustic and visual manipulations to enhance the accuracy of video forgery detection. Specifically, the proposed model integrates several purely transformer-based variants that capture video, audio, and audio-visual salient cues to reach a consensus in prediction. For evaluation, we use the recently released benchmark multimodal audio-video FakeAVCeleb dataset. For a detailed analysis, we evaluate AVTENet, its variants, and several existing methods on multiple test sets of the FakeAVCeleb dataset. Experimental results show that the proposed model outperforms all existing methods and achieves state-of-the-art performance on Testset-I and Testset-II of the FakeAVCeleb dataset. We also compare AVTENet against humans in detecting video forgery. The results show that AVTENet significantly outperforms humans. © 2016 IEEE.

关键词： audio-visual AVTENet Deepfake detection

来源：评论

学校读者我要写书评

暂无评论

HAAQI-Net: A Non-Intrusive Neural Music Audio Quality Assessment Model for Hearing Aids

引用

IEEE Transactions on Audio, Speech and Language Processing 2025年 33卷 1877-1892页

作者： Dyah A. M. G. Wisnu Stefano Rini Ryandhimas E. Zezario Hsin-Min Wang Yu Tsao Taiwan International Graduate Program—Social Network and Human-Centered Computing Institute of Information Science Academia Sinica Taipei Taiwan College of Informatics National Chengchi University Taipei Taiwan Institute of Communications Engineering National Yang Ming Chiao Tung University Hsinchu Taiwan Research Center for Information Technology Innovation Academia Sinica Taipei Taiwan Institute of Information Science Academia Sinica Taipei Taiwan Department of Electrical Engineering Chung Yuan Christian University Taiwan

This paper introduces HAAQI-Net, a non-intrusive music audio quality assessment model for hearing aid users. Unlike traditional methods such as Hearing Aid Audio Quality Index (HAAQI), which requires intrusive reference signal comparisons, HAAQI-Net offers a more accessible and computationally efficient alternative. Leveraging a bidirectional long short-term memory architecture with attention mechanisms and features extracted from a pre-trained BEATs model, it can predict HAAQI scores directly from music audio clips and hearing loss patterns. The experimental results demonstrate that, compared to the traditional HAAQI as the reference, HAAQI-Net achieves a linear correlation coefficient (LCC) of 0.9368, a Spearman's rank correlation coefficient (SRCC) of 0.9486, and a mean squared error (MSE) of 0.0064, while significantly reducing the inference time from 62.52 seconds to 2.54 seconds. Furthermore, a knowledge distillation strategy was applied, reducing the parameters by 75.85% and inference time by 96.46%, while maintaining strong performance (LCC: 0.9071, SRCC: 0.9307, MSE: 0.0091). To expand its capabilities, HAAQI-Net was adapted to predict subjective human scores, mean opinion score (MOS), by fine-tuning. This adaptation significantly improved the prediction accuracy. Furthermore, the robustness of HAAQI-Net was evaluated under varying sound pressure level (SPL) conditions, revealing optimal performance at a reference SPL of 65 dB, with the accuracy gradually decreasing as SPL deviated from this point. The advancements in subjective score prediction, SPL robustness, and computational efficiency position HAAQI-Net as a reliable solution for music audio quality assessment, significantly contributing to the development of efficient and accurate models in audio signal processing and hearing aid technology.

关键词： Hearing aids Quality assessment Training Feature extraction Speech processing Auditory system Computational modeling Adaptation models Transformers Multiple signal classification

来源：评论

学校读者我要写书评

暂无评论

Demo: CAR: The cleanest air routing algorithm for path navigation with minimal PM2.5 exposure on the move 16

Demo: CAR: The cleanest air routing algorithm for path navig...

引用

16th ACM International Conference on Mobile Systems, Applications, and Services,MobiSys 2018

作者： Mahajan, Sachit Tang, Yu-Siou Wu, Dong-Yi Tsai, Tzu-Chieh Chen, Ling-Jyh Social Networks and Human Centered Computing Program TIGP Taiwan Institute of Information Science Academia Sinica Taiwan National Chengchi University Taiwan

来源：评论

学校读者我要写书评

暂无评论

An empirical study of PM2.5 forecasting using neural network

An empirical study of PM2.5 forecasting using neural network

引用

2017 IEEE SmartWorld Ubiquitous Intelligence and computing, Advanced and Trusted Computed, Scalable computing and Communications, Cloud and Big Data computing, Internet of People and Smart City Innovation, SmartWorld/SCALCOM/UIC/ATC/CBDCom/IOP/SCI 2017

作者： Mahajan, Sachit Chen, Ling-Jyh Tsai, Tzu-Chieh Social Networks and Human Centered Computing Program Taiwan International Graduate Program Taiwan Institute of Information Science Academia Sinica Taiwan Department of Computer Science National Chengchi University Taiwan

ISBN: (纸本)9781538604342

In the recent years, a lot of efforts have been made to regulate air pollutant levels in most of the developed and developing countries. Fine particulate matter (PM2.5) is considered to be one of the major reasons behind deteriorating public health and a lot of efforts are being made to keep a check on PM2.5 levels. Accurately forecasting PM2.5 level is a challenging task and has been highly dependent on model based approaches. In this paper, we explore new possibilities to hourly forecast PM2.5. Choosing the right forecasting model becomes a very important aspect when it comes to improvement in prediction accuracy. We used Neural Network Autoregression (NNAR) method for the prediction task. The paper also provides a comparative analysis of prediction performance for additive version of Holt-Winters method, autoregressive integrated moving average (ARIMA) model and NNAR model. The experimentation and evaluation is done using real world measurement data from Airbox Project, which shows that our proposed method accurately does the prediction with significantly low error. © 2017 IEEE.

关键词： Neural networks

来源：评论

学校读者我要写书评

暂无评论

Audio-Driven Facial Landmark Generation in Violin Performance using 3DCNN Network with Self Attention Model 48

Audio-Driven Facial Landmark Generation in Violin Performanc...

引用

48th IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2023

作者： Lin, Ting-Wei Liu, Chao-Lin Su, Li Taiwan International Graduate Program Social Networks and Human-Centered Computing Taiwan Academia Sinica Institute of Information Science Taipei Taiwan National Chengchi University Department of Computer Science Taipei Taiwan

ISBN: (纸本)9781728163277

In a music scenario, both auditory and visual elements are essential to achieve an outstanding performance. Recent research has focused on the generation of body movements or fingering from audio in music performance. The audio-driven face generation technique in music performance is still deficient. In this paper, we compile a violin soundtrack and facial expression dataset (VSFE) for modeling facial expressions in violin performance. To our knowledge, this is the first dataset mapping the relationship between violin performance audio and musicians' facial expressions. We then propose a 3DCNN network with self-attention and residual blocks for audio-driven facial expression generation. In the experiments, we compare our methods with three baselines on talking face generation. The codes and dataset are available on the Github (https://***/kevinlin91/icassp-music2face). © 2023 IEEE.

关键词： Music

来源：评论

学校读者我要写书评

暂无评论

Text and Speech Adjustable Bimodal Emotion Recognition in Conversational Interface 1

Text and Speech Adjustable Bimodal Emotion Recognition in Co...

引用

1st IEEE International Symposium on Consumer Technology, ISCT 2024

作者： Rinartha, Komang Ku, Lun-Wei Social Networks and Human Centered Computing Program Taiwan International Graduate Program Institute of Information Science 128 Academia Road Sec. 2. Nangang Taipei115 Taiwan Institute of Information Science Academia Sinica Taipei Taiwan

ISBN: (纸本)9798350365191

Speech emotion recognition (SER) is a Machine Learning (ML) topic that is now receiving a lot of research attention. This can be attributed to its growing capacity, improvements in algorithms, and utilization in practical situations. Furthermore, speech contains not just auditory information but also lexical information. Both speech-based emotion recognition and text-based emotion recognition, called bimodal emotion recognition, can be highly beneficial for human interaction. Take an example, maintaining the communication between elder parents who are left behind after their children leave for education, employment, and family dynamics who can be greatly affected, both physically and emotionally, by feelings of loneliness and social isolation. Bimodal emotion recognition employs both text-based and audio-based methods to identify emotions. Each algorithm was trained separately and then combined using a late fusion technique. RoBERTa text-based emotion recognition can be combined with CNN audio-based emotion recognition by using late fusion to create a bimodal emotion recognition that can obtain more than 50% of accuracy by combining text and audio in a similar proportion or relying on text more than audio. By adjusting the proportion of text or audio emotion recognition, we can get better result of recognition and it can be customized by the user's preference. We elaborate on creating bimodal emotion recognition, conversational interface, a framework for emotion monitoring system, and give the example of a combined algorithm to perform emotion recognition. © 2024 IEEE.

关键词： Emotion Recognition

来源：评论

学校读者我要写书评

暂无评论

SwapItUp: A Face Swap Application for Privacy Protection 31

SwapItUp: A Face Swap Application for Privacy Protection

引用

31st IEEE International Conference on Advanced Information Networking and Applications (IEEE AINA)

作者： Mahajan, Sachit Chen, Ling-Jyh Tsai, Tzu-Chieh Taiwan Int Grad Program Social Networks & Human Centered Comp Program Taipei Taiwan Acad Sinica Inst Informat Sci Taipei Taiwan Natl Chengchi Univ Dept Comp Sci Taipei Taiwan

ISBN: (纸本)9781509060290

There is a growing concern over the issues related to online privacy due to large availability of high quality images. To tackle the privacy concerns a face swapping application is proposed. There is a library of face images, which is created by downloading images from different sources on Internet. For any given image, first of all facial landmarks are detected. The second image is rotated and scaled so that it can properly fit over the input image. To make sure that the new image looks natural, color balance adjustment is done. After that blending of features from the second image onto the input image is done. It is also shown how this system can be used to creating appealing and funny photographs for entertainment purposes. We conclude with a study that shows the high quality of images produced by this system as compared to existing face swap applications and also limitations of this system.

关键词： face swap feature detection privacy protection

来源：评论

学校读者我要写书评

暂无评论

Influence Value: Quantifying Topic Influence on social Media 2

Influence Value: Quantifying Topic Influence on Social Media

引用

2nd IEEE International Conference on Multimedia Big Data (BigMM)

作者： Calderon, Fernando H. Chen, Yi-Shin Natl Tsing Hua Univ Inst Informat Syst & Applicat Hsinchu 30013 Taiwan Acad Sinica Taiwan Int Grad Program Social Networks & Human Centered Comp Taipei 115 Taiwan

ISBN: (纸本)9781509021796

social Media specifically Micro-blogging sites have become very rich data repositories. However the generated data is dynamic by nature, tied to temporal conditions and the subjectivity of its users. Everyday life experiences, discussions or events have a direct impact on the behaviors reflected in social networks. It is therefore of great importance to asses the impact these interactions are having over a social group. An alternative to answer this is determining how influential a topic is according to the behavior presented on a social network over time. It is then necessary to find and develop methods that can contribute towards this task. Having identified a topic in social media we can first classify it as time specific or long term, then posts relevant to the topic are collected and each one is assigned an emotion label. We then propose an Influence Value score which will be given to each topic based on its lifespan, emotion transition and reach. This lays ground to quantify how influential a topic is over a social group, specifically from events detected on twitter.

关键词： Emotion detection Influence social media

来源：评论

学校读者我要写书评

暂无评论

Reactive Supervision: A New Method for Collecting Sarcasm Data

Reactive Supervision: A New Method for Collecting Sarcasm Da...

引用

Conference on Empirical Methods in Natural Language Processing (EMNLP)

作者： Shmueli, Boaz Kul, Lun-Wei Ray, Soumya Taiwan Int Grad Program Social Networks & Human Centered Comp Taipei Taiwan Acad Sinica Inst Informat Sci Taipei Taiwan Natl Tsing Hua Univ Inst Serv Sci Hsinchu Taiwan

ISBN: (纸本)9781952148606

Sarcasm detection is an important task in affective computing, requiring large amounts of labeled data. We introduce reactive supervision, a novel data collection method that utilizes the dynamics of online conversations to overcome the limitations of existing data collection techniques. We use the new method to create and release a first-of-its-kind large dataset of tweets with sarcasm perspective labels and new contextual features. The dataset is expected to advance sarcasm detection research. Our method can be adapted to other affective computing domains, thus opening up new research opportunities.

关键词： Large dataset

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：