检索结果-内蒙古大学图书馆

Towards Adversarial Robustness for Multi-Mode Data through Metric Learning

学校读者我要写书评

暂无评论

Sensors 2023年第13期23卷 6173页

作者： Khan, Sarwar Chen, Jun-Cheng Liao, Wen-Hung Chen, Chu-Song Research Center for Information Technology Innovation Academia Sinica Taipei11529 Taiwan Social Networks Human-Centered Computing Taiwan International Graduate Program Academia Sinica Taipei11529 Taiwan Department of Computer Science National Chengchi University Taipei11605 Taiwan Department of Computer Science and Information Engineering National Taiwan University Taipei106319 Taiwan

Adversarial attacks have become one of the most serious security issues in widely used deep neural networks. Even though real-world datasets usually have large intra-variations or multiple modes, most adversarial defense methods, such as adversarial training, which is currently one of the most effective defense methods, mainly focus on the single-mode setting and thus fail to capture the full data representation to defend against adversarial attacks. To confront this challenge, we propose a novel multi-prototype metric learning regularization for adversarial training which can effectively enhance the defense capability of adversarial training by preventing the latent representation of the adversarial example changing a lot from its clean one. With extensive experiments on CIFAR10, CIFAR100, MNIST, and Tiny ImageNet, the evaluation results show the proposed method improves the performance of different state-of-the-art adversarial training methods without additional computational cost. Furthermore, besides Tiny ImageNet, in the multi-prototype CIFAR10 and CIFAR100 where we reorganize the whole datasets of CIFAR10 and CIFAR100 into two and ten classes, respectively, the proposed method outperforms the state-of-the-art approach by 2.22% and 1.65%, respectively. Furthermore, the proposed multi-prototype method also outperforms its single-prototype version and other commonly used deep metric learning approaches as regularization for adversarial training and thus further demonstrates its effectiveness. © 2023 by the authors.

关键词： Deep neural networks

Self-Supervised Denoising Autoencoder with Linear Regression Decoder for Speech Enhancement

学校读者我要写书评

暂无评论

Self-Supervised Denoising Autoencoder with Linear Regression...

IEEE international Conference on Acoustics, Speech and Signal Processing

作者： Ryandhimas E. Zezario Tassadaq Hussain Xugang Lu Hsin-Min Wang Yu Tsao Research Center for Information Technology Innovation Academia Sinica Taiwan Taiwan International Graduate Program in Social Network and Human-Centered Computing Institute of Information Science Academia Sinica Taiwan National Institute of Information and Communications Technology Japan Institute of Information Science Academia Sinica Taiwan

ISBN: (数字)9781509066315

ISBN: (纸本)9781509066322

Nonlinear spectral mapping-based models based on supervised learning have successfully applied for speech enhancement. However, as supervised learning approaches, a large amount of labelled data (noisy-clean speech pairs) should be provided to train those models. In addition, their performances for unseen noisy conditions are not guaranteed, which is a common weak point of supervised learning approaches. In this study, we proposed an unsupervised learning approach for speech enhancement, i.e., denoising autoencoder with linear regression decoder (DAELD) model for speech enhancement. The DAELD is trained with noisy speech as both input and target output in a self-supervised learning manner. In addition, with properly setting a shrinkage threshold for internal hidden representations, noise could be removed during the reconstruction from the hidden representations via the linear regression decoder. Speech enhancement experiments were carried out to test the proposed model. Results confirmed that the proposed DAELD could achieve comparable and sometimes even better enhancement performance as compared to the conventional supervised speech enhancement approaches, in both seen and unseen noise environments. Moreover, we observe that higher performances tend to achieve by DAELD when the training data cover more diverse noise types and signal-tonoise-ratio (SNR) levels.

关键词：

CapST: An Enhanced and Lightweight Method for Deepfake Video Classification

学校读者我要写书评

暂无评论

arXiv 2023年

作者： Ahmad, Wasim Peng, Yan-Tsung Chang, Yuan-Hao Ganfure, Gaddisa Olani Khan, Sarwar Shahzad, Sahibzada Adil Institute of Information Science Academia Sinica Taipei11529 Taiwan Department of Computer Science National Chengchi University Taipei116 Taiwan Social Networks Human-Centered Computing Taiwan International Graduate Program Academia Sinica Taipei11529 Taiwan Dire Dawa University Ethiopia Research Center for Information Technology Innovation Academia Sinica Taipei11529 Taiwan

The proliferation of deepfake videos, synthetic media produced through advanced Artificial Intelligence techniques, has raised significant concerns across various sectors, encompassing realms such as politics, entertainment, and security. In response, this research introduces an innovative and streamlined model designed to classify deepfake videos generated by five distinct encoders adeptly. Our approach not only achieves state-of-the-art performance but also optimizes computational resources. At its core, our solution employs part of a VGG19_bn as a backbone to efficiently extract features, a strategy proven effective in image-related tasks. We integrate a Capsule Network coupled with a Spatial-Temporal attention mechanism to bolster the model’s classification capabilities while conserving resources. This combination captures intricate hierarchies among features, facilitating robust identification of deepfake attributes. Delving into the intricacies of our innovation, we introduce an existing video-level fusion technique that artfully capitalizes on temporal attention mechanisms. This mechanism serves to handle concatenated feature vectors, capitalizing on the intrinsic temporal dependencies embedded within deepfake videos. By aggregating insights across frames, our model gains a holistic comprehension of video content, resulting in more precise predictions. Experimental results on an extensive benchmark dataset of deepfake videos (DFDM) showcase the efficacy of our proposed method. Notably, our approach achieves up to a 4% improvement in accurately categorizing deepfake videos compared to baseline models, all while demanding fewer computational resources. Copyright © 2023, The Authors. All rights reserved.

关键词： Routing algorithms

Learning Adaptation and Generalization from human-Inspired Meta-Reinforcement Learning Using Bayesian Knowledge and Analysis

学校读者我要写书评

暂无评论

Learning Adaptation and Generalization from Human-Inspired M...

Artificial Intelligence and Knowledge Engineering (AIKE)

作者： Joshua Ho Chien-Min Wang Chung-Ta King Yi-Hsin You Chi-Wei Feng Yen-Min Chen Bo-Yi Kuo Academia Sinica Institute of Information Science Taipei Taiwan Social Networks and Human-Centered Computing Program Taiwan International Graduate Program Institute of Information Systems and Applications National Tsing Hua University Hsinchu Taiwan Center of Intelligent Healthcare National Taiwan University Hospital Taipei Taiwan Department of Computer Science National Tsing Hua University Hsinchu Taiwan Department of Computer Science National Taiwan University Taipei Taiwan

Over the last decades, there has been growing interest in research in multiple and interdisciplinary fields of human-AI computing. In particular, approaches integrating the intersecting design with reinforcement learning (RL) have received more attention. However, the current research on RL may need to consider its enhancement from a human-inspired approach further. In the present work, we focus on enabling a meta-reinforcement learning (meta-RL) agent to achieve adaptation and generalization according to modeling Markov decision processes using Bayesian knowledge and analysis. By introducing a novel framework called human-inspired meta-RL (HMRL), we incorporate the agent performing resilient actions to leverage the dynamic dense reward based on the knowledge and prediction of a Bayesian analysis. The proposed framework can make the agent learn generalization and prevent the agent from failing catastrophically. The experimental results show that our approach helps the agent reduce computational costs with learning adaptation. Finally, we conclude and anticipate that integrating human-inspired meta-RL can enable learning more formulations relating to robustness and scalability, leading to promising directions and more complex AI goals in the future.

关键词：

Zema Dataset: A Comprehensive Study of Yaredawi Zema with a Focus on Horologium Chants

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Muluneh, Mequanent Argaw Peng, Yan-Tsung Degife, Worku Abebe Tadesse, Nigussie Abate Demeku, Aknachew Mebreku Su, Li Institute of Information Science Academia Sinica Taipei Taiwan Department of Information System University of Gondar Gondar Ethiopia Department of Computer Science National Chengchi University Taipei Taiwan Social Networks and Human-Centered Computing Program Taiwan International Graduate Program Taiwan Department of Information Management National Taiwan University of Science and Technology Taipei Taiwan Department of Material Science and Engineering National Taiwan University of Science and Technology Taipei Taiwan

Computational music research plays a critical role in advancing music production, distribution, and understanding across various musical styles in the world. Despite the immense cultural and religious significance, the Ethiopian Orthodox Tewahedo Church (EOTC) chants are relatively underrepresented in computational music research. This paper contributes to this field by introducing a new dataset specifically tailored for the analysis of EOTC chants, also known as Yaredawi Zema. This work provides a comprehensive overview of a 10-hour dataset, 369 instances, creation, and curation process, including rigorous quality assurance measures. Our dataset has a detailed word-level temporal boundary and reading tone annotation along with the corresponding chanting mode label of audios. Moreover, we have also identified the chanting options associated with multiple chanting notations in the manuscript by annotating them accordingly. Our goal in making this dataset available to the public1 is to encourage more research and study of EOTC chants, including lyrics transcription, lyric-to-audio alignment, and music generation tasks. Such research work will advance knowledge and efforts to preserve this distinctive liturgical music, a priceless cultural artifact for the Ethiopian people. © 2024, CC BY.

关键词： Music

AV-Lip-Sync+: Leveraging AV-HuBERT to Exploit Multimodal Inconsistency for Video Deepfake Detection

学校读者我要写书评

暂无评论

arXiv 2023年

作者： Shahzad, Sahibzada Adil Hashmi, Ammarah Peng, Yan-Tsung Tsao, Yu Wang, Hsin-Min The Social Networks and Human-Centered Computing Program Taiwan International Graduate Program Academia Sinica Taipei11529 Taiwan The Department of Computer Science National Chengchi University Taipei11605 Taiwan The Institute of Information Systems and Applications National Tsing Hua University Hsinchu30013 Taiwan The Research Center for Information Technology Innovation Academia Sinica Taipei11529 Taiwan The Institute of Information Science Academia Sinica Taipei11529 Taiwan

Multimodal manipulations (also known as audiovisual deepfakes) make it difficult for unimodal deepfake detectors to detect forgeries in multimedia content. To avoid the spread of false propaganda and fake news, timely detection is crucial. The damage to either modality (i.e., visual or audio) can only be discovered through multi-modal models that can exploit both pieces of information simultaneously. Previous methods mainly adopt uni-modal video forensics and use supervised pre-training for forgery detection. This study proposes a new method based on a multi-modal self-supervised-learning (SSL) feature extractor to exploit inconsistency between audio and visual modalities for multi-modal video forgery detection. We use the transformer-based SSL pre-trained Audio-Visual HuBERT (AV-HuBERT) model as a visual and acoustic feature extractor and a multi-scale temporal convolutional neural network to capture the temporal correlation between the audio and visual modalities. Since AV-HuBERT only extracts visual features from the lip region, we also adopt another transformer-based video model to exploit facial features and capture spatial and temporal artifacts caused during the deepfake generation process. Experimental results show that our model outperforms all existing models and achieves new state-of-the-art performance on the FakeAVCeleb and DeepfakeTIMIT datasets. © 2023, CC BY.

关键词： Audio acoustics

Medical image denoising using sparse representations

学校读者我要写书评

暂无评论

Medical image denoising using sparse representations

international Conference on Awareness Science and Technology (iCAST)

作者： Fatma S. Abousaleh Neng-Hao Yu Kai-Lung Hua Wen-Huang Cheng Social Networks and Human Centered Computing Program Institute of Information Science Research Center for Information Technology Innovation Academia Sinica Taipei Taiwan Department of Computer Science National Chengchi University Taipei Taiwan Department of Computer Science and Information Engineering Taiwan University of Science and Technology Taipei Taiwan

Image noise reduction, or denoising, became an attractive research domain in the recent years. There are several published approaches in the image denoising field and each one has its presumptions, advantages, and limitations. In this paper, we address the problem of eliminating or even decreasing the noise from medical images. Some schemes for image noise reduction are proposed. The schemes include DCT_OMP, DCT_BOMP, Log Gabor_BOMP, DCT_OCMP and Wavelet_OMP. In these schemes, the image content is represented as a sparse linear combination of a set of atoms that can be obtained from some trained dictionaries. In literature, several algorithms are proposed to build up these kinds of dictionaries. In our study, the K-SVD algorithm is utilized to obtain a dictionary that can effectively describe the image. Also, some greedy algorithms are used to perform the sparse-coding of the signal. The proposed schemes have been tested on two different medical images with different levels of an additive noise. The Log Gabor_BOMP scheme showed a significant outperformance, in both the de-noising results and computational time, on all the other schemes.

关键词： Matching pursuit algorithms Dictionaries Approximation algorithms Image denoising Noise reduction Discrete cosine transforms Algorithm design and analysis

Multimodal deep learning framework for image popularity prediction on social media

学校读者我要写书评

暂无评论

arXiv 2021年

作者： Abousaleh, Fatma S. Cheng, Wen-Huang Yu, Neng-Hao Tsao, Yu Social Networks and Human-Centered Computing Program Taiwan International Graduate Program Institute of Information Science Research Center for Information Technology Innovation Academia Sinica Taipei11529 Taiwan Department of Computer Science National Chengchi University Taipei11605 Taiwan The Institute of Electronics Department of Electronics Engineering National Chiao Tung University Hsinchu30010 Taiwan The College of Design Department of Design National Taiwan University of Science and Technology Taipei10607 Taiwan The Research Center for Information Technology Innovation Academia Sinica Taipei11529 Taiwan

Billions of photos are uploaded to the web daily through various types of social networks. Some of these images receive millions of views and become popular, whereas others remain completely unnoticed. This raises the problem of predicting image popularity on social media. The popularity of an image can be affected by several factors, such as visual content, aesthetic quality, user, post metadata, and time. Thus, considering all these factors is essential for accurately predicting image popularity. In addition, the efficiency of the predictive model also plays a crucial role. In this study, motivated by multimodal learning, which uses information from various modalities, and the current success of convolutional neural networks (CNNs) in various fields, we propose a deep learning model, called visual-social convolutional neural network (VSCNN), which predicts the popularity of a posted image by incorporating various types of visual and social features into a unified network model. VSCNN first learns to extract high-level representations from the input visual and social features by utilizing two individual CNNs. The outputs of these two networks are then fused into a joint network to estimate the popularity score in the output layer. We assess the performance of the proposed method by conducting extensive experiments on a dataset of approximately 432K images posted on Flickr. The simulation results demonstrate that the proposed VSCNN model significantly outperforms state-of-the-art models, with a relative improvement of greater than 2.33%, 7.59%, and 14.16% in terms of Spearman’s Rho, mean absolute error, and mean squared error, respectively. Copyright © 2021, The Authors. All rights reserved.

关键词： social networking (online)

AVTENet: Audio-Visual Transformer-based Ensemble Network Exploiting Multiple Experts for Video Deepfake Detection

学校读者我要写书评

暂无评论

arXiv 2023年

作者： Hashmi, Ammarah Shahzad, Sahibzada Adil Lin, Chia-Wen Tsao, Yu Wang, Hsin-Min The Social Networks and Human-Centered Computing Program Taiwan International Graduate Program Academia Sinica Taipei11529 Taiwan The Institute of Information Systems and Applications National Tsing Hua University Hsinchu30013 Taiwan The Department of Computer Science National Chengchi University Taipei11605 Taiwan The Department of Electrical Engineering The Institute of Communications Engineering National Tsing Hua University Hsinchu300044 Taiwan The Research Center for Information Technology Innovation Academia Sinica Taipei11529 Taiwan The Institute of Information Science Academia Sinica Taipei11529 Taiwan

Forged content shared widely on social media platforms is a major social problem that requires increased regulation and poses new challenges to the research community. The recent proliferation of hyper-realistic deepfake videos has drawn attention to the threat of audio and visual forgeries. Most previous work on detecting AI-generated fake videos only utilizes visual modality or audio modality. While there are some methods in the literature that exploit audio and visual modalities to detect forged videos, they have not been comprehensively evaluated on multi-modal datasets of deepfake videos involving acoustic and visual manipulations. Moreover, these existing methods are mostly based on CNN and suffer from low detection accuracy. Inspired by the recent success of Transformer in various fields, to address the challenges posed by deepfake technology, in this paper, we propose an Audio-Visual Transformer-based Ensemble Network (AVTENet) framework that considers both acoustic manipulation and visual manipulation to achieve effective video forgery detection. Specifically, the proposed model integrates several purely transformer-based variants that capture video, audio, and audio-visual salient cues to reach a consensus in prediction. For evaluation, we use the recently released benchmark multi-modal audio-video FakeAVCeleb dataset. For a detailed analysis, we evaluate AVTENet, its variants, and several existing methods on multiple test sets of the FakeAVCeleb dataset. Experimental results show that our best model outperforms all existing methods and achieves state-of-the-art performance on Testset-I and Testset-II of the FakeAVCeleb dataset. © 2023, CC BY.

关键词： Statistical tests