Learning Features of Intra-Consistency and Inter-Diversity: Keys Toward Generalizable Deepfake Detection

作     者：Chen, Han Lin, Yuzhen Li, Bin Tan, Shunquan 

作者机构：Shenzhen Univ Guangdong Key Lab Intelligent Informat Proc Shenzhen Key Lab Media Secur Shenzhen 518060 Peoples R China Shenzhen Univ Guangdong Lab Artificial Intelligence & Digital Ec Shenzhen 518060 Peoples R China 

出 版 物：《IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY》 (IEEE Trans Circuits Syst Video Technol)

年 卷 期：2023年第33卷第3期

页      面：1468-1480页

核心收录：

学科分类：0808[工学-电气工程] 08[工学] 

基　　金：NSFC [61872244, 62272314] Basic and Applied Basic Research Foundation [2019B151502001] Shenzhen Researchand Development Program [JCYJ20200109105008228] 

主　　题：Deepfakes Task analysis Forgery Faces Feature extraction Data models Transformers Deepfake detection self-supervised learning masked image modeling transformer generalization 

摘      要：Public concerns about deepfake face forgery are continually rising in recent years. Most deepfake detection approaches attempt to learn discriminative features between real and fake faces through end-to-end trained deep neural networks. However, the majorities of them suffer from the problem of poor generalization across different data sources, forgery methods, and/or post-processing operations. In this paper, following the simple but effective principle in discriminative representation learning, i.e., towards learning features of intra-consistency within classes and inter-diversity between classes, we leverage a novel transformer-based self-supervised learning method and an effective data augmentation strategy towards generalizable deepfake detection. Considering the differences between the real and fake images are often subtle and local, the proposed method firstly utilizes Self Prediction Learning (SPL) to learn rich hidden representations by predicting masked patches at a pre-training stage. Intra-class consistency clues in images can be mined without deepfake labels. After pre-training, the discrimination model is then fine-tuned via multi-task learning, including a deepfake classification task and a forgery mask estimation task. It is facilitated by our new data augmentation method called Adjustable Forgery Synthesizer (AFS), which can conveniently simulate the process of synthesizing deepfake images with various levels of visual reality in an explicit manner. AFS greatly prevents overfitting due to insufficient diversity in training data. Comprehensive experiments demonstrate that our method outperforms the state-of-the-art competitors on several popular benchmark datasets in terms of generalization to unseen forgery methods and untrained datasets.