检索结果-内蒙古大学图书馆

ES3: Evolving Self-Supervised Learning of Robust Audio-Visual Speech Representations

学校读者我要写书评

暂无评论

ES3: Evolving Self-Supervised Learning of Robust Audio-Visua...

Conference on Computer Vision and Pattern Recognition (CVPR)

作者： Yuanhang Zhang Shuang Yang Shiguang Shan Xilin Chen Key Laboratory of Intelligent Information Processing of Chinese Academy of Sciences (CAS) Institute of Computing Technology CAS Beijing China University of Chinese Academy of Sciences Beijing China

ISBN: (数字)9798350353006

ISBN: (纸本)9798350353013

We propose a novel strategy, ES 3 , for self-supervised learning of robust audio-visual speech representations from unlabeled talking face videos. While many recent approaches for this task primarily rely on guiding the learning process using the audio modality alone to capture information shared between audio and video, we reframe the problem as the acquisition of shared, unique (modality-specific) and synergistic speech information to address the inherent asymmetry between the modalities. Based on this formulation, we propose a novel “evolving” strategy that progressively builds joint audio-visual speech representations that are strong for both uni-modal (audio & visual) and bi-modal (audio-visual) speech. First, we leverage the more easily learnable audio modality to initialize audio and visual representations by capturing audio-unique and shared speech information. Next, we incorporate video-unique speech information and bootstrap the audio-visual repre-sentations on top of the previously acquired shared knowledge. Finally, we maximize the total audio-visual speech information, including synergistic information to obtain robust and comprehensive representations. We implement ES 3 as a simple Siamese framework and experiments on both English benchmarks and a newly contributed large-scale Mandarin dataset show its effectiveness. In particular, on LRS2-BBC, our smallest model is on par with SoTA models with only 1/2 parameters and 1/8 unlabeled data (223h).

关键词： Visualization Computer vision Self-supervised learning Manuals Benchmark testing Data models Pattern recognition

Multi-View Consistent 3D GAN Inversion via Bidirectional Encoder

学校读者我要写书评

暂无评论

Multi-View Consistent 3D GAN Inversion via Bidirectional Enc...

International Conference on Automatic Face and Gesture Recognition

作者： Haozhan Wu Hu Han Shiguang Shan Xilin Chen Key Laboratory of Intelligent Information Processing Chinese Academy of Sciences (CAS) Institute of Computing Technology CAS Beijing China University of the Chinese Academy of Sciences Beijing China

ISBN: (数字)9798350394948

ISBN: (纸本)9798350394955

3D GAN inversion enables not only 3D reconstruction from a 2D image, but also novel view synthesis and image editing. Existing works ensure the novel view synthesis quality by constraining the synthesized views to conform to the real image distribution. However, most of the methods did not consider the multi-view consistency, i.e., different photos of the same 3D scene via 3D GAN inversion can be inverted to the same 3D scene. In this paper, we propose a bidirectional encoder (BiDiE) for 3D GAN inversion that can improve the multi-view consistency and alleviate the interference of camera parameter prediction errors. On the one hand, the bidirectional encoder takes real images as input, estimates the camera parameters, and performs 3D reconstruction. On the other hand, the bidirectional encoder takes randomly sampled latent code and camera parameters as input, and generates synthesized images to assist in the latent code learning process. In addition, we extend the latent space from W+ to W++ to improve its reconstruction and editing capabilities. Experiments on the FFHQ, CelebA-HQ and Multi-PIE datasets prove that our proposed method outperforms state-of-the-art methods in multi-view consistent reconstruction as well as editing capability. 1 1 Code and datasets are available at https://***/WHZMM/BiDiE

关键词： Three-dimensional displays Codes Face recognition Interference Gesture recognition Cameras Image reconstruction

Instruct-NeuralTalker: Editing Audio-Driven Talking Radiance Fields with Instructions

学校读者我要写书评

暂无评论

arXiv 2023年

作者： Sun, Yuqi He, Ruian Tan, Weimin Yan, Bo School of Computer Science Shanghai Key Laboratory of Intelligent Information Processing Shanghai Collaborative Innovation Center of Intelligent Visual Computing Fudan University Shanghai China

Recent neural talking radiance field methods have shown great success in photorealistic audio-driven talking face synthesis. In this paper, we propose the first novel interactive framework that utilizes human instructions to edit talking radiance fields to achieve personalized talking face generation. Given a short speech video, we first build an efficient talking radiance field and then apply the latest conditional diffusion model to edit images with human instructions during the optimization. To ensure audio-lip synchronization during the editing process, we propose a progressive dataset updating strategy to ensure successful editing while preventing distortion of lip shapes. We also introduce a lightweight refinement network for complementing image details and achieving controllable detail generation in the final rendered image. Our method enables real-time rendering at up to 30FPS on consumer hardware. Multiple metrics and user verification show that our approach significantly improves rendering quality compared to state-of-the-art methods. © 2023, CC BY-NC-ND.

关键词： Rendering (computer graphics)

Multiscale Adaptive Multifractal Cross-Correlation Analysis of Multivariate Time Series

学校读者我要写书评

暂无评论

SSRN

SSRN 2023年

作者： Jiang, Huanwen Wang, Xinyao Han, Guosheng Key Laboratory of Intelligent Computing and Information Processing Ministry of Education Hunan Key Laboratory for Computation and Simulation in Science and Engineering Xiangtan University Hunan Xiangtan411105 China

In the real world, most of the time series generated from complex systems are nonlinear. To effectively study its fractal properties, in this work, we first generalize the adaptive fractal analysis (AFA) to the adaptive multifractal cross-correlation analysis (AMFCCA), which can be used to study the multifractal cross-correlation between two time series. Considering the complexity of time series from complex systems, we extend AMFCCA to the case of multivariate time series, namely multivariate adaptive multifractal cross-correlation analysis (MV-AMFCCA). In order to detect the multifractality on different time scales, we propose multiscale adaptive multifractal cross-correlation analysis of multivariate time series (MMV-AMFCCA). By the numerical simulation on synthetic multivariate processes, our method shows the theoretical validity. Furthermore, we applied these methods to the multifractal analysis of three pollutants in urban and suburban areas in Beijing. After removing the seasonal trend, we find that the urban and suburban systems both have multifractality, especially the multifractality of the urban system are more evident than the suburban systems, the degree of multifractality in spring and winter stronger than that in summer and autumn. Therefore, our methods are suitable for systems with multiple outputs and provide more comprehensive methods for characterizing autocorrelation and cross-correlation in multivariate time series. © 2023, The Authors. All rights reserved.

关键词： Time series

Automatic skin lesion classification using a novel densely connected convolutional network integrated with an attention module

学校读者我要写书评

暂无评论

Journal of Ambient Intelligence and Humanized computing 2023年第7期14卷 8943-8956页

作者： Shan, Pufang Chen, Jialei Fu, Chong Cao, Lin Tie, Ming Sham, Chiu-Wing School of Computer Science and Engineering Northeastern University Shenyang110819 China Engineering Research Center of Security Technology of Complex Network System Ministry of Education Shenyang China Key Laboratory of Intelligent Computing in Medical Image Ministry of Education Northeastern University Shenyang110819 China School of Information and Communication Engineering Beijing Information Science and Technology University Beijing100101 China Science and Technology on Space Physics Laboratory Beijing100076 China School of Computer Science The University of Auckland Auckland New Zealand

Automated skin lesion classification in dermoscopy images remains challenging due to the existence of artefacts and intrinsic cutaneous features, diversity of lesion morphology, insufficiency of training data, and class imbalance problem. To address these challenges, we propose a new densely connected convolutional network termed AttDenseNet-121, which is obtained by integrating the convolutional block attention module (CBAM) into DenseNet-121. CBAM is a simple yet efficient attention module, which is further improved by adding a novel pooling layer (multiscale-pooling) to effectively enhance its attention capability. The optimized CBAM strengthens the representation power of DenseNet-121 by emphasizing meaningful features and suppressing unnecessary ones, thus significantly enhancing the classification performance. Besides, to handle the imbalanced dataset, we employ an improved focal loss function rather than the traditional cross-entropy loss function to train AttDenseNet-121. The improved focal loss is calculated from a different perspective compared with the original focal loss. It makes the distribution of positive samples and negative samples in each batch the same as that in the original data, successfully mitigating the negative influence of class imbalance on this multi-class classification task. We conduct extensive experiments on the public benchmark dataset (HAM10000) and the results indicate that our proposed method achieves superior performance compared with that of the baselines and state-of-the-art algorithms. © 2022, The Author(s), under exclusive licence to Springer-Verlag GmbH Germany, part of Springer Nature.

关键词： Convolution

Fine-Grained Blind Face Inpainting with 3D Face Component Disentanglement

学校读者我要写书评

暂无评论

Fine-Grained Blind Face Inpainting with 3D Face Component Di...

International Conference on Acoustics, Speech, and Signal processing (ICASSP)

作者： Yu Bai Ruian He Weimin Tan Bo Yan Yangle Lin School of Computer Science Shanghai Key Laboratory of Intelligent Information Processing Shanghai Collaborative Innovation Center of Intelligent Visual Computing Fudan University Shanghai China

Inpainting is a task to restore occlusion or other corruption on images. However, previous works require mask of the occluded area to restore the occluded image, which is inconvenient for application. Blind face inpainting aims to automatically restore the occluded face without position information of the corrupt region. In this paper, we propose a novel fine-grained blind face inpainting framework, combining 3D face components disentanglement with generative network. Canonical face texture and shape disentangled by unsupervised 3D face model is restored separately to get occlusion-free rendered result. Finally, the pixel-to-pixel generative module utilize the occlusion image and the coarse de-occlusion face to get refined inpainted result. We also build up a new dataset called CelebO-3D which consists of occluded face images synthesized with 3D occlusion and rendered by 3D face model. Extensive experiments show that the proposed method is effective and robust in face blind inpainting both in synthesized and real images. Extensive evaluations and comparison with previous methods also show our superior effectiveness and lightweight architecture.

关键词： Solid modeling Three-dimensional displays Shape Speech enhancement Signal processing Acoustics Image restoration

MVFlow: Deep Optical Flow Estimation of Compressed Videos with Motion Vector Prior

学校读者我要写书评

暂无评论

arXiv 2023年

作者： Zhou, Shili Jiang, Xuhao Tan, Weimin He, Ruian Yan, Bo School of Computer Science Shanghai Key Laboratory of Intelligent Information Processing Shanghai Collaborative Innovation Center of Intelligent Visual Computing Fudan University Shanghai China

In recent years, many deep learning-based methods have been proposed to tackle the problem of optical flow estimation and achieved promising results. However, they hardly consider that most videos are compressed and thus ignore the pre-computed information in compressed video streams. Motion vectors, one of the compression information, record the motion of the video frames. They can be directly extracted from the compression code stream without computational cost and serve as a solid prior for optical flow estimation. Therefore, we propose an optical flow model, MVFlow, which uses motion vectors to improve the speed and accuracy of optical flow estimation for compressed videos. In detail, MVFlow includes a key Motion-Vector Converting Module, which ensures that the motion vectors can be transformed into the same domain of optical flow and then be utilized fully by the flow estimation module. Meanwhile, we construct four optical flow datasets for compressed videos containing frames and motion vectors in pairs. The experimental results demonstrate the superiority of our proposed MVFlow, which can reduce the AEPE by 1.09 compared to existing models or save 52% time to achieve similar accuracy to existing models. Copyright © 2023, The Authors. All rights reserved.

关键词： Optical flows

Uncertainty-Guided Spatial Pruning Architecture for Efficient Frame Interpolation

学校读者我要写书评

暂无评论

arXiv 2023年

作者： Cheng, Ri Jiang, Xuhao He, Ruian Zhou, Shili Tan, Weimin Yan, Bo School of Computer Science Shanghai Key Laboratory of Intelligent Information Processing Shanghai Collaborative Innovation Center of Intelligent Visual Computing Fudan University Shanghai China

The video frame interpolation (VFI) model applies the convolution operation to all locations, leading to redundant computations in regions with easy motion. We can use dynamic spatial pruning method to skip redundant computation, but this method cannot properly identify easy regions in VFI tasks without supervision. In this paper, we develop an Uncertainty-Guided Spatial Pruning (UGSP) architecture to skip redundant computation for efficient frame interpolation dynamically. Specifically, pixels with low uncertainty indicate easy regions, where the calculation can be reduced without bringing undesirable visual results. Therefore, we utilize uncertainty-generated mask labels to guide our UGSP in properly locating the easy region. Furthermore, we propose a self-contrast training strategy that leverages an auxiliary non-pruning branch to improve the performance of our UGSP. Extensive experiments show that UGSP maintains performance but reduces FLOPs by 34%/52%/30% compared to baseline without pruning on Vimeo90K/UCF101/MiddleBury datasets. In addition, our method achieves state-of-the-art performance with lower FLOPs on multiple benchmarks. Copyright © 2023, The Authors. All rights reserved.

关键词： Benchmarking

Towards Robust Semantic Segmentation against Patch-based Attack via Attention Refinement

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Yuan, Zheng Zhang, Jie Wang, Yude Shan, Shiguang Chen, Xilin Key Laboratory of Intelligent Information Processing Institute of Computing Technology Chinese Academy of Sciences Beijing100190 China University of Chinese Academy of Sciences Beijing100049 China

The attention mechanism has been proven effective on various visual tasks in recent years. In the semantic segmentation task, the attention mechanism is applied in various methods, including the case of both Convolution Neural networks (CNN) and Vision Transformer (ViT) as backbones. However, we observe that the attention mechanism is vulnerable to patch-based adversarial attacks. Through the analysis of the effective receptive field, we attribute it to the fact that the wide receptive field brought by global attention may lead to the spread of the adversarial patch. To address this issue, in this paper, we propose a Robust Attention Mechanism (RAM) to improve the robustness of the semantic segmentation model, which can notably relieve the vulnerability against patch-based attacks. Compared to the vallina attention mechanism, RAM introduces two novel modules called Max Attention Suppression and Random Attention Dropout, both of which aim to refine the attention matrix and limit the influence of a single adversarial patch on the semantic segmentation results of other positions. Extensive experiments demonstrate the effectiveness of our RAM to improve the robustness of semantic segmentation models against various patch-based attack methods under different attack settings. Copyright © 2024, The Authors. All rights reserved.

关键词： Semantics