检索结果-内蒙古大学图书馆

Audio-Driven High Definetion and Lip-Synchronized Talking Face Generation Based on Face Reenactment

学校读者我要写书评

暂无评论

Audio-Driven High Definetion and Lip-Synchronized Talking Fa...

International Conference on Acoustics, Speech, and Signal Processing (ICASSP)

作者： Xianyu Wang Yuhan Zhang Weihua He Yaoyuan Wang Minglei Li Yuchen Wang Jingyi Zhang Shunbo Zhou Ziyang Zhang Advanced Computing and Storage Lab Huawei Technologies Co. Ltd. China Language & Speech Innovation Lab Huawei Technologies Co. Ltd. China NCE Dept. Huawei Technologies Co. Ltd. China Ascend Lab Huawei Technologies Co. Ltd. China Edge Cloud Innovation Lab Huawei Technologies Co. Ltd. China

Generating audio-driven photo-realistic talking face has received intensive attention due to its ability to bring more new human-computer interaction experiences. However, previous works struggled to balance high definition, lip synchronization, and low customization costs, which would degrade the user experience. In this paper, a novel audio-driven talking face generation method was proposed, which subtly converts the problem of improving video definition into the problem of face reenactment to produce both lip-synchronized and high- definition face video. The framework is decoupled, meaning that the same trained model can be used on arbitrary characters and audio without further customizing training for specific people, thus significantly reducing costs. Experiment results show that our proposed method achieves the high video definition, and comparable lip synchronization performance with the existing state-of-the-art methods.

关键词： Training Image quality Human computer interaction Costs Lips Signal processing User experience

Inherent Redundancy in Spiking Neural Networks

学校读者我要写书评

暂无评论

Inherent Redundancy in Spiking Neural Networks

International Conference on Computer Vision (ICCV)

作者： Man Yao Jiakui Hu Guangshe Zhao Yaoyuan Wang Ziyang Zhang Bo Xu Guoqi Li School of Automation Science and Engineering Xi’an Jiaotong University Xi’an China Institute of Automation Chinese Academy of Sciences Beijing China Peng Cheng Laboratory Shenzhen China Peking University Health Science Center Peking University Beijing China Advanced Computing and Storage Lab Huawei Technologies Co. Ltd.

Spiking Neural Networks (SNNs) are well known as a promising energy-efficient alternative to conventional artificial neural networks. Subject to the preconceived impression that SNNs are sparse firing, the analysis and optimization of inherent redundancy in SNNs have been largely overlooked, thus the potential advantages of spike-based neuromorphic computing in accuracy and energy efficiency are interfered. In this work, we pose and focus on three key questions regarding the inherent redundancy in SNNs. We argue that the redundancy is induced by the spatio-temporal invariance of SNNs, which enhances the efficiency of parameter utilization but also invites lots of noise spikes. Further, we analyze the effect of spatio-temporal invariance on the spatio-temporal dynamics and spike firing of SNNs. Then, motivated by these analyses, we propose an Advance Spatial Attention (ASA) module to harness SNNs’ redundancy, which can adaptively optimize their membrane potential distribution by a pair of individual spatial attention sub-modules. In this way, noise spike features are accurately regulated. Experimental results demonstrate that the proposed method can significantly drop the spike firing with better performance than state-of-the-art SNN baselines. Our code is available in https://***/BIClab/ASA-SNN.

关键词：

Statistical Physics of Deep Neural Networks: Initialization toward Optimal Channels

学校读者我要写书评

暂无评论

arXiv 2022年

作者： Weng, Kangyu Cheng, Aohua Zhang, Ziyang Sun, Pei Tian, Yang Tsien Excellence in Engineering Program Xingjian College Tsinghua University Beijing100084 China Laboratory of Advanced Computing and Storage Central Research Institute 2012 Laboratories Huawei Technologies Co. Ltd. Beijing100084 China Department of Psychology & Tsinghua Brain and Intelligence Lab Tsinghua University Beijing100084 China

In deep learning, neural networks serve as noisy channels between input data and its representation. This perspective naturally relates deep learning with the pursuit of constructing channels with optimal performance in information transmission and representation. While considerable efforts are concentrated on realizing optimal channel properties during network optimization, we study a frequently overlooked possibility that neural networks can be initialized toward optimal channels. Our theory, consistent with experimental validation, identifies primary mechanics underlying this unknown possibility and suggests intrinsic connections between statistical physics and deep learning. Unlike the conventional theories that characterize neural networks applying the classic mean-filed approximation, we offer analytic proof that this extensively applied simplification scheme is not valid in studying neural networks as information channels. To fill this gap, we develop a corrected mean-field framework applicable for characterizing the limiting behaviors of information propagation in neural networks without strong assumptions on inputs. Based on it, we propose an analytic theory to prove that mutual information maximization is realized between inputs and propagated signals when neural networks are initialized at dynamic isometry, a case where information transmits via norm-preserving mappings. These theoretical predictions are validated by experiments on real neural networks, suggesting the robustness of our theory against finite-size effects. Finally, we analyze our findings with information bottleneck theory to confirm the precise relations among dynamic isometry, mutual information maximization, and optimal channel properties in deep learning. Our work may lay a cornerstone for promoting deep learning in terms of network initialization and suggest general statistical physics mechanisms underlying diverse deep learning *** Codes 82C44, 82C05 © 2022, CC BY.

关键词： Statistical Physics

MSS-DepthNet: Depth Prediction with Multi-Step Spiking Neural Network

学校读者我要写书评

暂无评论

arXiv 2022年

作者： Wu, Xiaoshan He, Weihua Yao, Man Zhang, Ziyang Wang, Yaoyuan Li, Guoqi ZJU-UIUC Institute Zhejiang University Zhejiang Haining China Department of Precision Instrument Tsinghua University Beijing China School of Automation Science and Engineering Xi'an Jiaotong University Shaanxi Xi'An China Advanced Computing and Storage Lab Huawei Technologies Co. Ltd Beijing China Institute of Automation Chinese Academy of Sciences Beijing China

Event cameras are considered to have great potential for computer vision and robotics applications because of their high temporal resolution and low power consumption characteristics. However, the event streams output from event cameras have asynchronous, sparse characteristics that cannot be handled by existing computer vision algorithms. Spiking neural network is a novel event-based computational paradigm that is considered to be well suited for processing event camera tasks. However, direct training of deep SNNs suffers from degradation problems. This work addresses these problems by proposing a spiking nerual network architecture with novel residual block designed and multi-dimension attention modules combined, focusing on the problem of depth prediction. In addition, a novel event stream representation method is proposed specifically for SNNs. This model outperforms previous ANN networks of the same size on the MVSEC dataset and shows great computational efficiency. Copyright © 2022, The Authors. All rights reserved.

关键词： Neural networks

Accurate and Efficient Event-based Semantic Segmentation Using Adaptive Spiking Encoder-Decoder Network

学校读者我要写书评

暂无评论

arXiv 2023年

作者： Zhang, Rui Leng, Luziwei Che, Kaiwei Zhang, Hu Cheng, Jie Guo, Qinghai Liao, Jianxing Cheng, Ran Department of Computer Science and Engineering Southern University of Science and Technology Shenzhen518055 China Department of Electrical and Electronic Engineering Southern University of Science and Technology Shenzhen518055 China ACSLab Huawei Technologies Co. Ltd China Advanced Computing and Storage Lab Huawei Technologies Co. Ltd. Shenzhen518055 China

Spiking neural networks (SNNs), known for their low-power, event-driven computation and intrinsic temporal dynamics, are emerging as promising solutions for processing dynamic, asynchronous signals from event-based sensors. Despite their potential, SNNs face challenges in training and architectural design, resulting in limited performance in challenging eventbased dense prediction tasks compared to artificial neural networks (ANNs). In this work, we develop an efficient spiking encoder-decoder network (SpikingEDN) for large-scale eventbased semantic segmentation tasks. To enhance the learning efficiency from dynamic event streams, we harness the adaptive threshold which improves network accuracy, sparsity and robustness in streaming inference. Moreover, we develop a dualpath Spiking Spatially-Adaptive Modulation module, which is specifically tailored to enhance the representation of sparse events and multi-modal inputs, thereby considerably improving network performance. Our SpikingEDN attains a mean intersection over union (MIoU) of 72.57% on the DDD17 dataset and 58.32% on the larger DSEC-Semantic dataset, showing competitive results to the state-of-the-art ANNs while requiring substantially fewer computational resources. Our results shed light on the untapped potential of SNNs in event-based vision applications. The source code will be made publicly available. Copyright © 2023, The Authors. All rights reserved.

关键词： Semantic Segmentation

Deep Directly-Trained Spiking Neural Networks for Object Detection

学校读者我要写书评

暂无评论

Deep Directly-Trained Spiking Neural Networks for Object Det...

International Conference on Computer Vision (ICCV)

作者： Qiaoyi Su Yuhong Chou Yifan Hu Jianing Li Shijie Mei Ziyang Zhang Guoqi Li School of Artificial Intelligence University of Chinese Academy of Sciences College of Artificial Intelligence Xi’an Jiaotong University Institute of Automation Chinese Academy of Sciences Department of Precision Instrument Tsinghua University School of Computer Science Peking University School of Vehicle and Mobility Tsinghua University Advanced Computing and Storage Lab Huawei Technologies Co. Ltd.

Spiking neural networks (SNNs) are brain-inspired energy-efficient models that encode information in spatiotemporal dynamics. Recently, deep SNNs trained directly have shown great success in achieving high performance on classification tasks with very few time steps. However, how to design a directly-trained SNN for the regression task of object detection still remains a challenging problem. To address this problem, we propose EMS-YOLO, a novel directly-trained SNN framework for object detection, which is the first trial to train a deep SNN with surrogate gradients for object detection rather than ANN-SNN conversion strategies. Specifically, we design a full-spike residual block, EMS-ResNet, which can effectively extend the depth of the directly-trained SNN with low power consumption. Furthermore, we theoretically analyze and prove the EMS-ResNet could avoid gradient vanishing or exploding. The results demonstrate that our approach outperforms the state-of-the-art ANN-SNN conversion methods (at least 500 time steps) in extremely fewer time steps (only 4 time steps). It is shown that our model could achieve comparable performance to the ANN with the same architecture while consuming 5.83× less energy on the frame-based COCO Dataset and the event-based Gen1 Dataset. Our code is available in https://***/BIClab/EMS-YOLO.

关键词：

Test-Time Training-Free Domain Adaptation

学校读者我要写书评

暂无评论

Test-Time Training-Free Domain Adaptation

International Conference on Acoustics, Speech, and Signal Processing (ICASSP)

作者： Yongxiang Feng Weihua He Kaichao You Bing Liu Ziyang Zhang Yaoyuan Wang Minglei Li Yihang Lou Jiawei Li Guoqi Li Jianxing Liao Advanced Computing and Storage Lab Huawei Technologies Co. Ltd. School of Electronics Engineering and Computer Science Peking University China Language & Speech Innovation Lab Huawei Technologies Co. Ltd. GoTen AI Lab Department of Intelligent Vision Huawei Technologies Co. Ltd. China Department of Production Automation Development Huawei Technologies Co. Ltd. China Institute of Automation Chinese Academy of Sciences China

Deploying deep learning models to new environments is very challenging. Domain adaptation (DA) is a promising paradigm to solve the problem by collecting and adapting to unlabeled data in new environments. Though research efforts have led to steady performance improvement over the past decade, DA algorithms are still hard to deploy, as training on unlabeled new data makes tuning difficult and not feasible for inference-only devices. To make DA practical, in this paper we study a new problem named Test-time Training-Free Domain Adaptation (TTDA), where trained models must adapt to a single input (mimicking the test-time scenario) without training. By exploiting spatial activation that was previously overlooked and simply averaged out, we propose a simple method based on Feature Statistics Transformation (FST) on-the-fly for each test example. The proposed algorithm is tested in the TTDA setting on two standard DA benchmarks. Surprisingly, it surpasses or performs on par with state-of-the-art DA methods, even though they require additional training. We envision that this training-free paradigm has the potential to bring DA to embedded devices and would be of interest to audience of community.

关键词： Training Performance evaluation Adaptation models Transfer learning Signal processing algorithms Statistical distributions Classification algorithms

Deep Directly-Trained Spiking Neural Networks for Object Detection

学校读者我要写书评

暂无评论

arXiv 2023年

作者： Su, Qiaoyi Chou, Yuhong Hu, Yifan Li, Jianing Mei, Shijie Zhang, Ziyang Li, Guoqi School of Artificial Intelligence University of Chinese Academy of Sciences China College of Artificial Intelligence Xi'an Jiaotong University China Department of Precision Instrument Tsinghua University China School of Computer Science Peking University China School of Vehicle and Mobility Tsinghua University China Advanced Computing and Storage Lab Huawei Technologies Co Ltd. China Institute of Automation Chinese Academy of Sciences China

关键词： Object detection

A unified theory of information transfer and causal relation

学校读者我要写书评

暂无评论

arXiv 2022年

作者： Tian, Yang Hou, Hedong Wang, Yaoyuan Zhang, Ziyang Sun, Pei Department of Psychology Tsinghua Laboratory of Brain and Intelligence Tsinghua University Beijing100084 China UFR de Mathematiques Universite de Paris Paris75013 France Department of Psychology Tsinghua Brain and Intelligence Lab Tsinghua University Beijing100084 China Laboratory of Advanced Computing and Storage Central Research Institute 2012 Laboratories Huawei Technologies Co. Ltd. Beijing100084 China

Information transfer between coupled stochastic dynamics, measured by transfer entropy and information flow, is suggested as a physical process underlying the causal relation of systems. While information transfer analysis has booming applications in both science and engineering fields, critical mysteries about its foundations remain unsolved. Fundamental yet difficult questions concern how information transfer and causal relation originate, what they depend on, how they differ from each other, and if they are created by a unified and general quantity. These questions essentially determine the validity of causal relation measurement via information transfer. Here we pursue to lay a complete theoretical basis of information transfer and causal relation. Beyond the well-known relations between these concepts that conditionally hold, we demonstrate that information transfer and causal relation universally originate from specific information synergy and redundancy phenomena characterized by high-order mutual information. More importantly, our theory analytically explains the mechanisms for information transfer and causal relation to originate, vanish, and differ from each other. Moreover, our theory naturally defines the effect sizes of information transfer and causal relation based on high-dimensional coupling events. These results may provide a unified view of information, synergy, and causal relation to bridge Pearl's causal inference theory in computer science and information transfer analysis in physics. © 2022, CC BY.

关键词： Stochastic systems