CochleaNet: A robust language-independent audio-visual model for speech enhancement

作     者：Gogate, Mandar Dashtipour, Kia Adeel, Ahsan Hussain, Amir 

作者机构：Edinburgh Napier University School of Computing EdinburghEH10 5DT United Kingdom University of Stirling Division of Computing Science and Maths StirlingFK9 4LA United Kingdom University of Wolverhampton School of Mathematics and Computer Science Wolverhampton United Kingdom 

出 版 物：《arXiv》 (arXiv)

年 卷 期：2019年

核心收录：

主　　题：Audition 

摘      要：Noisy situations cause huge problems for suffers of hearing loss as hearing aids often make the signal more audible but do not always restore the intelligibility. In noisy settings, humans routinely exploit the audio-visual (AV) nature of the speech to selectively suppress the background noise and to focus on the target speaker. In this paper, we present a causal, language, noise and speaker independent AV deep neural network (DNN) architecture for speech enhancement (SE). The model exploits the noisy acoustic cues and noise robust visual cues to focus on the desired speaker and improve the speech intelligibility. To evaluate the proposed SE framework a first of its kind AV binaural speech corpus, called ASPIRE, is recorded in real noisy environments including cafeteria and restaurant. We demonstrate superior performance of our approach in terms of objective measures and subjective listening tests over the state-of-the-art SE approaches as well as recent DNN based SE models. In addition, our work challenges a popular belief that a scarcity of multi-language large vocabulary AV corpus and wide variety of noises is a major bottleneck to build a robust language, speaker and noise independent SE systems. We show that a model trained on synthetic mixture of Grid corpus (with 33 speakers and a small English vocabulary) and ChiME 3 Noises (consisting of only bus, pedestrian, cafeteria, and street noises) generalise well not only on large vocabulary corpora but also on completely unrelated languages (such as Mandarin), wide variety of speakers and noises. Copyright © 2019, The Authors. All rights reserved.