检索结果-内蒙古大学图书馆

Training noisy single-channel speech separation with noisy oracle sources: A large gap and a small step

学校读者我要写书评

暂无评论

arXiv 2020年

作者： Maciejewski, Matthew Shi, Jing Watanabe, Shinji Khudanpur, Sanjeev Center for Language and Speech Processing The Johns Hopkins University United States Human Language Technology Center of Excellence The Johns Hopkins University United States Institute of Automation Chinese Academy of Sciences China

As the performance of single-channel speech separation systems has improved, there has been a desire to move to more challenging conditions than the clean, near-field speech that initial systems were developed on. When training deep learning separation models, a need for ground truth leads to training on synthetic mixtures. As such, training in noisy conditions requires either using noise synthetically added to clean speech, preventing the use of in-domain data for a noisy-condition task, or training using mixtures of noisy speech, requiring the network to additionally separate the noise. We demonstrate the relative inseparability of noise and that this noisy speech paradigm leads to significant degradation of system performance. We also propose an SI-SDR–inspired training objective that tries to exploit the inseparability of noise to implicitly partition the signal and discount noise separation errors, enabling the training of better separation systems with noisy oracle sources. Copyright © 2020, The Authors. All rights reserved.

关键词： speech analysis

Influence of emotional states on incongruity-resolution and nonsense humour processing

学校读者我要写书评

暂无评论

International journal of psychology : Journal international de psychologie 2024年第6期59卷 1091-1100页

作者： Roy Soh Ching-Lin Wu Ping-Hsun Tsai Hsueh-Chih Chen School of Education and Human Sciences Albukhary International University Alor Setar Malaysia. Program of Learning Sciences National Taiwan Normal University Taipei Taiwan. Institute for Research Excellence in Learning Sciences National Taiwan Normal University Taipei Taiwan. Department of Educational Psychology and Counseling National Taiwan Normal University Taipei Taiwan. Chinese Language and Technology Center National Taiwan Normal University Taipei Taiwan.

Humour processing is complex and may be influenced by an individual's emotional state. This study explores the differences in incongruity-resolution and nonsense joke humour processing between three groups with different emotional states (positive, neutral and negative). Each group consisted of 32 participants. The positive and negative emotion groups were asked to elicit positive and negative emotions, respectively, through autobiographical recall. In contrast, the neutral emotion group was asked to answer questions unrelated to emotions. Participants were also asked to rate their emotional valence, arousal and conformity with the provided emotion words. Subsequently, they were directed to rate their humour appreciation, humour comprehension and level of surprise at the jokes. The results indicate that these variables interacted with the level of amusement derived from the two humour types. The positive emotion group perceived nonsense jokes as funnier than the negative emotion group, while the negative emotion group perceived nonsense jokes as less amusing than incongruity-resolution jokes. No significant difference was found between the three emotion groups in terms of humour comprehension and surprise at the jokes. The results deepen the understanding of the correlation between emotional states and processing of different humour types.

关键词： Emotional state Humour appreciation Humour comprehension Incongruity‐resolution humour Nonsense humour

WIKIVIDEO: Article Generation from Multiple Videos

学校读者我要写书评

暂无评论

arXiv 2025年

作者： Martin, Alexander Kriz, Reno Walden, William Gantt Sanders, Kate Recknor, Hannah Yang, Eugene Ferraro, Francis Van Durme, Benjamin Johns Hopkins University United States Human Language Technology Center of Excellence United States University of Maryland Baltimore County United States

We present the challenging task of automatically creating a high-level Wikipedia-style article that aggregates information from multiple diverse videos about real-world events, such as natural disasters or political elections. Videos are intuitive sources for retrieval-augmented generation (RAG), but most contemporary RAG workflows focus heavily on text while existing methods for video-based summarization focus on low-level scene understanding rather than high-level event semantics. To close this gap, we introduce WIKIVIDEO, a benchmark consisting of expert-written articles and densely annotated videos that provide evidence for articles’ claims, facilitating the integration of video into RAG pipelines and enabling the creation of in-depth content that is grounded in multimodal sources. We further propose Collaborative Article Generation (CAG), a novel interactive method for article creation from multiple videos. CAG leverages an iterative interaction between an r1-style reasoning model and a VideoLLM to draw higher-level inferences about the target event than is possible with VideoLLMs alone, which fixate on low-level visual features. We benchmark state-of-the-art VideoLLMs and CAG in both oracle retrieval and RAG settings and find that CAG consistently outperforms alternative methods, while suggesting intriguing avenues for future work. © 2025, CC BY-SA.

关键词：

Dover-lap: A method for combining overlap-aware diarization outputs

学校读者我要写书评

暂无评论

arXiv 2020年

作者： Raj, Desh Garcia-Perera, Leibny Paola Huang, Zili Watanabe, Shinji Povey, Daniel Stolcke, Andreas Khudanpur, Sanjeev Center for Language and Speech Processing & Human Language Technlogy Center of Excellence The Johms Hopkins University BaltimoreMD21218 United States Human Language Technology Center of Excellence Johns Hopkins University BaltimoreMD21218 United States Xiaomi Corp. Beijing China Amazon Alexa Speech SunnyvaleCA United States

Several advances have been made recently towards handling overlapping speech for speaker diarization. Since speech and natural language tasks often benefit from ensemble techniques, we propose an algorithm for combining outputs from such diarization systems through majority voting. Our method, DOVER-Lap, is inspired from the recently proposed DOVER algorithm, but is designed to handle overlapping segments in diarization outputs. We also modify the pair-wise incremental label mapping strategy used in DOVER, and propose an approximation algorithm based on weighted k-partite graph matching, which performs this mapping using a global cost tensor. We demonstrate the strength of our method by combining outputs from diverse systems — clustering-based, region proposal networks, and target-speaker voice activity detection — on AMI and LibriCSS datasets, where it consistently outperforms the single best system. Additionally, we show that DOVER-Lap can be used for late fusion in multichannel diarization, and compares favorably with early fusion methods like beamforming. © 2020, CC-BY.

关键词： Approximation algorithms

BISECT: Learning to split and rephrase sentences with bitexts

学校读者我要写书评

暂无评论

arXiv 2021年

作者： Kim, Joongwon Maddela, Mounica Kriz, Reno Xu, Wei Callison-Burch, Chris Department of Computer and Information Science University of Pennsylvania School of Interactive Computing Georgia Institute of Technology Human Language Technology Center of Excellence Johns Hopkins University

An important task in NLP applications such as sentence simplification is the ability to take a long, complex sentence and split it into shorter sentences, rephrasing as necessary. We introduce a novel dataset and a new model for this 'split and rephrase' task. Our BISECT training data consists of 1 million long English sentences paired with shorter, meaning-equivalent English sentences. We obtain these by extracting 1-2 sentence alignments in bilingual parallel corpora and then using machine translation to convert both sides of the corpus into the same language. BISECT contains higher quality training examples than previous Split and Rephrase corpora, with sentence splits that require more significant modifications. We categorize examples in our corpus, and use these categories in a novel model that allows us to target specific regions of the input sentence to be split and edited. Moreover, we show that models trained on BISECT can perform a wider variety of split operations and improve upon previous state-of-the-art approaches in automatic and human evaluations. © 2021, CC BY.

关键词：

SPECTRE: Visual speech-Informed Perceptual 3D Facial Expression Reconstruction from Videos

学校读者我要写书评

暂无评论

SPECTRE: Visual Speech-Informed Perceptual 3D Facial Express...

IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)

作者： Panagiotis P. Filntisis George Retsinas Foivos Paraperas-Papantoniou Athanasios Katsamanis Anastasios Roussos Petros Maragos Institute of Robotics Athena Research Center Maroussi Greece School of Electrical & Computer Engineering National Technical University of Athens Greece Imperial College London UK Institute for Language and Speech Processing Athena R.C. Greece Institute of Computer Science (ICS) Foundation for Research & Technology - Hellas (FORTH) Greece College of Engineering Mathematics and Physical Sciences University of Exeter UK

The recent state of the art on monocular 3D face reconstruction from image data has made some impressive advancements, thanks to the advent of Deep Learning. However, it has mostly focused on input coming from a single RGB image, overlooking the following important factors: a) Nowadays, the vast majority of facial image data of interest do not originate from single images but rather from videos, which contain rich dynamic information. b) Furthermore, these videos typically capture individuals in some form of verbal communication (public talks, teleconferences, audiovisual human-computer interactions, interviews, monologues/dialogues in movies, etc). When existing 3D face reconstruction methods are applied in such videos, the artifacts in the reconstruction of the shape and motion of the mouth area are often severe, since they do not match well with the speech *** overcome the aforementioned limitations, we present the first method for visual speech-informed perceptual reconstruction of 3D mouth expressions. We do this by proposing a "lipreading" loss, which guides the fitting process so that the elicited perception from the 3D reconstructed talking head resembles that of the original video footage. We demonstrate that, interestingly, the lipreading loss is better suited for 3D reconstruction of mouth movements compared to traditional landmark losses, and even direct 3D supervision. Furthermore, the devised method does not rely on any text transcriptions or corresponding audio, rendering it ideal for training in unlabeled datasets. We verify the efficiency of our method through objective evaluations on three large-scale datasets, as well as subjective evaluation with two web-based user studies. Project webpage: https://***/spectre/

关键词：

PYCHAIN: A fully parallelized pytorch implementation of LF-MMI for end-to-end ASR

学校读者我要写书评

暂无评论

arXiv 2020年

作者： Shao, Yiwen Wang, Yiming Povey, Daniel Khudanpur, Sanjeev Center for Language and Speech Processing Johns Hopkins University BaltimoreMD United States Human Language Technology Center of Excellence Johns Hopkins University BaltimoreMD United States Xiaomi Inc. Beijing China

We present PYCHAIN, a fully parallelized PyTorch implementation of end-to-end lattice-free maximum mutual information (LF-MMI) training for the so-called chain models in the Kaldi automatic speech recognition (ASR) toolkit. Unlike other PyTorch and Kaldi based ASR toolkits, PYCHAIN is designed to be as flexible and light-weight as possible so that it can be easily plugged into new ASR projects, or other existing PyTorch-based ASR tools, as exemplified respectively by a new project PYCHAIN-EXAMPLE, and ESPRESSO, an existing end-to-end ASR toolkit. PYCHAIN's efficiency and flexibility is demonstrated through such novel features as full GPU training on numerator/denominator graphs, and support for unequal length sequences. Experiments on the WSJ dataset show that with simple neural networks and commonly used machine learning techniques, PYCHAIN can achieve competitive results that are comparable to Kaldi and better than other end-to-end ASR systems. Copyright © 2020, The Authors. All rights reserved.

关键词： speech recognition

Study of pre-processing defenses against adversarial attacks on state-of-the-art speaker recognition systems

学校读者我要写书评

暂无评论

arXiv 2021年

作者： Joshi, Sonal Villalba, Jesús Zelasko, Piotr Moro-Velázquez, Laureano Dehak, Najim Johns Hopkins University BaltimoreMD21218 United States The Human Language Technology Center of Excellence Johns Hopkins University BaltimoreMD21218 United States

Adversarial examples to speaker recognition (SR) systems are generated by adding a carefully crafted noise to the speech signal to make the system fail while being imperceptible to humans. Such attacks pose severe security risks, making it vital to deep-dive and understand how much the state-of-the-art SR systems are vulnerable to these attacks. Moreover, it is of greater importance to propose defenses that can protect the systems against these attacks. Addressing these concerns, this paper at first investigates how state-of-the-art x-vector based SR systems are affected by white-box adversarial attacks, i.e., when the adversary has full knowledge of the system. x-Vector based SR systems are evaluated against white-box adversarial attacks common in the literature like fast gradient sign method (FGSM), basic iterative method (BIM)-a.k.a. iterative-FGSM-, projected gradient descent (PGD), and Carlini-Wagner (CW) attack. To mitigate against these attacks, the paper proposes four pre-processing defenses. It evaluates them against powerful adaptive white-box adversarial attacks, i.e., when the adversary has full knowledge of the system, including the defense. The four pre-processing defenses-viz. randomized smoothing, DefenseGAN, variational autoencoder (VAE), and Parallel WaveGAN vocoder (PWG) are compared against the baseline defense of adversarial training. Conclusions indicate that SR systems were extremely vulnerable under BIM, PGD, and CW attacks. Among the proposed pre-processing defenses, PWG combined with randomized smoothing offers the most protection against the attacks, with accuracy averaging 93% compared to 52% in the undefended system and an absolute improvement > 90% for BIM attacks with L∞ > 0.001 and CW attack. Copyright © 2021, The Authors. All rights reserved.

关键词： Gradient methods

A call for prudent choice of subword merge operations in neural machine translation

学校读者我要写书评

暂无评论

arXiv 2019年

作者： Ding, Shuoyang Renduchintala, Adithya Duh, Kevin Center for Language and Speech Processing Human Language Technology Center of Excellence Johns Hopkins University

Most neural machine translation systems are built upon subword units extracted by methods such as Byte-Pair Encoding (BPE) or wordpiece. However, the choice of number of merge operations is generally made by following existing recipes. In this paper, we conduct a systematic exploration on different numbers of BPE merge operations to understand how it interacts with the model architecture, the strategy to build vocabularies and the language pair. Our exploration could provide guidance for selecting proper BPE configurations in the future. Most prominently: we show that for LSTM-based architectures, it is necessary to experiment with a wide range of different BPE operations as there is no typical optimal BPE configuration, whereas for Transformer architectures, smaller BPE size tends to be a typically optimal choice. We urge the community to make prudent choices with subword merge operations, as our experiments indicate that a sub-optimal BPE configuration alone could easily reduce the system performance by 3–4 BLEU points. Copyright © 2019, The Authors. All rights reserved.

关键词： Computational linguistics