发帖
雷达卡
摘要翻译：
我们提出了一个联合视听模型，用于从其他说话人和背景噪声等混合声音中分离单个语音信号。仅使用音频作为输入来解决这个任务是极具挑战性的，并且不能提供分离的语音信号与视频中的扬声器的关联。在本文中，我们提出了一个基于深度网络的模型，结合视觉和听觉信号来解决这个问题。视觉特征用于将音频“聚焦”在场景中所需的扬声器上，并提高语音分离质量。为了训练我们的联合视听模型，我们引入了AVSpeech,这是一个新的数据集，由数千小时的网络视频片段组成。我们演示了我们的方法对经典语音分离任务的适用性，以及包括激烈的采访、嘈杂的酒吧和尖叫的儿童在内的真实世界场景，只需要用户指定视频中他们想要分离其语音的人的脸。在混合语音的情况下，我们的方法比现有的纯音频语音分离方法有明显的优势。此外，我们的模型是与说话人无关的（只训练一次，适用于任何说话人），比最近的与说话人相关的视听语音分离方法（需要为每个感兴趣的说话人训练一个单独的模型）产生更好的结果。
---
英文标题：
《Looking to Listen at the Cocktail Party: A Speaker-Independent
Audio-Visual Model for Speech Separation》
---
作者：
Ariel Ephrat, Inbar Mosseri, Oran Lang, Tali Dekel, Kevin Wilson,
Avinatan Hassidim, William T. Freeman and Michael Rubinstein
---
最新提交年份：
2018
---
分类信息：
一级分类：Computer Science 计算机科学
二级分类：Sound 声音
分类描述：Covers all aspects of computing with sound, and sound as an information channel. Includes models of sound, analysis and synthesis, audio user interfaces, sonification of data, computer music, and sound signal processing. Includes ACM Subject Class H.5.5, and intersects with H.1.2, H.5.1, H.5.2, I.2.7, I.5.4, I.6.3, J.5, K.4.2.
涵盖了声音计算的各个方面，以及声音作为一种信息通道。包括声音模型、分析和合成、音频用户界面、数据的可听化、计算机音乐和声音信号处理。包括ACM学科类H.5.5，并与H.1.2、H.5.1、H.5.2、I.2.7、I.5.4、I.6.3、J.5、K.4.2交叉。
--
一级分类：Computer Science 计算机科学
二级分类：Computer Vision and Pattern Recognition 计算机视觉与模式识别
分类描述：Covers image processing, computer vision, pattern recognition, and scene understanding. Roughly includes material in ACM Subject Classes I.2.10, I.4, and I.5.
涵盖图像处理、计算机视觉、模式识别和场景理解。大致包括ACM课程I.2.10、I.4和I.5中的材料。
--
一级分类：Electrical Engineering and Systems Science 电气工程与系统科学
二级分类：Audio and Speech Processing 音频和语音处理
分类描述：Theory and methods for processing signals representing audio, speech, and language, and their applications. This includes analysis, synthesis, enhancement, transformation, classification and interpretation of such signals as well as the design, development, and evaluation of associated signal processing systems. Machine learning and pattern analysis applied to any of the above areas is also welcome. Specific topics of interest include: auditory modeling and hearing aids; acoustic beamforming and source localization; classification of acoustic scenes; speaker separation; active noise control and echo cancellation; enhancement; de-reverberation; bioacoustics; music signals analysis, synthesis and modification; music information retrieval; audio for multimedia and joint audio-video processing; spoken and written language modeling, segmentation, tagging, parsing, understanding, and translation; text mining; speech production, perception, and psychoacoustics; speech analysis, synthesis, and perceptual modeling and coding; robust speech recognition; speaker recognition and characterization; deep learning, online learning, and graphical models applied to speech, audio, and language signals; and implementation aspects ranging from system architecture to fast algorithms.
处理代表音频、语音和语言的信号的理论和方法及其应用。这包括分析、合成、增强、转换、分类和解释这些信号，以及相关信号处理系统的设计、开发和评估。机器学习和模式分析应用于上述任何领域也是受欢迎的。感兴趣的具体主题包括：听觉建模和助听器；声波束形成与声源定位；声场景分类；说话人分离；有源噪声控制和回声消除；增强；去混响；生物声学；音乐信号的分析、合成与修饰；音乐信息检索；多媒体音频和联合音视频处理；口语和书面语建模、切分、标注、句法分析、理解和翻译；文本挖掘；言语产生、感知和心理声学；语音分析、合成、感知建模和编码；鲁棒语音识别；说话人识别与特征描述；应用于语音、音频和语言信号的深度学习、在线学习和图形模型；以及从系统架构到快速算法的实现方面。
--
---
英文摘要：
We present a joint audio-visual model for isolating a single speech signal from a mixture of sounds such as other speakers and background noise. Solving this task using only audio as input is extremely challenging and does not provide an association of the separated speech signals with speakers in the video. In this paper, we present a deep network-based model that incorporates both visual and auditory signals to solve this task. The visual features are used to "focus" the audio on desired speakers in a scene and to improve the speech separation quality. To train our joint audio-visual model, we introduce AVSpeech, a new dataset comprised of thousands of hours of video segments from the Web. We demonstrate the applicability of our method to classic speech separation tasks, as well as real-world scenarios involving heated interviews, noisy bars, and screaming children, only requiring the user to specify the face of the person in the video whose speech they want to isolate. Our method shows clear advantage over state-of-the-art audio-only speech separation in cases of mixed speech. In addition, our model, which is speaker-independent (trained once, applicable to any speaker), produces better results than recent audio-visual speech separation methods that are speaker-dependent (require training a separate model for each speaker of interest).
---
PDF链接：
https://arxiv.org/pdf/1804.03619
京公网安备 11010802022788号
