微软推出语音合成模型 NaturalSpeech2：语音重构“更准确”，不会“棒读”

 7 月 27 日消息，微软日前推出了一款名为 NaturalSpeech2 的语音模型，该模型采用“潜在扩散”式设计，在零样本语音合成层面效果出众，微软宣称该模型提供了“商业级”的语音 / 歌唱解决方案，能够给予用户高质量、多样化的语音合成体验。

微软对 NaturalSpeech2 进行了一系列演示，展示了其在零样本情况下生成具有不同说话人身份、韵律和风格（如唱歌）的语音的能力。

据悉，与传统的语音转文字（TTS）系统不同，微软的 NaturalSpeech2 使用“连续向量”取代“离散标记”来表示语音，从而生成更完整的语音片段，不会产生“缺乏感情”的“棒读（一字一顿地讲话）”现象。

实验结果表明，NaturalSpeech2 在零样本条件下生成的语音与语音提示和真实语音的韵律近乎一致，并且在 LibriTTS 和 VCTK 测试集上的自然度（以 CMOS 为度量）与真人语音难以区分。

该项目的论文目前已经发布于 GitHub 中，论文摘要：

NaturalSpeech 2:Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizers

Abstract. While text-to-speech (TTS) systems (e.g., NaturalSpeech) have achieved high speech quality on single-speaker recording-studio datasets, these datasets are not enough to capture the diversity in human speech such as speaker identities, prosodies, styles (e.g., singing). When scaling to large-scale, multi-speaker, and in- the-wild datasets, current TTS systems usually quantize speech into discrete tokens and use language models to generate these tokens one by one, which suffer from unstable prosody, word skipping/repeating issue, and poor voice quality. In this paper, we develop NaturalSpeech 2, a TTS system that uses a latent diffusion model to synthesize natural voices with high expressiveness/robustness/fidelity and strong zero-shot ability. Specifically, we leverage a neural audio codec with residual vector quantizers to reconstruct speech waveform and get the quantized latent vectors, and then use a diffusion model to generate these latent vectors conditioned on text input. To enhance the zero-shot capability, we design a speech prompting mechanism to facilitate in-context learning in the duration/pitch predictor and diffusion model. We scale NaturalSpeech 2 to large-scale datasets with 44K hours of speech and singing data and evaluate its voice quality on unseen (zero-shot) speakers. NaturalSpeech 2 outperforms previous TTS systems by a large margin in terms of prosody/timbre similarity, robustness, and voice quality, and can perform novel zero-shot singing synthesis with only a speech prompt.

This research is done in alignment with Microsoft’s responsible AI principles.

查看更多，请点击链接：NaturalSpeech 2 (speechresearch.github.io)