TtsVoice<T>

Models & Types · AiDotNet.TextToSpeech

The voice a multi-speaker or adaptive text-to-speech model synthesizes in: a speaker and, for models that read one, a reference recording of that speaker.

For Beginners

Pick the speaker (its index in the speakers the model was trained on) and give a short recording of them, as a mel spectrogram, so the model can match how they sound.

How It Works

Some papers define inference only relative to a voice. AdaSpeech (Chen et al. 2021, §3) synthesizes with the speaker’s embedding and “the utterance-level acoustic conditions … extracted from another reference speech of the speaker”; text alone does not determine its output. Such a model declares what its voice must carry (SynthesisVoiceRequirement) and refuses to synthesize until Voice is set.

Constructors

Constructor Summary
TtsVoice()

Properties

Property Summary
Tensor<T> Description { get; set; } A natural-language description of the voice (“A female speaker delivers a slightly expressive speech with a moderate pitch”) as the model’s description token ids [tokens], for models that synthesize from one (Parler-TTS).
Tensor<T> Reference { get; set; } A reference recording of the speaker as a mel spectrogram, [frames, melChannels].
Tensor<T> ReferenceAudio { get; set; } A reference recording of the speaker as a waveform, [samples] at the model’s sample rate, for models whose speaker encoder reads audio (YourTTS’s H/ASP speaker encoder).
Tensor<T> ReferenceCodecTokens { get; set; } The discrete speech tokens of the reference recording [frames], for models that continue a prompt in token space (CosyVoice’s S3 tokens, which its language model and its flow both read).
Tensor<T> ReferenceTokens { get; set; } The transcript of Reference as the model’s text tokens, for models that continue a prompt from its text (E2 TTS, F5-TTS prefix it to the text to synthesize and take the speaking rate from the pair).
Tensor<T> SpeakerEmbedding { get; set; } A speaker embedding [dim] computed from a recording of the speaker, for models conditioned on one (CosyVoice’s CAM++ x-vector).
int SpeakerId { get; set; } Index of the speaker in the model’s speaker table.
int? LanguageId { get; set; } Index of the language in a multilingual model’s language table.
int[] ReferenceDurations { get; set; } Each ReferenceTokens token’s duration in the reference recording’s frames (a forced aligner’s), for models that read the prompt at the phoneme level (Mega-TTS pools its prosody encoder over the prompt’s phonemes).