TtsVoice<T>
Models & Types · AiDotNet.TextToSpeech
The voice a multi-speaker or adaptive text-to-speech model synthesizes in: a speaker and, for models that read one, a reference recording of that speaker.
For Beginners
Pick the speaker (its index in the speakers the model was trained on) and give a short recording of them, as a mel spectrogram, so the model can match how they sound.
How It Works
Some papers define inference only relative to a voice. AdaSpeech (Chen et al. 2021, §3) synthesizes with the
speaker’s embedding and “the utterance-level acoustic conditions … extracted from another reference speech of
the speaker”; text alone does not determine its output. Such a model declares what its voice must carry
(SynthesisVoiceRequirement) and refuses to synthesize until
Voice is set.
Constructors
| Constructor | Summary |
|---|---|
TtsVoice() |
Properties
| Property | Summary |
|---|---|
Tensor<T> Description { get; set; } |
A natural-language description of the voice (“A female speaker delivers a slightly expressive speech with a moderate pitch”) as the model’s description token ids [tokens], for models that synthesize from one (Parler-TTS). |
Tensor<T> Reference { get; set; } |
A reference recording of the speaker as a mel spectrogram, [frames, melChannels]. |
Tensor<T> ReferenceAudio { get; set; } |
A reference recording of the speaker as a waveform, [samples] at the model’s sample rate, for models whose speaker encoder reads audio (YourTTS’s H/ASP speaker encoder). |
Tensor<T> ReferenceCodecTokens { get; set; } |
The discrete speech tokens of the reference recording [frames], for models that continue a prompt in token space (CosyVoice’s S3 tokens, which its language model and its flow both read). |
Tensor<T> ReferenceTokens { get; set; } |
The transcript of Reference as the model’s text tokens, for models that continue a prompt from its text (E2 TTS, F5-TTS prefix it to the text to synthesize and take the speaking rate from the pair). |
Tensor<T> SpeakerEmbedding { get; set; } |
A speaker embedding [dim] computed from a recording of the speaker, for models conditioned on one (CosyVoice’s CAM++ x-vector). |
int SpeakerId { get; set; } |
Index of the speaker in the model’s speaker table. |
int? LanguageId { get; set; } |
Index of the language in a multilingual model’s language table. |
int[] ReferenceDurations { get; set; } |
Each ReferenceTokens token’s duration in the reference recording’s frames (a forced aligner’s), for models that read the prompt at the phoneme level (Mega-TTS pools its prosody encoder over the prompt’s phonemes). |