TtsTrainingSample<T>

Models & Types · AiDotNet.TextToSpeech

One utterance of text-to-speech training data, with whatever supervision the model’s paper trains on.

For Beginners

Fill in the text tokens and either the recording or its mel spectrogram. Add durations when your model needs an alignment; the error message tells you if it does.

How It Works

A plain Train(tokens, mel) call carries the text and the target spectrogram and nothing else, but most acoustic models are trained on more: FastSpeech 2 on forced-alignment durations and frame-level pitch and energy (Ren et al. 2021, §2.3), codec language models on codec tokens and a speaker prompt, zero-shot models on reference audio. This type carries all of it. Anything a paper derives from the recording itself — pitch with WORLD, energy from the STFT, the mel spectrogram — the model derives from Audio when it is not supplied; anything a paper takes from an external tool (FastSpeech 2’s Montreal Forced Aligner durations) must be supplied, and the model says so by name when it is missing.

Constructors

Constructor Summary
TtsTrainingSample()

Properties

Property Summary
Tensor<T> Audio { get; set; } The recording, [samples], at the model’s sample rate.
Tensor<T> CodecTokens { get; set; } Discrete codec tokens of the recording, [frames, codebooks].
Tensor<T> Description { get; set; } The description of how the recording sounds (speaker, pitch, speed, recording quality), as the model’s description token ids [tokens], for models that synthesize from a description (Parler-TTS).
Tensor<T> Energy { get; set; } Energy per mel frame.
Tensor<T> LinearSpectrogram { get; set; } Target log-magnitude linear spectrogram, [frames, fftSize / 2 + 1], ln(max(clip, |STFT|)).
Tensor<T> Mel { get; set; } Target mel spectrogram, [frames, melChannels].
Tensor<T> Pitch { get; set; } F0 per mel frame in Hz, 0 where unvoiced.
Tensor<T> PromptCodecTokens { get; set; } Discrete codec tokens of another utterance of the same speaker, [frames, codebooks], for models that train with an acoustic prompt from a different sentence (VALL-E X).
Tensor<T> SpeakerReference { get; set; } Reference audio or embedding for the target speaker or style.
Tensor<T> Tokens { get; set; } Text (phoneme or character) token ids, [tokens].
Tensor<T> Transcript { get; set; } The utterance’s text as the token ids [tokens] of the language model a model reads it through, for models that embed the text itself as well as its phonemes (EmotiVoice’s SimBERT content embedding).
int? ClassLabel { get; set; } A global discrete label of the recording, such as the spoken word (DiffWave’s class conditioner).
int? LanguageId { get; set; } Index of the language in a multilingual model’s language table.
int? SpeakerId { get; set; } Index of the speaker in a multi-speaker model’s speaker table.
int[] Durations { get; set; } Frames per token from a forced alignment; the values sum to the number of mel frames.
int[] WordLengths { get; set; } Tokens per word, in order, summing to the token count, for models that align at the word level (PortaSpeech).
string Text { get; set; } The utterance’s text, for models that run their own front end and text encoders on it during training (MeloTTS reads its phonemes, tones and BERT features from the sentence).