TtsModelBase<T>
Base Classes · AiDotNet.TextToSpeech
Base class for text-to-speech neural networks that can operate in both ONNX inference and native training modes.
Inheritance: NeuralNetworkBase<T> → TtsModelBase<T>
Implements: IShapeContract
Known derived types: AcousticModelBase<T>, AlignTTS<T>, AudioPaLM<T>, BarkModel<T>, CSM<T>, ChatTTS<T>, CoMoSpeech<T>, CosyVoiceBase<T>, DeepVoice3<T>, DiTToTTS<T>, Dia<T>, E3TTS<T>, EmotiVoice<T>, FireRedTTS<T>, FishSpeechBase<T>, FlowMatchingTtsModelBase<T>, ForwardTacotron<T>, GLM4Voice<T>, GPTSoVITS<T>, GlowTTS<T>, GradTTS<T>, IndexTTS2<T>, IndexTTS<T>, KaniTTSBase<T>, Kokoro<T>, LlamaOmni<T>, Llasa<T>, MARS5TTS<T>, MaskGCT<T>, MatchaTTS<T>, MegaTTS2<T>, MegaTTS3<T>, MegaTTS<T>, MeloTTS<T>, MetaVoice1B<T>, MinMo<T>, Moshi<T>, NaturalSpeech2<T>, NaturalSpeech3<T>, NaturalSpeech<T>, NonAttentiveTacotron<T>, OpenVoiceV2<T>, OpenVoice<T>, OrpheusTTS<T>, OuteTTS<T>, ParlerTTS<T>, Pheme<T>, PortaSpeech<T>, SPEARTTS<T>, SeedTTS<T>, SoundStorm<T>, SparkTTS<T>, SpeechGPT<T>, SpeechT5<T>, SpeedySpeech<T>, SpiritLM<T>, StepAudio<T>, StyleTTS2<T>, StyleTTSZS<T>, StyleTTS<T> (+16 more)
For Beginners
Text-to-speech models convert written text into spoken audio. This base class provides:
- Support for pre-trained ONNX models (fast inference with existing models)
- Full training capability from scratch (like other neural networks)
- Audio preprocessing utilities (mel-spectrogram computation, normalization)
- Text encoding utilities (phoneme/token conversion)
You can use this class in two ways:
- Load a pre-trained ONNX model for quick inference
- Build and train a new model from scratch
How It Works
This class extends NeuralNetworkBase to provide TTS-specific functionality
while maintaining full integration with the AiDotNet neural network infrastructure.
Properties
| Property | Summary |
|---|---|
ILossFunction<T> DefaultLossFunction { get; } |
Gets the default loss function for this model. |
TtsSupervision SynthesisVoiceRequirement { get; } |
What a voice must carry for this model to synthesize; None when text alone determines the output. |
TtsSupervision TrainingSupervision { get; } |
Supervision outside the recording that this model trains on; anything other than None means training goes through Train. |
TtsVoice<T> Voice { get; set; } |
The voice synthesis and prediction use, for models whose paper defines inference relative to a speaker or a reference recording (see SynthesisVoiceRequirement). |
bool IsOnnxMode { get; } |
Gets whether this model is running in ONNX inference mode. |
bool SupportsTraining { get; } |
Gets whether this network supports training. |
bool TrainsOnSampleObjective { get; } |
Whether the model trains and is measured through its sample-level objective (Train, EvaluateTrainingObjective): always when it needs supervision a token/mel pair lacks, and for a model whose synthesized length follows its own prediction… |
int CodecTokenCodebooks { get; } |
Codebooks per frame of the codec tokens this model trains on (0 for a model that does not). |
int CodecTokenVocabulary { get; } |
Codes per codebook of those codec tokens (0 for a model that does not train on them). |
int HiddenDim { get; set; } |
Gets the model’s hidden dimension. |
int HopSize { get; set; } |
Gets the hop size in audio samples for mel-spectrogram computation. |
int LinearSpectrogramBins { get; } |
Bins of the linear spectrogram target, fftSize / 2 + 1. |
int MelChannels { get; set; } |
Gets the number of mel-spectrogram frequency channels. |
int SampleRate { get; set; } |
Gets the audio sample rate in Hz. |
Methods
| Method | Summary |
|---|---|
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank) |
The TTS family’s output law: [Batch, Time, OutputFeatureWidth], where Time is the input’s second axis carried through. |
T EvaluateTrainingObjective(TtsTrainingSample<T> sample) |
Evaluates the training objective on sample without updating the model, for models whose training goes through Train. |
T Train(TtsTrainingSample<T> sample) |
Trains on several utterances, one step each. |
Tensor<T> Synthesize(string text) |
|
Tensor<T> TextToTokens(string text) |
Synthesizes speech (or the model’s acoustic output) from text: tokenize, run the network, postprocess. |
void Train(IEnumerable<TtsTrainingSample<T>> samples) |
Trains on several utterances, one step each. |