TtsModelBase<T>

Base Classes · AiDotNet.TextToSpeech

Base class for text-to-speech neural networks that can operate in both ONNX inference and native training modes.

Inheritance: NeuralNetworkBase<T> → TtsModelBase<T>

Implements: IShapeContract

Known derived types: AcousticModelBase<T>, AlignTTS<T>, AudioPaLM<T>, BarkModel<T>, CSM<T>, ChatTTS<T>, CoMoSpeech<T>, CosyVoiceBase<T>, DeepVoice3<T>, DiTToTTS<T>, Dia<T>, E3TTS<T>, EmotiVoice<T>, FireRedTTS<T>, FishSpeechBase<T>, FlowMatchingTtsModelBase<T>, ForwardTacotron<T>, GLM4Voice<T>, GPTSoVITS<T>, GlowTTS<T>, GradTTS<T>, IndexTTS2<T>, IndexTTS<T>, KaniTTSBase<T>, Kokoro<T>, LlamaOmni<T>, Llasa<T>, MARS5TTS<T>, MaskGCT<T>, MatchaTTS<T>, MegaTTS2<T>, MegaTTS3<T>, MegaTTS<T>, MeloTTS<T>, MetaVoice1B<T>, MinMo<T>, Moshi<T>, NaturalSpeech2<T>, NaturalSpeech3<T>, NaturalSpeech<T>, NonAttentiveTacotron<T>, OpenVoiceV2<T>, OpenVoice<T>, OrpheusTTS<T>, OuteTTS<T>, ParlerTTS<T>, Pheme<T>, PortaSpeech<T>, SPEARTTS<T>, SeedTTS<T>, SoundStorm<T>, SparkTTS<T>, SpeechGPT<T>, SpeechT5<T>, SpeedySpeech<T>, SpiritLM<T>, StepAudio<T>, StyleTTS2<T>, StyleTTSZS<T>, StyleTTS<T> (+16 more)

For Beginners

Text-to-speech models convert written text into spoken audio. This base class provides:

  • Support for pre-trained ONNX models (fast inference with existing models)
  • Full training capability from scratch (like other neural networks)
  • Audio preprocessing utilities (mel-spectrogram computation, normalization)
  • Text encoding utilities (phoneme/token conversion)

You can use this class in two ways:

  1. Load a pre-trained ONNX model for quick inference
  2. Build and train a new model from scratch

How It Works

This class extends NeuralNetworkBase to provide TTS-specific functionality while maintaining full integration with the AiDotNet neural network infrastructure.

Properties

Property Summary
ILossFunction<T> DefaultLossFunction { get; } Gets the default loss function for this model.
TtsSupervision SynthesisVoiceRequirement { get; } What a voice must carry for this model to synthesize; None when text alone determines the output.
TtsSupervision TrainingSupervision { get; } Supervision outside the recording that this model trains on; anything other than None means training goes through Train.
TtsVoice<T> Voice { get; set; } The voice synthesis and prediction use, for models whose paper defines inference relative to a speaker or a reference recording (see SynthesisVoiceRequirement).
bool IsOnnxMode { get; } Gets whether this model is running in ONNX inference mode.
bool SupportsTraining { get; } Gets whether this network supports training.
bool TrainsOnSampleObjective { get; } Whether the model trains and is measured through its sample-level objective (Train, EvaluateTrainingObjective): always when it needs supervision a token/mel pair lacks, and for a model whose synthesized length follows its own prediction…
int CodecTokenCodebooks { get; } Codebooks per frame of the codec tokens this model trains on (0 for a model that does not).
int CodecTokenVocabulary { get; } Codes per codebook of those codec tokens (0 for a model that does not train on them).
int HiddenDim { get; set; } Gets the model’s hidden dimension.
int HopSize { get; set; } Gets the hop size in audio samples for mel-spectrogram computation.
int LinearSpectrogramBins { get; } Bins of the linear spectrogram target, fftSize / 2 + 1.
int MelChannels { get; set; } Gets the number of mel-spectrogram frequency channels.
int SampleRate { get; set; } Gets the audio sample rate in Hz.

Methods

Method Summary
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank) The TTS family’s output law: [Batch, Time, OutputFeatureWidth], where Time is the input’s second axis carried through.
T EvaluateTrainingObjective(TtsTrainingSample<T> sample) Evaluates the training objective on sample without updating the model, for models whose training goes through Train.
T Train(TtsTrainingSample<T> sample) Trains on several utterances, one step each.
Tensor<T> Synthesize(string text)
Tensor<T> TextToTokens(string text) Synthesizes speech (or the model’s acoustic output) from text: tokenize, run the network, postprocess.
void Train(IEnumerable<TtsTrainingSample<T>> samples) Trains on several utterances, one step each.