KaniTTSBase<T>

Base Classes · AiDotNet.TextToSpeech.Latest

What both KaniTTS releases share: an LFM2 language model writing NVIDIA NanoCodec codes after the release’s prompt, the codec, the speech extraction, the training sequence and its loss, and loading. Kani 2’s frame-level positions, learnable rotary scale and speaker embedding are options (KaniTTSOptions) that KaniTTS2 turns on.

Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → KaniTTSBase<T>

Implements: ICodecTts<T>

Known derived types: KaniTTS2<T>, KaniTTS<T>

Properties

Property Summary
bool TrainsOnSampleObjective { get; }
int CodebookSize { get; }
int CodecFrameRate { get; } Codec frames per second, rounded down (NanoCodec’s 22,050 / 1764 is 12.5).
int CodecTokenCodebooks { get; }
int CodecTokenVocabulary { get; }
int MaxTextLength { get; }
int NumCodebooks { get; }
string Speaker { get; set; } The speaker a multi-speaker checkpoint speaks as (speaker_id, such as “andrew”); null for none.

Methods

Method Summary
LayerInputDomain GetInputDomain(int[] inputShape)
ModelMetadata<T> GetModelMetadata()
T CompleteTrainingStep() Makes an optimizer step on the samples accumulated since the last one, if any (as the Trainer does with the remainder at an epoch’s end); returns their loss, or zero when none were waiting.
T EvaluateTrainingObjective(TtsTrainingSample<T> sample)
Tensor<T> DecodeFromTokens(Tensor<T> tokens)
Tensor<T> EncodeToTokens(Tensor<T> audio)
void LoadCodecWeights(string path) Loads NanoCodec from NeMo’s .nemo archive (nemo-nano-codec-22khz-0.6kbps-12.5fps.nemo) or the model_weights.ckpt inside it; the discriminators’ tensors are not needed.
void LoadReferenceWeights(string directory) Loads a released Kani 1 checkpoint directory: model.safetensors (transformers’ Lfm2ForCausalLM), tokenizer.json and, for a multi-speaker checkpoint, config.json’s speaker list.
void LoadTokenizer(string path) Reads the text tokenizer from a Hugging Face tokenizer.json.