KaniTTSBase<T>
Base Classes · AiDotNet.TextToSpeech.Latest
What both KaniTTS releases share: an LFM2 language model writing NVIDIA NanoCodec codes after the release’s prompt, the
codec, the speech extraction, the training sequence and its loss, and loading. Kani 2’s frame-level positions,
learnable rotary scale and speaker embedding are options (KaniTTSOptions) that KaniTTS2
turns on.
Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → KaniTTSBase<T>
Implements: ICodecTts<T>
Known derived types: KaniTTS2<T>, KaniTTS<T>
Properties
| Property | Summary |
|---|---|
bool TrainsOnSampleObjective { get; } |
|
int CodebookSize { get; } |
|
int CodecFrameRate { get; } |
Codec frames per second, rounded down (NanoCodec’s 22,050 / 1764 is 12.5). |
int CodecTokenCodebooks { get; } |
|
int CodecTokenVocabulary { get; } |
|
int MaxTextLength { get; } |
|
int NumCodebooks { get; } |
|
string Speaker { get; set; } |
The speaker a multi-speaker checkpoint speaks as (speaker_id, such as “andrew”); null for none. |
Methods
| Method | Summary |
|---|---|
LayerInputDomain GetInputDomain(int[] inputShape) |
|
ModelMetadata<T> GetModelMetadata() |
|
T CompleteTrainingStep() |
Makes an optimizer step on the samples accumulated since the last one, if any (as the Trainer does with the remainder at an epoch’s end); returns their loss, or zero when none were waiting. |
T EvaluateTrainingObjective(TtsTrainingSample<T> sample) |
|
Tensor<T> DecodeFromTokens(Tensor<T> tokens) |
|
Tensor<T> EncodeToTokens(Tensor<T> audio) |
|
void LoadCodecWeights(string path) |
Loads NanoCodec from NeMo’s .nemo archive (nemo-nano-codec-22khz-0.6kbps-12.5fps.nemo) or the model_weights.ckpt inside it; the discriminators’ tensors are not needed. |
void LoadReferenceWeights(string directory) |
Loads a released Kani 1 checkpoint directory: model.safetensors (transformers’ Lfm2ForCausalLM), tokenizer.json and, for a multi-speaker checkpoint, config.json’s speaker list. |
void LoadTokenizer(string path) |
Reads the text tokenizer from a Hugging Face tokenizer.json. |