KaniTTS2<T>

Models & Types · AiDotNet.TextToSpeech.Latest

KaniTTS 2 (nineninesix.ai, 2026): Kani’s LFM2 language model writing NVIDIA NanoCodec codes, with frame-level rotary positions, a learnable rotary scale per attention layer and a speaker embedding from a recording (WavLM-Large speaker embeddings) inserted after the first token, so it speaks in a reference speaker’s voice.

Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → KaniTTSBase<T> → KaniTTS2<T>

For Beginners

Give KaniTTS 2 a few seconds of someone speaking and it reads text aloud in that voice; optionally a language or accent tag picks how it pronounces the text.

How It Works

KaniTTS 2 has no paper; its release is the reference: kani_tts/model.py’s KaniTTS2ForCausalLM (positions, learnable rotary scale, speaker insertion, generation), kani_tts/core.py (the prompt with a language tag, sampling, the speech span), kani_tts/speaker_embedder.py, and kani-tts-2-pretrain (the data collator and trainer).

Example

var architecture = new NeuralNetworkArchitecture<float>(InputType.OneDimensional, NeuralNetworkTaskType.Regression, inputSize: 1, outputSize: 1);
var model = new KaniTTS2<float>(architecture, new KaniTTS2Options());
model.LoadReferenceWeights("kani-tts-2-en");                                 // model.safetensors, tokenizer.json, config.json
model.LoadCodecWeights("nemo-nano-codec-22khz-0.6kbps-12.5fps.nemo");
model.LoadSpeakerEmbedderWeights("speaker-emb-tbr");                         // nineninesix/speaker-emb-tbr
model.SpeakerEmbedding = model.EmbedSpeaker(referenceRecording);              // 22.05 kHz
model.LanguageTag = "en_us";
var audio = model.Synthesize("Hello, how are you doing today?");

Constructors

Constructor Summary
KaniTTS2(NeuralNetworkArchitecture<T> architecture) Calls the fully specified overload with the omitted parameters’ defaults.
KaniTTS2(NeuralNetworkArchitecture<T> architecture, KaniTTS2Options options) Creates a native, trainable KaniTTS 2.
KaniTTS2(NeuralNetworkArchitecture<T> architecture, string modelPath, KaniTTS2Options options) Creates an ONNX-backed KaniTTS 2 for inference.

Properties

Property Summary
IDictionary<int, Tensor<T>> TrainingSpeakerEmbeddings { get; } Speaker embeddings by training sample SpeakerId — the release trains each sample on its speaker’s averaged embedding (AverageSpeakerEmbedding).
Tensor<T> SpeakerEmbedding { get; set; } The speaker embedding synthesis speaks with (SpeakerEmbeddingDim values, such as EmbedSpeaker returns); null to use the voice’s reference recording, or no speaker.
string LanguageTag { get; set; } The language or accent tag synthesis puts before the text (“en_us: text”); null for none.

Methods

Method Summary
Tensor<T> AverageSpeakerEmbedding(IEnumerable<Tensor<T>> recordings) The release’s per-speaker embedding (kanitts-2-dataset-pipeline’s regrouping): the plain mean of the speaker’s recordings’ embeddings.
Tensor<T> EmbedSpeaker(Tensor<T> recording) The speaker embedding of a recording at the model’s sample rate (kani_tts’s SpeakerEmbedder.embed_audio: resampled to 16 kHz by torchaudio’s resampler, at most 30 seconds, through the WavLM-Large embedder).
void LoadSpeakerEmbedderWeights(string path) Loads the speaker embedder: nineninesix/speaker-emb-tbr’s model.safetensors (a file, or the folder holding it).