Llasa<T>

Models & Types · AiDotNet.TextToSpeech.CodecBased

Llasa (Ye et al. 2025): text-to-speech as plain language modeling — a Llama model reads the text in a chat turn and answers with X-codec2 speech tokens, one per 20 ms, which X-codec2’s decoder turns into 16 kHz audio.

Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → Llasa<T>

Implements: ICodecTts<T>

For Beginners

Llasa treats speech as a language a chatbot can write: asked to “convert the text to speech”, it answers in sound tokens, and a neural codec turns those tokens into audio.

How It Works

The paper (§2.1–2.2, §3.2.1) fixes the design: a single Transformer initialized from Llama, the text tokens on the left and the speech tokens on the right, the loss on the speech tokens, and X-codec2 (a w2v-BERT 2.0 semantic encoder and a convolutional acoustic encoder fused and quantized by one FSQ codebook, decoded by a Transformer Vocos). The official code fills the rest: the chat prompt and sampling of the HKUSTAudio/Llasa-1B model card, the training rows of zhenye234/LLaSA_training (train_tts.py, get_memmap_from_token.py), and the codec of the xcodec2 0.1.5 package.

A turn is Llama 3.2’s chat template: the system header with its dates, the user’s “Convert the text to speech:<|TEXT_UNDERSTANDING_START|>{text}<|TEXT_UNDERSTANDING_END|>”, and the assistant’s “<|SPEECH_GENERATION_START|>” followed by speech tokens <|s_k|> (id 128264 + k) until “<|SPEECH_GENERATION_END|>”. Given a recording and its transcript as a prompt, the transcript goes before the text and the recording’s codes open the answer, and the model continues in that voice.

Example

var architecture = new NeuralNetworkArchitecture<float>(InputType.OneDimensional, NeuralNetworkTaskType.Regression, inputSize: 1, outputSize: 1);
var model = new Llasa<float>(architecture, new LlasaOptions());
model.LoadReferenceWeights("Llasa-1B");               // model.safetensors (and tokenizer.json)
model.LoadCodecWeights("xcodec2/model.safetensors");
var audio = model.Synthesize("Dealing with family secrets is never easy.");

Constructors

Constructor Summary
Llasa(NeuralNetworkArchitecture<T> architecture) Calls the fully specified overload with the omitted parameters’ defaults.
Llasa(NeuralNetworkArchitecture<T> architecture, LlasaOptions options) Creates a native, trainable Llasa.
Llasa(NeuralNetworkArchitecture<T> architecture, string modelPath, LlasaOptions options) Creates an ONNX-backed Llasa for inference.

Properties

Property Summary
bool TrainsOnSampleObjective { get; }
int CodebookSize { get; }
int CodecFrameRate { get; } Codes per second (50 at 16 kHz).
int CodecTokenCodebooks { get; }
int CodecTokenVocabulary { get; }
int MaxTextLength { get; }
int NumCodebooks { get; }

Methods

Method Summary
LayerInputDomain GetInputDomain(int[] inputShape)
ModelMetadata<T> GetModelMetadata()
T EvaluateTrainingObjective(TtsTrainingSample<T> sample)
Tensor<T> DecodeFromTokens(Tensor<T> tokens)
Tensor<T> EncodeToTokens(Tensor<T> audio)
Tensor<T> Synthesize(string text)
Tensor<T> SynthesizeWithPrompt(string text, Tensor<T> promptAudio, string promptText) Speaks text in the voice of a 16 kHz recording whose transcript is promptText, as the model card’s prompted synthesis does: the transcript and the text joined by a space between the text-understanding markers, the recording’s codes open…
void LoadCodecWeights(string path) Loads X-codec2: HKUSTAudio/xcodec2’s model.safetensors (or its pytorch_model.bin).
void LoadReferenceWeights(string directory) Loads a released Llasa model directory: its *.safetensors (the Hugging Face LlamaForCausalLM state dict) and, when present, its tokenizer.json.
void LoadTokenizer(string path) Reads the language model’s tokenizer from a Hugging Face tokenizer.json (a fine-tune’s own).