Llasa<T>
Models & Types · AiDotNet.TextToSpeech.CodecBased
Llasa (Ye et al. 2025): text-to-speech as plain language modeling — a Llama model reads the text in a chat turn and answers with X-codec2 speech tokens, one per 20 ms, which X-codec2’s decoder turns into 16 kHz audio.
Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → Llasa<T>
Implements: ICodecTts<T>
For Beginners
Llasa treats speech as a language a chatbot can write: asked to “convert the text to speech”, it answers in sound tokens, and a neural codec turns those tokens into audio.
How It Works
The paper (§2.1–2.2, §3.2.1) fixes the design: a single Transformer initialized from Llama, the text tokens on the
left and the speech tokens on the right, the loss on the speech tokens, and X-codec2 (a w2v-BERT 2.0 semantic encoder
and a convolutional acoustic encoder fused and quantized by one FSQ codebook, decoded by a Transformer Vocos). The
official code fills the rest: the chat prompt and sampling of the HKUSTAudio/Llasa-1B model card, the training rows of
zhenye234/LLaSA_training (train_tts.py, get_memmap_from_token.py), and the codec of the xcodec2 0.1.5
package.
A turn is Llama 3.2’s chat template: the system header with its dates, the user’s
“Convert the text to speech:<|TEXT_UNDERSTANDING_START|>{text}<|TEXT_UNDERSTANDING_END|>”, and the assistant’s
“<|SPEECH_GENERATION_START|>” followed by speech tokens <|s_k|> (id 128264 + k) until
“<|SPEECH_GENERATION_END|>”. Given a recording and its transcript as a prompt, the transcript goes before the text
and the recording’s codes open the answer, and the model continues in that voice.
Example
var architecture = new NeuralNetworkArchitecture<float>(InputType.OneDimensional, NeuralNetworkTaskType.Regression, inputSize: 1, outputSize: 1);
var model = new Llasa<float>(architecture, new LlasaOptions());
model.LoadReferenceWeights("Llasa-1B"); // model.safetensors (and tokenizer.json)
model.LoadCodecWeights("xcodec2/model.safetensors");
var audio = model.Synthesize("Dealing with family secrets is never easy.");
Constructors
| Constructor | Summary |
|---|---|
Llasa(NeuralNetworkArchitecture<T> architecture) |
Calls the fully specified overload with the omitted parameters’ defaults. |
Llasa(NeuralNetworkArchitecture<T> architecture, LlasaOptions options) |
Creates a native, trainable Llasa. |
Llasa(NeuralNetworkArchitecture<T> architecture, string modelPath, LlasaOptions options) |
Creates an ONNX-backed Llasa for inference. |
Properties
| Property | Summary |
|---|---|
bool TrainsOnSampleObjective { get; } |
|
int CodebookSize { get; } |
|
int CodecFrameRate { get; } |
Codes per second (50 at 16 kHz). |
int CodecTokenCodebooks { get; } |
|
int CodecTokenVocabulary { get; } |
|
int MaxTextLength { get; } |
|
int NumCodebooks { get; } |
Methods
| Method | Summary |
|---|---|
LayerInputDomain GetInputDomain(int[] inputShape) |
|
ModelMetadata<T> GetModelMetadata() |
|
T EvaluateTrainingObjective(TtsTrainingSample<T> sample) |
|
Tensor<T> DecodeFromTokens(Tensor<T> tokens) |
|
Tensor<T> EncodeToTokens(Tensor<T> audio) |
|
Tensor<T> Synthesize(string text) |
|
Tensor<T> SynthesizeWithPrompt(string text, Tensor<T> promptAudio, string promptText) |
Speaks text in the voice of a 16 kHz recording whose transcript is promptText, as the model card’s prompted synthesis does: the transcript and the text joined by a space between the text-understanding markers, the recording’s codes open… |
void LoadCodecWeights(string path) |
Loads X-codec2: HKUSTAudio/xcodec2’s model.safetensors (or its pytorch_model.bin). |
void LoadReferenceWeights(string directory) |
Loads a released Llasa model directory: its *.safetensors (the Hugging Face LlamaForCausalLM state dict) and, when present, its tokenizer.json. |
void LoadTokenizer(string path) |
Reads the language model’s tokenizer from a Hugging Face tokenizer.json (a fine-tune’s own). |