LlamaOmni<T>
Models & Types · AiDotNet.TextToSpeech.MultiModal
LLaMA-Omni (Fang et al., ICLR 2025): low-latency speech interaction with an LLM. A frozen Whisper encoder and a speech adaptor feed speech to Llama-3.1-8B-Instruct, which writes the text response; a non-autoregressive streaming speech decoder reads the LLM’s hidden states as they come and writes discrete units by CTC, which a unit vocoder voices.
Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → LlamaOmni<T>
For Beginners
LLaMA-Omni is a talking chatbot built on Llama 3. It listens to your question, writes its answer, and speaks it at the same time.
How It Works
Speech in (§2.1–2.2): whisper-large-v3’s encoder on the 30-second log-mel window, every 5 frames
concatenated and mapped by a 2-layer ReLU perceptron into the LLM’s embeddings, in place of <speech> in
the llama_3 prompt.
Speech out (§2.4): each hidden state c_i (the one that predicts response token y_i) is repeated λ = 25 times, projected, and passed through two causal Llama layers to logits over K = 1000 mHuBERT units and a blank; the best alignment, collapsed, is the unit sequence; fairseq’s unit HiFi-GAN with a duration predictor voices it, chunk by chunk of Ω units while the text is still being written (Algorithm 1), or all at once (Ω = +∞).
Training (§2.5, §4.1): stage 1 trains the adaptor and the LLM on the text response; stage 2 trains only the speech decoder by CTC on the deduplicated units of the response’s speech.
Synthesize reads a text aloud: the text is the user’s turn and is fed as the response, and the
speech decoder voices it. Chat answers 16 kHz speech.
Example
var architecture = new NeuralNetworkArchitecture<float>(InputType.OneDimensional, NeuralNetworkTaskType.Regression, inputSize: 1, outputSize: 1);
var model = new LlamaOmni<float>(architecture, new LlamaOmniOptions { TokenizerPath = "Llama-3.1-8B-Omni/tokenizer.json" });
model.LoadReleasedWeights("Llama-3.1-8B-Omni"); // speech encoder, adaptor, LLM and speech decoder
model.LoadVocoder("vocoder/g_00500000");
var response = model.Chat(question16kHz); // response.Text, response.Audio
var audio = model.Synthesize("Hello world.");
Constructors
| Constructor | Summary |
|---|---|
LlamaOmni(NeuralNetworkArchitecture<T> architecture) |
Calls the fully specified overload with the omitted parameters’ defaults. |
LlamaOmni(NeuralNetworkArchitecture<T> architecture, LlamaOmniOptions options) |
Creates a native, trainable LLaMA-Omni. |
LlamaOmni(NeuralNetworkArchitecture<T> architecture, string modelPath, LlamaOmniOptions options) |
Creates an ONNX-backed LLaMA-Omni for inference. |
Properties
| Property | Summary |
|---|---|
bool TrainsOnSampleObjective { get; } |
|
int MaxTextLength { get; } |
The longest sequence the language model takes (the release’s 2048). |
Methods
| Method | Summary |
|---|---|
LayerInputDomain GetInputDomain(int[] inputShape) |
|
LlamaOmniResponse<T> Chat(Tensor<T> recording) |
LLaMA-Omni’s response to 16 kHz speech: the text, the units and the speech. |
ModelMetadata<T> GetModelMetadata() |
|
T EvaluateTrainingObjective(TtsTrainingSample<T> sample) |
|
T TrainTurn(Tensor<T> instruction, string response, Tensor<T> responseSpeech) |
One update on an InstructS2S triplet (§3): a 16 kHz speech instruction, the text response and the response’s 16 kHz speech (whose units the decoder stages read; not needed for the text stage). |
Tensor<T> SpeechEmbeddings(Tensor<T> recording) |
The speech embeddings [frames / k, d] of 16 kHz audio: pad_or_trim to the encoder’s window (30 s), Whisper’s log-mel, the encoder, the adaptor (§2.1–2.2). |
Tensor<T> Synthesize(string text) |
|
int[] TokenizeText(string text) |
The text token ids of text (Llama 3’s tokenizer, no special tokens added). |
string DecodeText(IEnumerable<int> ids) |
The text of token ids, special tokens dropped. |
void LoadReleasedWeights(string directory) |
Loads ICTNLP/Llama-3.1-8B-Omni (its safetensors shards): the speech encoder (model.speech_encoder), the adaptor (model.speech_projector), the LLM (model.*, lm_head) and the speech decoder (speech_generator), and, unless one is set… |
void LoadUnitExtractor(string hubertCheckpoint, string kmeansModel) |
Loads the unit extractor that makes the decoder’s training targets: fairseq’s mHuBERT checkpoint and its k-means model (mhubert_base_vp_en_es_fr_it3.pt, mhubert_base_vp_en_es_fr_it3_L11_km1000.bin). |
void LoadVocoder(string path) |
Loads fairseq’s unit HiFi-GAN (g_00500000 of mhubert_vp_en_es_fr_it3_400k_layer11_km1000_lj). |