LlamaOmni<T>

Models & Types · AiDotNet.TextToSpeech.MultiModal

LLaMA-Omni (Fang et al., ICLR 2025): low-latency speech interaction with an LLM. A frozen Whisper encoder and a speech adaptor feed speech to Llama-3.1-8B-Instruct, which writes the text response; a non-autoregressive streaming speech decoder reads the LLM’s hidden states as they come and writes discrete units by CTC, which a unit vocoder voices.

Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → LlamaOmni<T>

For Beginners

LLaMA-Omni is a talking chatbot built on Llama 3. It listens to your question, writes its answer, and speaks it at the same time.

How It Works

Speech in (§2.1–2.2): whisper-large-v3’s encoder on the 30-second log-mel window, every 5 frames concatenated and mapped by a 2-layer ReLU perceptron into the LLM’s embeddings, in place of <speech> in the llama_3 prompt.

Speech out (§2.4): each hidden state c_i (the one that predicts response token y_i) is repeated λ = 25 times, projected, and passed through two causal Llama layers to logits over K = 1000 mHuBERT units and a blank; the best alignment, collapsed, is the unit sequence; fairseq’s unit HiFi-GAN with a duration predictor voices it, chunk by chunk of Ω units while the text is still being written (Algorithm 1), or all at once (Ω = +∞).

Training (§2.5, §4.1): stage 1 trains the adaptor and the LLM on the text response; stage 2 trains only the speech decoder by CTC on the deduplicated units of the response’s speech.

Synthesize reads a text aloud: the text is the user’s turn and is fed as the response, and the speech decoder voices it. Chat answers 16 kHz speech.

Example

var architecture = new NeuralNetworkArchitecture<float>(InputType.OneDimensional, NeuralNetworkTaskType.Regression, inputSize: 1, outputSize: 1);
var model = new LlamaOmni<float>(architecture, new LlamaOmniOptions { TokenizerPath = "Llama-3.1-8B-Omni/tokenizer.json" });
model.LoadReleasedWeights("Llama-3.1-8B-Omni");          // speech encoder, adaptor, LLM and speech decoder
model.LoadVocoder("vocoder/g_00500000");
var response = model.Chat(question16kHz);                // response.Text, response.Audio
var audio = model.Synthesize("Hello world.");

Constructors

Constructor Summary
LlamaOmni(NeuralNetworkArchitecture<T> architecture) Calls the fully specified overload with the omitted parameters’ defaults.
LlamaOmni(NeuralNetworkArchitecture<T> architecture, LlamaOmniOptions options) Creates a native, trainable LLaMA-Omni.
LlamaOmni(NeuralNetworkArchitecture<T> architecture, string modelPath, LlamaOmniOptions options) Creates an ONNX-backed LLaMA-Omni for inference.

Properties

Property Summary
bool TrainsOnSampleObjective { get; }
int MaxTextLength { get; } The longest sequence the language model takes (the release’s 2048).

Methods

Method Summary
LayerInputDomain GetInputDomain(int[] inputShape)
LlamaOmniResponse<T> Chat(Tensor<T> recording) LLaMA-Omni’s response to 16 kHz speech: the text, the units and the speech.
ModelMetadata<T> GetModelMetadata()
T EvaluateTrainingObjective(TtsTrainingSample<T> sample)
T TrainTurn(Tensor<T> instruction, string response, Tensor<T> responseSpeech) One update on an InstructS2S triplet (§3): a 16 kHz speech instruction, the text response and the response’s 16 kHz speech (whose units the decoder stages read; not needed for the text stage).
Tensor<T> SpeechEmbeddings(Tensor<T> recording) The speech embeddings [frames / k, d] of 16 kHz audio: pad_or_trim to the encoder’s window (30 s), Whisper’s log-mel, the encoder, the adaptor (§2.1–2.2).
Tensor<T> Synthesize(string text)
int[] TokenizeText(string text) The text token ids of text (Llama 3’s tokenizer, no special tokens added).
string DecodeText(IEnumerable<int> ids) The text of token ids, special tokens dropped.
void LoadReleasedWeights(string directory) Loads ICTNLP/Llama-3.1-8B-Omni (its safetensors shards): the speech encoder (model.speech_encoder), the adaptor (model.speech_projector), the LLM (model.*, lm_head) and the speech decoder (speech_generator), and, unless one is set…
void LoadUnitExtractor(string hubertCheckpoint, string kmeansModel) Loads the unit extractor that makes the decoder’s training targets: fairseq’s mHuBERT checkpoint and its k-means model (mhubert_base_vp_en_es_fr_it3.pt, mhubert_base_vp_en_es_fr_it3_L11_km1000.bin).
void LoadVocoder(string path) Loads fairseq’s unit HiFi-GAN (g_00500000 of mhubert_vp_en_es_fr_it3_400k_layer11_km1000_lj).