LlamaOmniOptions
Options & Configuration · AiDotNet.TextToSpeech.MultiModal
Options for LlamaOmni.
Inheritance: ModelOptions → TtsModelOptions → LlamaOmniOptions
For Beginners
LLaMA-Omni lets a text language model listen and talk. A speech encoder turns your voice into vectors the language model reads; while it writes its text answer, a small decoder reads the language model’s internal states and writes “speech units”, which a vocoder turns into sound.
How It Works
The defaults are the release, ICTNLP/Llama-3.1-8B-Omni:
- whisper-large-v3’s encoder;
- a 5× concatenating adaptor (2048 hidden);
- Llama-3.1-8B-Instruct: 32 layers, d 4096, 32 heads with 8 key-value heads, SwiGLU 14336, rope θ 500000 with
Llama 3’s ×8 scaling;
- a 2-layer, 4096-wide CTC speech decoder (32 heads, 11008) over 1000 mHuBERT units, upsampled 25×;
- fairseq’s unit HiFi-GAN at 16 kHz.
Constructors
| Constructor | Summary |
|---|---|
LlamaOmniOptions() |
Creates the release’s configuration. |
Properties
| Property | Summary |
|---|---|
LlamaOmniAlignment Alignment { get; set; } |
Which hidden state stands for a response token in decoder training: the paper’s. |
LlamaOmniTrainingStage Stage { get; set; } |
The stage a training step follows; the speech decoder’s by default, which TTS training is. |
double CtcLossWeight { get; set; } |
The CTC loss’s weight in joint training (the release’s ctc_loss_weight, 1). |
double MaxGradientNorm { get; set; } |
Gradient clipping (Hugging Face Trainer’s default, 1.0). |
double RmsEpsilon { get; set; } |
RMSNorm ε (1e-5). |
double RopeHighFrequencyFactor { get; set; } |
|
double RopeLowFrequencyFactor { get; set; } |
|
double RopeScalingFactor { get; set; } |
|
double RopeTheta { get; set; } |
The rotary base (500000) and Llama 3’s scaling: factor 8, low and high frequency factors 1 and 4, over 8192 original positions. |
double SpeechDecoderLearningRate { get; set; } |
The speech decoder’s peak rate (§4.1: 2e-4); the text stage’s is LearningRate (2e-5). |
double Temperature { get; set; } |
Sampling temperature: 0 is the paper’s greedy search (§4.2). |
double TopP { get; set; } |
The nucleus when sampling (1: none). |
double WarmupFraction { get; set; } |
The fraction of TotalSteps that warms up (§4.1: 3 %). |
int AdaptorDownsample { get; set; } |
Frames the adaptor concatenates (k = 5) and its hidden width (2048). |
int AdaptorHidden { get; set; } |
|
int EncoderContext { get; set; } |
|
int EncoderHeads { get; set; } |
|
int EncoderLayers { get; set; } |
|
int EncoderMels { get; set; } |
The Whisper encoder’s mel bands (128), frames after its stride (1500, 30 s), width (1280), heads (20) and layers (32): whisper-large-v3. |
int EncoderWidth { get; set; } |
|
int HubertConvChannels { get; set; } |
|
int HubertHeads { get; set; } |
|
int HubertIntermediate { get; set; } |
|
int HubertLayer { get; set; } |
|
int HubertWidth { get; set; } |
mHuBERT’s width (768), layers built (11, the layer read), heads (12), feed-forward width (3072) and feature-encoder channels (512), whose layer-11 units are the decoder’s targets. |
int Intermediate { get; set; } |
|
int KeyValueHeads { get; set; } |
|
int Layers { get; set; } |
Llama layers (32), key-value heads (8) and SwiGLU width (14336). |
int MaxNewTokens { get; set; } |
The most tokens a response may have (the release’s 256). |
int RopeOriginalMaxPositions { get; set; } |
|
int SpeechDecoderHeads { get; set; } |
|
int SpeechDecoderIntermediate { get; set; } |
|
int SpeechDecoderLayers { get; set; } |
The CTC decoder’s layers (2), width (4096), heads (32) and feed-forward width (11008). |
int SpeechDecoderWidth { get; set; } |
|
int TotalSteps { get; set; } |
Steps of the cosine schedule (the paper’s 3 epochs of 200K samples at batch 32: 18750), of which the first 3 % warm up. |
int UnitChunkSize { get; set; } |
The minimum unit chunk Ω the vocoder voices at a time while text is generated; 0 waits for all the units (the paper’s offline Ω = +∞). |
int Units { get; set; } |
Discrete units, K (1000 mHuBERT clusters). |
int UpsampleFactor { get; set; } |
Frames per text token, λ (25). |
int VocoderChannels { get; set; } |
|
int VocoderEmbeddingDim { get; set; } |
The unit vocoder’s embedding width (128), first width (512) and upsampling (5·4·4·2·2 = 320 samples per unit frame, 50 Hz at 16 kHz). |
int[] VocoderUpsampleKernels { get; set; } |
|
int[] VocoderUpsampleRates { get; set; } |
|
string SpeechInstruction { get; set; } |
The instruction after the speech in the user’s turn (the release’s evaluation questions). |
string SystemPrompt { get; set; } |
The system prompt (the release’s llama_3 template). |
string TokenizerPath { get; set; } |
A Hugging Face tokenizer.json for Llama 3; text needs it. |