LlamaOmniOptions

Options & Configuration · AiDotNet.TextToSpeech.MultiModal

Options for LlamaOmni.

Inheritance: ModelOptions → TtsModelOptions → LlamaOmniOptions

For Beginners

LLaMA-Omni lets a text language model listen and talk. A speech encoder turns your voice into vectors the language model reads; while it writes its text answer, a small decoder reads the language model’s internal states and writes “speech units”, which a vocoder turns into sound.

How It Works

The defaults are the release, ICTNLP/Llama-3.1-8B-Omni:

  • whisper-large-v3’s encoder;
  • a 5× concatenating adaptor (2048 hidden);
  • Llama-3.1-8B-Instruct: 32 layers, d 4096, 32 heads with 8 key-value heads, SwiGLU 14336, rope θ 500000 with

Llama 3’s ×8 scaling;

  • a 2-layer, 4096-wide CTC speech decoder (32 heads, 11008) over 1000 mHuBERT units, upsampled 25×;
  • fairseq’s unit HiFi-GAN at 16 kHz.

Constructors

Constructor Summary
LlamaOmniOptions() Creates the release’s configuration.

Properties

Property Summary
LlamaOmniAlignment Alignment { get; set; } Which hidden state stands for a response token in decoder training: the paper’s.
LlamaOmniTrainingStage Stage { get; set; } The stage a training step follows; the speech decoder’s by default, which TTS training is.
double CtcLossWeight { get; set; } The CTC loss’s weight in joint training (the release’s ctc_loss_weight, 1).
double MaxGradientNorm { get; set; } Gradient clipping (Hugging Face Trainer’s default, 1.0).
double RmsEpsilon { get; set; } RMSNorm ε (1e-5).
double RopeHighFrequencyFactor { get; set; }
double RopeLowFrequencyFactor { get; set; }
double RopeScalingFactor { get; set; }
double RopeTheta { get; set; } The rotary base (500000) and Llama 3’s scaling: factor 8, low and high frequency factors 1 and 4, over 8192 original positions.
double SpeechDecoderLearningRate { get; set; } The speech decoder’s peak rate (§4.1: 2e-4); the text stage’s is LearningRate (2e-5).
double Temperature { get; set; } Sampling temperature: 0 is the paper’s greedy search (§4.2).
double TopP { get; set; } The nucleus when sampling (1: none).
double WarmupFraction { get; set; } The fraction of TotalSteps that warms up (§4.1: 3 %).
int AdaptorDownsample { get; set; } Frames the adaptor concatenates (k = 5) and its hidden width (2048).
int AdaptorHidden { get; set; }
int EncoderContext { get; set; }
int EncoderHeads { get; set; }
int EncoderLayers { get; set; }
int EncoderMels { get; set; } The Whisper encoder’s mel bands (128), frames after its stride (1500, 30 s), width (1280), heads (20) and layers (32): whisper-large-v3.
int EncoderWidth { get; set; }
int HubertConvChannels { get; set; }
int HubertHeads { get; set; }
int HubertIntermediate { get; set; }
int HubertLayer { get; set; }
int HubertWidth { get; set; } mHuBERT’s width (768), layers built (11, the layer read), heads (12), feed-forward width (3072) and feature-encoder channels (512), whose layer-11 units are the decoder’s targets.
int Intermediate { get; set; }
int KeyValueHeads { get; set; }
int Layers { get; set; } Llama layers (32), key-value heads (8) and SwiGLU width (14336).
int MaxNewTokens { get; set; } The most tokens a response may have (the release’s 256).
int RopeOriginalMaxPositions { get; set; }
int SpeechDecoderHeads { get; set; }
int SpeechDecoderIntermediate { get; set; }
int SpeechDecoderLayers { get; set; } The CTC decoder’s layers (2), width (4096), heads (32) and feed-forward width (11008).
int SpeechDecoderWidth { get; set; }
int TotalSteps { get; set; } Steps of the cosine schedule (the paper’s 3 epochs of 200K samples at batch 32: 18750), of which the first 3 % warm up.
int UnitChunkSize { get; set; } The minimum unit chunk Ω the vocoder voices at a time while text is generated; 0 waits for all the units (the paper’s offline Ω = +∞).
int Units { get; set; } Discrete units, K (1000 mHuBERT clusters).
int UpsampleFactor { get; set; } Frames per text token, λ (25).
int VocoderChannels { get; set; }
int VocoderEmbeddingDim { get; set; } The unit vocoder’s embedding width (128), first width (512) and upsampling (5·4·4·2·2 = 320 samples per unit frame, 50 Hz at 16 kHz).
int[] VocoderUpsampleKernels { get; set; }
int[] VocoderUpsampleRates { get; set; }
string SpeechInstruction { get; set; } The instruction after the speech in the user’s turn (the release’s evaluation questions).
string SystemPrompt { get; set; } The system prompt (the release’s llama_3 template).
string TokenizerPath { get; set; } A Hugging Face tokenizer.json for Llama 3; text needs it.