MoshiOptions

Options & Configuration · AiDotNet.TextToSpeech.MultiModal

Options for Moshi and its speech recognizer.

Inheritance: ModelOptions → TtsModelOptions → MoshiOptions

For Beginners

Moshi listens and speaks at the same time: every 80 ms it reads the user’s sound, writes a token of what it is about to say, and the sound of its own voice. It can talk over you, pause, and be interrupted.

How It Works

The defaults are the paper’s Moshi: the Helium 7B Temporal Transformer (32 layers, d 4096, 32 heads, feed-forward 11264 behind a SiLU gate, RMSNorm, RoPE, a 3000-step context), the 6-layer, 1024-wide Depth Transformer with weights per codebook, Mimi’s 8 codebooks of 2048 at 12.5 Hz, the inner monologue’s 32,000-token text stream, an acoustic delay of 1, and the Depth Transformer predicting every stream (Eq. 6). Release is kyutai/moshiko.

Constructors

Constructor Summary
MoshiOptions() Creates the paper’s Moshi.

Properties

Property Summary
MoshiDepthLayout DepthLayout { get; set; } Which streams the Depth Transformer predicts.
MoshiTrainingStage Stage { get; set; } The stage a language-model step follows.
MoshiVariant Variant { get; set; } The text/audio delay mode, and its size for Tts and Asr (App.
bool IncludeMimiTraining { get; set; } Whether to build what training Mimi needs: the multi-scale STFT discriminator and WavLM for distillation.
bool MimiQuantizerDropout { get; set; }
double AudioTemperature { get; set; } Sampling (the release’s LMGen): audio at temperature 0.8 with top-k 250, text at 0.7 with top-k 25.
double DepthLearningRate { get; set; } Pre-training’s Depth Transformer rate (Table 1: 2e-4); LearningRate is its Temporal Transformer’s (3e-5).
double MaxGradientNorm { get; set; } Gradient clipping (1.0).
double MimiAdversarialWeight { get; set; } The adversarial-only losses’ weights (Audiocraft’s compression defaults the paper points to: adversarial 4, feature 4) under the loss balancer, and the WavLM distillation’s (not stated; 1).
double MimiDistillationWeight { get; set; }
double MimiFeatureWeight { get; set; }
double MimiLearningRate { get; set; } Mimi’s optimizer (§3.3): AdamW at 8e-4 with β (0.5, 0.9), weight decay 5e-2 on the Transformers only, and an exponential moving average of the weights at 0.99.
double MimiQuantizationRate { get; set; } The probability of quantizing a training sequence (§3.3: 0.5), and whether the acoustic codebooks drop out.
double MimiWeightAverageDecay { get; set; }
double MimiWeightDecay { get; set; }
double SemanticWeight { get; set; }
double TextAudioDelaySeconds { get; set; }
double TextDelaySeconds { get; set; }
double TextMaskProbability { get; set; } Pre-training’s text masking probability (0.3) and text delay range (±0.6 s).
double TextPaddingWeight { get; set; } The weight of PAD and EPAD in the text loss (0.5) and of each speaker’s semantic codebook in the audio loss (α = 100, Eq.
double TextRateMultiplier { get; set; } The text embedding’s and text linear’s rate multiplier on audio batches (0.75).
double TextTemperature { get; set; }
double TtsPaddingBonus { get; set; }
double TtsPaddingTarget { get; set; } App.
double TtsTemperature { get; set; } The TTS sampling temperature (§5.7: 0.6); ASR decodes greedily.
int AcousticDelay { get; set; } The acoustic delay τ in steps (1; Table 1’s pre-training used 2).
int AudioTopK { get; set; }
int CodebookSize { get; set; }
int Codebooks { get; set; } Mimi codebooks per speaker, Q (8), and their size (2048).
int Context { get; set; }
int DepthDim { get; set; } The Depth Transformer’s width (1024), layers (6), heads (16) and feed-forward width before the gate (6144, Table 1’s MLP of 4096; the release’s 4224 gives 2816).
int DepthFeedForward { get; set; }
int DepthHeads { get; set; }
int DepthLayers { get; set; }
int FeedForward { get; set; }
int Layers { get; set; } The Temporal Transformer’s layers (32), feed-forward width before the gate (4.125 · 4096 = 16896, a gate of 11264) and context (3000 steps, 4 min).
int MimiCodebookDim { get; set; }
int MimiCodebooks { get; set; }
int MimiContext { get; set; }
int MimiDimension { get; set; }
int MimiDiscriminatorFilters { get; set; }
int MimiFeedForward { get; set; }
int MimiFilters { get; set; } Mimi’s SEANet width (64), latent width (512), Transformer layers (8) and codebooks it holds (32, the first Codebooks used).
int MimiHeads { get; set; }
int MimiTransformerLayers { get; set; }
int NumDecoderLayers { get; set; } The shared decoder depth maps to Mimi’s decoder Transformer, with the same depth as its encoder.
int NumEncoderLayers { get; set; } The shared encoder depth maps to Mimi’s encoder Transformer.
int TextEndPadding { get; set; }
int TextPadding { get; set; } The text stream’s PAD (3) and EPAD (0) tokens.
int TextTopK { get; set; }
int TotalSteps { get; set; } Steps of the pre-training cosine schedule (1M) and its linear warmup (not stated; 1000).
int WarmupSteps { get; set; }
int WavLmConvChannels { get; set; }
int WavLmHeads { get; set; }
int WavLmLayers { get; set; }
int WavLmWidth { get; set; } WavLM Large’s width (1024), layers (24), heads (16) and feature-encoder channels (512) for the distillation targets.
int[] MimiDiscriminatorWindows { get; set; } The multi-scale STFT discriminator’s windows (Audiocraft’s compression default: 1024, 2048, 512, 256, 128, hop a quarter) and filters (32).
int[] MimiRatios { get; set; } Mimi’s SEANet ratios as the release lists them (8, 6, 5, 4), its Transformers’ heads (8), feed-forward width (2048), context (250 frames) and quantizer width (256).
string TokenizerPath { get; set; } Helium’s SentencePiece model (tokenizer_spm_32k_3.model); text needs it.

Methods

Method Summary
MoshiOptions Release() The released kyutai/moshiko-pytorch-bf16: the Depth Transformer predicts Moshi’s streams only, behind a 2816-wide gate (feed-forward 4224).