Moshi<T>
Models & Types · AiDotNet.TextToSpeech.MultiModal
Moshi (Défossez et al. 2024): a speech-text foundation model for full-duplex dialogue. Every 80 ms it reads a frame of the user’s speech and writes a text token of its inner monologue and a frame of its own speech in Mimi’s codes.
Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → Moshi<T>
Implements: ICodecTts<T>
For Beginners
Moshi talks with you in real time: it listens and speaks at once, can be interrupted, and writes down what it is about to say a moment before saying it, which keeps its speech coherent.
How It Works
The paper’s model (the defaults): the RQ-Transformer (§3.4) — a 7B Temporal Transformer over the sum of the
streams’ embeddings, a Depth Transformer predicting the text and then each codebook of each speaker in turn (Eq. 6) —
over Mimi (§3.3), a streaming codec with a split RVQ whose first level is distilled from WavLM, and Helium’s
tokenizer. Synthesize is App. C’s streaming TTS; Chat is the dialogue;
Moshi is App. C’s streaming ASR. Training covers Mimi’s
adversarial-only objective with distillation (TrainMimi) and the language model’s four stages
(TrainConversation, TrainText). Release is kyutai/moshiko, whose
weights LoadReleasedWeights reads.
Example
var architecture = new NeuralNetworkArchitecture<float>(InputType.OneDimensional, NeuralNetworkTaskType.Regression,
inputHeight: 1, inputWidth: 1, inputDepth: 1, outputSize: 1);
var moshi = new Moshi<float>(architecture, MoshiOptions.Release());
moshi.LoadReleasedWeights("kyutai/moshiko-pytorch-bf16");
var (reply, monologue) = moshi.Chat(userSpeech24kHz);
Constructors
| Constructor | Summary |
|---|---|
Moshi(NeuralNetworkArchitecture<T> architecture) |
Calls the fully specified overload with the omitted parameters’ defaults. |
Moshi(NeuralNetworkArchitecture<T> architecture, MoshiOptions options) |
Creates Moshi with its networks (randomly initialized as the release initializes them). |
Moshi(NeuralNetworkArchitecture<T> architecture, string modelPath, MoshiOptions options) |
Creates Moshi running an ONNX export. |
Properties
| Property | Summary |
|---|---|
IReadOnlyList<Tensor<T>> AugmentationNoise { get; set; } |
Noise clips (24 kHz) for the instruct stage’s user-stream augmentation (§4.4 uses the Deep Noise Suppression challenge’s); without them that augmentation adds no noise. |
bool TrainsOnSampleObjective { get; } |
|
double MaxGradNormValue { get; } |
|
int CodebookSize { get; } |
|
int CodecFrameRate { get; } |
|
int CodecTokenCodebooks { get; } |
|
int CodecTokenVocabulary { get; } |
|
int MaxTextLength { get; } |
The longest text the TTS takes, in tokens. |
int NumCodebooks { get; } |
Methods
| Method | Summary |
|---|---|
LayerInputDomain GetInputDomain(int[] inputShape) |
|
ModelMetadata<T> GetModelMetadata() |
|
T EvaluateTrainingObjective(TtsTrainingSample<T> sample) |
|
T TrainConversation(MoshiConversation<T> conversation) |
One language-model step on a conversation at the current stage (Eq. |
T TrainMimi(Tensor<T> audio) |
One step of Mimi’s training (§3.3) on 24 kHz speech (the paper’s random 12 s windows): the codebooks’ moving averages and the reconstruction from a gradient-free pass, the discriminator’s hinge step on it, then the codec’s step — the balanc… |
T TrainText(string text) |
One step on a text-only batch (§4.4: half of pre-training’s batches, 10% of post-training’s), the text in Helium’s tokens, with its own optimizer state. |
Tensor<T> DecodeFromTokens(Tensor<T> tokens) |
|
Tensor<T> EncodeToTokens(Tensor<T> audio) |
|
Tensor<T> Synthesize(string text) |
|
ValueTuple<Tensor<T>, IReadOnlyList<MoshiTimedWord>> Speak(string text) |
App. |
ValueTuple<Tensor<T>, string> Chat(Tensor<T> userSpeech) |
The dialogue: Moshi listens to the user’s 24 kHz userSpeech frame by frame and answers as it goes. |
int[] TokenizeText(string text) |
Helium’s token ids of text (no |
void ApplyMimiWeightAverage() |
Replaces Mimi’s weights by their moving average over the training steps (§3.3’s EMA at MimiWeightAverageDecay), as the trained codec is used. |
void LoadLanguageModel(string path) |
Loads the release’s LMModel state dict (a .safetensors file). |
void LoadMimi(string path) |
Loads Mimi’s state dict (the release’s MimiModel names, a .safetensors file). |
void LoadReleasedWeights(string directory) |
Loads the release (kyutai/moshiko-pytorch-bf16 or moshika): the language model from model.safetensors, Mimi from tokenizer-e351c8d8-checkpoint125.safetensors and Helium’s tokenizer tokenizer_spm_32k_3.model when TokenizerPath is u… |
void LoadWavLm(string path) |
Loads WavLM Large for Mimi’s distillation (transformers’ microsoft/wavlm-large: a directory with pytorch_model.bin or model.safetensors, or the file). |