Moshi<T>

Models & Types · AiDotNet.TextToSpeech.MultiModal

Moshi (Défossez et al. 2024): a speech-text foundation model for full-duplex dialogue. Every 80 ms it reads a frame of the user’s speech and writes a text token of its inner monologue and a frame of its own speech in Mimi’s codes.

Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → Moshi<T>

Implements: ICodecTts<T>

For Beginners

Moshi talks with you in real time: it listens and speaks at once, can be interrupted, and writes down what it is about to say a moment before saying it, which keeps its speech coherent.

How It Works

The paper’s model (the defaults): the RQ-Transformer (§3.4) — a 7B Temporal Transformer over the sum of the streams’ embeddings, a Depth Transformer predicting the text and then each codebook of each speaker in turn (Eq. 6) — over Mimi (§3.3), a streaming codec with a split RVQ whose first level is distilled from WavLM, and Helium’s tokenizer. Synthesize is App. C’s streaming TTS; Chat is the dialogue; Moshi is App. C’s streaming ASR. Training covers Mimi’s adversarial-only objective with distillation (TrainMimi) and the language model’s four stages (TrainConversation, TrainText). Release is kyutai/moshiko, whose weights LoadReleasedWeights reads.

Example

var architecture = new NeuralNetworkArchitecture<float>(InputType.OneDimensional, NeuralNetworkTaskType.Regression,
    inputHeight: 1, inputWidth: 1, inputDepth: 1, outputSize: 1);
var moshi = new Moshi<float>(architecture, MoshiOptions.Release());
moshi.LoadReleasedWeights("kyutai/moshiko-pytorch-bf16");
var (reply, monologue) = moshi.Chat(userSpeech24kHz);

Constructors

Constructor Summary
Moshi(NeuralNetworkArchitecture<T> architecture) Calls the fully specified overload with the omitted parameters’ defaults.
Moshi(NeuralNetworkArchitecture<T> architecture, MoshiOptions options) Creates Moshi with its networks (randomly initialized as the release initializes them).
Moshi(NeuralNetworkArchitecture<T> architecture, string modelPath, MoshiOptions options) Creates Moshi running an ONNX export.

Properties

Property Summary
IReadOnlyList<Tensor<T>> AugmentationNoise { get; set; } Noise clips (24 kHz) for the instruct stage’s user-stream augmentation (§4.4 uses the Deep Noise Suppression challenge’s); without them that augmentation adds no noise.
bool TrainsOnSampleObjective { get; }
double MaxGradNormValue { get; }
int CodebookSize { get; }
int CodecFrameRate { get; }
int CodecTokenCodebooks { get; }
int CodecTokenVocabulary { get; }
int MaxTextLength { get; } The longest text the TTS takes, in tokens.
int NumCodebooks { get; }

Methods

Method Summary
LayerInputDomain GetInputDomain(int[] inputShape)
ModelMetadata<T> GetModelMetadata()
T EvaluateTrainingObjective(TtsTrainingSample<T> sample)
T TrainConversation(MoshiConversation<T> conversation) One language-model step on a conversation at the current stage (Eq.
T TrainMimi(Tensor<T> audio) One step of Mimi’s training (§3.3) on 24 kHz speech (the paper’s random 12 s windows): the codebooks’ moving averages and the reconstruction from a gradient-free pass, the discriminator’s hinge step on it, then the codec’s step — the balanc…
T TrainText(string text) One step on a text-only batch (§4.4: half of pre-training’s batches, 10% of post-training’s), the text in Helium’s tokens, with its own optimizer state.
Tensor<T> DecodeFromTokens(Tensor<T> tokens)
Tensor<T> EncodeToTokens(Tensor<T> audio)
Tensor<T> Synthesize(string text)
ValueTuple<Tensor<T>, IReadOnlyList<MoshiTimedWord>> Speak(string text) App.
ValueTuple<Tensor<T>, string> Chat(Tensor<T> userSpeech) The dialogue: Moshi listens to the user’s 24 kHz userSpeech frame by frame and answers as it goes.
int[] TokenizeText(string text) Helium’s token ids of text (no ).
void ApplyMimiWeightAverage() Replaces Mimi’s weights by their moving average over the training steps (§3.3’s EMA at MimiWeightAverageDecay), as the trained codec is used.
void LoadLanguageModel(string path) Loads the release’s LMModel state dict (a .safetensors file).
void LoadMimi(string path) Loads Mimi’s state dict (the release’s MimiModel names, a .safetensors file).
void LoadReleasedWeights(string directory) Loads the release (kyutai/moshiko-pytorch-bf16 or moshika): the language model from model.safetensors, Mimi from tokenizer-e351c8d8-checkpoint125.safetensors and Helium’s tokenizer tokenizer_spm_32k_3.model when TokenizerPath is u…
void LoadWavLm(string path) Loads WavLM Large for Mimi’s distillation (transformers’ microsoft/wavlm-large: a directory with pytorch_model.bin or model.safetensors, or the file).