MinMo<T>

Models & Types · AiDotNet.TextToSpeech.MultiModal

MinMo (Chen et al. 2025): a multimodal LLM for voice interaction. A voice encoder and projector feed speech into Qwen2.5-7B-Instruct, whose answer a streaming voice decoder (CosyVoice 2’s language model and token2wav) speaks five text tokens at a time, and a full duplex predictor decides when to answer and when to stop.

Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → MinMo<T>

For Beginners

MinMo is a voice assistant made from a text chatbot: it listens, writes an answer, and reads the answer aloud while it is still writing it.

How It Works

There is no official code; the same team’s FunASR LLM-ASR and CosyVoice 2 fill what the paper leaves out (see MinMoOptions). Speech in and out is at SampleRate (24 kHz); the encoder reads it resampled to 16 kHz.

Tasks: Synthesize is LLM-driven text-to-speech (the instruction and the text go through the LLM, whose tokens and hidden states drive the voice decoder); Understand answers an instruction about a recording (recognition, translation, emotion, …); Chat answers speech with text and speech; DuplexActions runs the full duplex predictor. Training follows §3.4’s stages through Stage.

Constructors

Constructor Summary
MinMo(NeuralNetworkArchitecture<T> architecture) Calls the fully specified overload with the omitted parameters’ defaults.
MinMo(NeuralNetworkArchitecture<T> architecture, MinMoOptions options) Creates MinMo with its networks (randomly initialized).
MinMo(NeuralNetworkArchitecture<T> architecture, string modelPath, MinMoOptions options) Creates MinMo running an ONNX export.

Properties

Property Summary
TtsVoice<T> SpeakerVoice { get; set; } The voice speech is spoken in (CosyVoice 2’s: a reference recording’s speaker embedding, or a zero-shot prompt; see CosyVoice 2’s CreateVoice).
bool TrainsOnSampleObjective { get; }
int MaxTextLength { get; } The longest answer, in tokens.

Methods

Method Summary
LayerInputDomain GetInputDomain(int[] inputShape)
MinMoDuplexAction[] DuplexActions(Tensor<T> speech) The full duplex predictor’s action at each speech position of a recording of the user (the input projector’s rate: 25 a second with SenseVoice-Large).
ModelMetadata<T> GetModelMetadata()
T EvaluateTrainingObjective(TtsTrainingSample<T> sample)
T TrainDuplex(Tensor<T> speech, IReadOnlyList<int> labels) One duplex step (§3.4 stage 4): the full duplex predictor learns an action per speech position of a conversation recording (−1 unlabelled), labelled by §3.3’s rules — the assistant’s turn starts where the user’s ends (Respond), the user t…
T TrainSpeechToSpeech(Tensor<T> speech, string answer, Tensor<T> answerRecording, string instruction) One speech-to-speech step (§3.4 stage 3): the user’s recording as the prompt, the answer’s text and its recording; the output projector and the voice token LM learn.
T TrainSpeechToText(Tensor<T> speech, string instruction, string answer) One speech-to-text step (§3.4 stage 1): the ChatML prompt with the instruction and the recording, the answer’s cross-entropy.
T TrainTextToSpeech(string text, Tensor<T> recording, string instruction, string answer) One text-to-speech step (§3.4 stage 2): the user asks “{instruction}: {text}”, the assistant’s answer (answer, the text itself by default, or Table 3’s tagged form) is spoken by the recording; the output projector, and at TextToSpeech a…
Tensor<T> Synthesize(string text)
TtsVoice<T> CreateVoice(Tensor<T> recording, string transcript) A voice from a reference recording, made by the voice decoder.
ValueTuple<Tensor<T>, string> Chat(Tensor<T> speech, string instruction) A spoken answer to a spoken question (speech-to-speech): the answer’s text and its speech.
ValueTuple<Tensor<T>, string> Speak(string text, string instruction) Speaks text the way MinMo’s text-to-speech alignment trains it (Table 3): the user asks “{instruction}: {text}”, the LLM answers, and the voice decoder speaks the answer from its tokens and hidden states.
int SpeechPositions(Tensor<T> speech) The number of speech positions (duplex labels) a recording gives.
int[] TokenizeText(string text) Qwen2.5’s token ids of text.
string Transcribe(Tensor<T> speech) The recording’s transcript (TranscriptionInstruction).
string Understand(Tensor<T> speech, string instruction) The LLM’s answer to an instruction about a recording (speech-to-text: “Speech Transcription”, “Translate {source} into {target}”, an emotion or event question, …).