MinMo<T>
Models & Types · AiDotNet.TextToSpeech.MultiModal
MinMo (Chen et al. 2025): a multimodal LLM for voice interaction. A voice encoder and projector feed speech into Qwen2.5-7B-Instruct, whose answer a streaming voice decoder (CosyVoice 2’s language model and token2wav) speaks five text tokens at a time, and a full duplex predictor decides when to answer and when to stop.
Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → MinMo<T>
For Beginners
MinMo is a voice assistant made from a text chatbot: it listens, writes an answer, and reads the answer aloud while it is still writing it.
How It Works
There is no official code; the same team’s FunASR LLM-ASR and CosyVoice 2 fill what the paper leaves out (see
MinMoOptions). Speech in and out is at SampleRate (24 kHz); the encoder reads it
resampled to 16 kHz.
Tasks: Synthesize is LLM-driven text-to-speech (the instruction and the text go through the LLM,
whose tokens and hidden states drive the voice decoder); Understand answers an instruction about a
recording (recognition, translation, emotion, …); Chat answers speech with text and speech;
DuplexActions runs the full duplex predictor. Training follows §3.4’s stages through
Stage.
Constructors
| Constructor | Summary |
|---|---|
MinMo(NeuralNetworkArchitecture<T> architecture) |
Calls the fully specified overload with the omitted parameters’ defaults. |
MinMo(NeuralNetworkArchitecture<T> architecture, MinMoOptions options) |
Creates MinMo with its networks (randomly initialized). |
MinMo(NeuralNetworkArchitecture<T> architecture, string modelPath, MinMoOptions options) |
Creates MinMo running an ONNX export. |
Properties
| Property | Summary |
|---|---|
TtsVoice<T> SpeakerVoice { get; set; } |
The voice speech is spoken in (CosyVoice 2’s: a reference recording’s speaker embedding, or a zero-shot prompt; see CosyVoice 2’s CreateVoice). |
bool TrainsOnSampleObjective { get; } |
|
int MaxTextLength { get; } |
The longest answer, in tokens. |
Methods
| Method | Summary |
|---|---|
LayerInputDomain GetInputDomain(int[] inputShape) |
|
MinMoDuplexAction[] DuplexActions(Tensor<T> speech) |
The full duplex predictor’s action at each speech position of a recording of the user (the input projector’s rate: 25 a second with SenseVoice-Large). |
ModelMetadata<T> GetModelMetadata() |
|
T EvaluateTrainingObjective(TtsTrainingSample<T> sample) |
|
T TrainDuplex(Tensor<T> speech, IReadOnlyList<int> labels) |
One duplex step (§3.4 stage 4): the full duplex predictor learns an action per speech position of a conversation recording (−1 unlabelled), labelled by §3.3’s rules — the assistant’s turn starts where the user’s ends (Respond), the user t… |
T TrainSpeechToSpeech(Tensor<T> speech, string answer, Tensor<T> answerRecording, string instruction) |
One speech-to-speech step (§3.4 stage 3): the user’s recording as the prompt, the answer’s text and its recording; the output projector and the voice token LM learn. |
T TrainSpeechToText(Tensor<T> speech, string instruction, string answer) |
One speech-to-text step (§3.4 stage 1): the ChatML prompt with the instruction and the recording, the answer’s cross-entropy. |
T TrainTextToSpeech(string text, Tensor<T> recording, string instruction, string answer) |
One text-to-speech step (§3.4 stage 2): the user asks “{instruction}: {text}”, the assistant’s answer (answer, the text itself by default, or Table 3’s tagged form) is spoken by the recording; the output projector, and at TextToSpeech a… |
Tensor<T> Synthesize(string text) |
|
TtsVoice<T> CreateVoice(Tensor<T> recording, string transcript) |
A voice from a reference recording, made by the voice decoder. |
ValueTuple<Tensor<T>, string> Chat(Tensor<T> speech, string instruction) |
A spoken answer to a spoken question (speech-to-speech): the answer’s text and its speech. |
ValueTuple<Tensor<T>, string> Speak(string text, string instruction) |
Speaks text the way MinMo’s text-to-speech alignment trains it (Table 3): the user asks “{instruction}: {text}”, the LLM answers, and the voice decoder speaks the answer from its tokens and hidden states. |
int SpeechPositions(Tensor<T> speech) |
The number of speech positions (duplex labels) a recording gives. |
int[] TokenizeText(string text) |
Qwen2.5’s token ids of text. |
string Transcribe(Tensor<T> speech) |
The recording’s transcript (TranscriptionInstruction). |
string Understand(Tensor<T> speech, string instruction) |
The LLM’s answer to an instruction about a recording (speech-to-text: “Speech Transcription”, “Translate {source} into {target}”, an emotion or event question, …). |