Moshi<T>

Models & Types · AiDotNet.SpeechRecognition.Streaming

Moshi’s streaming speech recognizer (Défossez et al. 2024 App. C): the same RQ-Transformer and Mimi as Moshi, trained with the text stream 2 s behind the audio, so that at inference the recording’s codes are forced into the audio streams and the text is decoded greedily, with each word’s time.

Inheritance: NeuralNetworkBase<T> → AudioNeuralNetworkBase<T> → Moshi<T>

Implements: ISpeechRecognizer<T>

For Beginners

This listens to speech and writes it down as it goes, about two seconds behind.

How It Works

The options default to Asr. Training pairs a recording with its transcript’s Helium ids; without word times, every word is taken to start at the beginning, so the transcript runs from the first frame after the delay (the release interleaver’s keep-and-shift for zero timestamps). TrainConversation takes word times, as the paper’s Whisper alignment gives them.

Constructors

Constructor Summary
Moshi(NeuralNetworkArchitecture<T> architecture, MoshiOptions options) Creates the recognizer (randomly initialized as the release initializes it).

Properties

Property Summary
IReadOnlyList<string> SupportedLanguages { get; }
TrainingObjectiveKind TrainingObjectiveKind { get; }
bool SupportsStreaming { get; }
bool SupportsWordTimestamps { get; }
double MaxGradNormValue { get; }

Methods

Method Summary
IReadOnlyDictionary<string, T> DetectLanguageProbabilities(Tensor<T> audio)
IStreamingTranscriptionSession<T> StartStreamingSession(string language)
LayerInputDomain GetOutputDomain(int[] outputShape)
ModelMetadata<T> GetModelMetadata()
T EvaluateTrainingObjective(Tensor<T> input, Tensor<T> target)
T TrainConversation(MoshiConversation<T> conversation) One step on a recording with its words and times (the main stream; User is not read).
Task<TranscriptionResult<T>> TranscribeAsync(Tensor<T> audio, string language, bool? includeTimestamps, CancellationToken cancellationToken)
Tensor<T> ResolveTrainingTarget(Tensor<T> input, Tensor<T> proposedTarget)
TranscriptionResult<T> Transcribe(Tensor<T> audio, string language, bool? includeTimestamps)
int[] TokenizeText(string text) Helium’s token ids of text (no ).
string DetectLanguage(Tensor<T> audio)
void LoadReleasedWeights(string directory) Loads released weights: model.safetensors (the language model), Mimi’s tokenizer-*.safetensors and Helium’s tokenizer.