Moshi<T>
Models & Types · AiDotNet.SpeechRecognition.Streaming
Moshi’s streaming speech recognizer (Défossez et al. 2024 App. C): the same RQ-Transformer and Mimi as
Moshi, trained with the text stream 2 s behind the audio, so that at
inference the recording’s codes are forced into the audio streams and the text is decoded greedily, with each word’s
time.
Inheritance: NeuralNetworkBase<T> → AudioNeuralNetworkBase<T> → Moshi<T>
Implements: ISpeechRecognizer<T>
For Beginners
This listens to speech and writes it down as it goes, about two seconds behind.
How It Works
The options default to Asr. Training pairs a recording with its transcript’s Helium
ids; without word times, every word is taken to start at the beginning, so the transcript runs from the first frame
after the delay (the release interleaver’s keep-and-shift for zero timestamps). TrainConversation takes
word times, as the paper’s Whisper alignment gives them.
Constructors
| Constructor | Summary |
|---|---|
Moshi(NeuralNetworkArchitecture<T> architecture, MoshiOptions options) |
Creates the recognizer (randomly initialized as the release initializes it). |
Properties
| Property | Summary |
|---|---|
IReadOnlyList<string> SupportedLanguages { get; } |
|
TrainingObjectiveKind TrainingObjectiveKind { get; } |
|
bool SupportsStreaming { get; } |
|
bool SupportsWordTimestamps { get; } |
|
double MaxGradNormValue { get; } |
Methods
| Method | Summary |
|---|---|
IReadOnlyDictionary<string, T> DetectLanguageProbabilities(Tensor<T> audio) |
|
IStreamingTranscriptionSession<T> StartStreamingSession(string language) |
|
LayerInputDomain GetOutputDomain(int[] outputShape) |
|
ModelMetadata<T> GetModelMetadata() |
|
T EvaluateTrainingObjective(Tensor<T> input, Tensor<T> target) |
|
T TrainConversation(MoshiConversation<T> conversation) |
One step on a recording with its words and times (the main stream; User is not read). |
Task<TranscriptionResult<T>> TranscribeAsync(Tensor<T> audio, string language, bool? includeTimestamps, CancellationToken cancellationToken) |
|
Tensor<T> ResolveTrainingTarget(Tensor<T> input, Tensor<T> proposedTarget) |
|
TranscriptionResult<T> Transcribe(Tensor<T> audio, string language, bool? includeTimestamps) |
|
int[] TokenizeText(string text) |
Helium’s token ids of text (no |
string DetectLanguage(Tensor<T> audio) |
|
void LoadReleasedWeights(string directory) |
Loads released weights: model.safetensors (the language model), Mimi’s tokenizer-*.safetensors and Helium’s tokenizer. |