UniSpeech<T>
Models & Types · AiDotNet.SpeechRecognition.Foundation
UniSpeech: unified pre-training for speech representation learning
Inheritance: NeuralNetworkBase<T> → AudioNeuralNetworkBase<T> → UniSpeech<T>
Implements: ISpeechRecognizer<T>
For Beginners
UniSpeech unifies self-supervised and supervised pre-training in a single framework. The model jointly optimizes a contrastive self-supervised loss and a CTC loss when labeled data is available. This multi-task pre-training allows the model to lev…
How It Works
References:
- Paper: “UniSpeech: Unified Pre-training for Self-Supervised Learning and Supervised Learning for ASR” (Wang et al., Microsoft, 2021)
UniSpeech unifies self-supervised and supervised pre-training in a single framework. The model jointly optimizes a contrastive self-supervised loss and a CTC loss when labeled data is available. This multi-task pre-training allows the model to leverage both unlabeled and labeled speech data during pre-training, producing representations that are more aligned with the downstream ASR task than purely self-supervised approaches.
Example
// Create a UniSpeech model for unified self-supervised and supervised pre-training
var architecture = new NeuralNetworkArchitecture<double>(
inputType: InputType.OneDimensional,
taskType: NeuralNetworkTaskType.SpeechRecognition,
inputHeight: 16000, inputWidth: 1, inputDepth: 1, outputSize: 5000);
var model = new UniSpeech<double>(architecture);
// Or load a pre-trained ONNX model for multi-task speech recognition
var onnxModel = new UniSpeech<double>(architecture, "unispeech.onnx");
Constructors
| Constructor | Summary |
|---|---|
UniSpeech(NeuralNetworkArchitecture<T> architecture, UniSpeechOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
|
UniSpeech(NeuralNetworkArchitecture<T> architecture, string modelPath, UniSpeechOptions options) |
Properties
| Property | Summary |
|---|---|
IReadOnlyList<string> SupportedLanguages { get; } |
|
bool SupportsStreaming { get; } |
|
bool SupportsWordTimestamps { get; } |
Methods
| Method | Summary |
|---|---|
IReadOnlyDictionary<string, T> DetectLanguageProbabilities(Tensor<T> audio) |
|
IStreamingTranscriptionSession<T> StartStreamingSession(string language) |
|
ModelMetadata<T> GetModelMetadata() |
|
Task<TranscriptionResult<T>> TranscribeAsync(Tensor<T> audio, string language, bool? includeTimestamps, CancellationToken cancellationToken) |
|
Tensor<T> ForwardForTraining(Tensor<T> input) |
|
TranscriptionResult<T> Transcribe(Tensor<T> audio, string language, bool? includeTimestamps) |
Transcribes audio using UniSpeech’s unified SSL/supervised encoder with CTC. |
string DetectLanguage(Tensor<T> audio) |
|
void LoadTorchWeights(Func<string, int[], double[]> read) |
Loads a transformers *ForCTC state dict straight from source. |