Data2VecASR<T>
Models & Types · AiDotNet.SpeechRecognition.Foundation
Data2Vec ASR: general-purpose self-supervised learning
Inheritance: NeuralNetworkBase<T> → AudioNeuralNetworkBase<T> → Data2VecASR<T>
Implements: ISpeechRecognizer<T>
For Beginners
Data2Vec uses a shared self-supervised framework across speech, vision, and language. The teacher network produces contextualized target representations from unmasked input, while the student network predicts these targets from masked input. Unlik…
How It Works
References:
- Paper: “data2vec: A General Framework for Self-Supervised Learning in Speech, Vision and Language” (Baevski et al., Meta, 2022)
Data2Vec uses a shared self-supervised framework across speech, vision, and language. The teacher network produces contextualized target representations from unmasked input, while the student network predicts these targets from masked input. Unlike HuBERT’s discrete pseudo-labels, data2vec regresses continuous latent representations. For speech, it achieves competitive ASR performance while using a simpler training objective without quantization.
Example
// Create a Data2Vec ASR model for general self-supervised speech learning
var architecture = new NeuralNetworkArchitecture<double>(
inputType: InputType.OneDimensional,
taskType: NeuralNetworkTaskType.SpeechRecognition,
inputHeight: 16000, inputWidth: 1, inputDepth: 1, outputSize: 5000);
var model = new Data2VecASR<double>(architecture);
// Or load a pre-trained ONNX model for continuous representation prediction
var onnxModel = new Data2VecASR<double>(architecture, "data2vecasr.onnx");
Constructors
| Constructor | Summary |
|---|---|
Data2VecASR(NeuralNetworkArchitecture<T> architecture, Data2VecASROptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
|
Data2VecASR(NeuralNetworkArchitecture<T> architecture, string modelPath, Data2VecASROptions options) |
Properties
| Property | Summary |
|---|---|
IReadOnlyList<string> SupportedLanguages { get; } |
|
bool SupportsStreaming { get; } |
|
bool SupportsWordTimestamps { get; } |
Methods
| Method | Summary |
|---|---|
IReadOnlyDictionary<string, T> DetectLanguageProbabilities(Tensor<T> audio) |
|
IStreamingTranscriptionSession<T> StartStreamingSession(string language) |
|
ModelMetadata<T> GetModelMetadata() |
|
Task<TranscriptionResult<T>> TranscribeAsync(Tensor<T> audio, string language, bool? includeTimestamps, CancellationToken cancellationToken) |
|
Tensor<T> ForwardForTraining(Tensor<T> input) |
|
TranscriptionResult<T> Transcribe(Tensor<T> audio, string language, bool? includeTimestamps) |
Transcribes audio using data2vec’s continuous SSL encoder with CTC decoding. |
string DetectLanguage(Tensor<T> audio) |
|
void LoadTorchWeights(Func<string, int[], double[]> read) |
Loads a transformers *ForCTC state dict straight from source. |