Data2VecASR<T>

Models & Types · AiDotNet.SpeechRecognition.Foundation

Data2Vec ASR: general-purpose self-supervised learning

Inheritance: NeuralNetworkBase<T> → AudioNeuralNetworkBase<T> → Data2VecASR<T>

Implements: ISpeechRecognizer<T>

For Beginners

Data2Vec uses a shared self-supervised framework across speech, vision, and language. The teacher network produces contextualized target representations from unmasked input, while the student network predicts these targets from masked input. Unlik…

How It Works

References:

  • Paper: “data2vec: A General Framework for Self-Supervised Learning in Speech, Vision and Language” (Baevski et al., Meta, 2022)

Data2Vec uses a shared self-supervised framework across speech, vision, and language. The teacher network produces contextualized target representations from unmasked input, while the student network predicts these targets from masked input. Unlike HuBERT’s discrete pseudo-labels, data2vec regresses continuous latent representations. For speech, it achieves competitive ASR performance while using a simpler training objective without quantization.

Example

// Create a Data2Vec ASR model for general self-supervised speech learning
var architecture = new NeuralNetworkArchitecture<double>(
    inputType: InputType.OneDimensional,
    taskType: NeuralNetworkTaskType.SpeechRecognition,
    inputHeight: 16000, inputWidth: 1, inputDepth: 1, outputSize: 5000);
var model = new Data2VecASR<double>(architecture);
            
// Or load a pre-trained ONNX model for continuous representation prediction
var onnxModel = new Data2VecASR<double>(architecture, "data2vecasr.onnx");

Constructors

Constructor Summary
Data2VecASR(NeuralNetworkArchitecture<T> architecture, Data2VecASROptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer)
Data2VecASR(NeuralNetworkArchitecture<T> architecture, string modelPath, Data2VecASROptions options)

Properties

Property Summary
IReadOnlyList<string> SupportedLanguages { get; }
bool SupportsStreaming { get; }
bool SupportsWordTimestamps { get; }

Methods

Method Summary
IReadOnlyDictionary<string, T> DetectLanguageProbabilities(Tensor<T> audio)
IStreamingTranscriptionSession<T> StartStreamingSession(string language)
ModelMetadata<T> GetModelMetadata()
Task<TranscriptionResult<T>> TranscribeAsync(Tensor<T> audio, string language, bool? includeTimestamps, CancellationToken cancellationToken)
Tensor<T> ForwardForTraining(Tensor<T> input)
TranscriptionResult<T> Transcribe(Tensor<T> audio, string language, bool? includeTimestamps) Transcribes audio using data2vec’s continuous SSL encoder with CTC decoding.
string DetectLanguage(Tensor<T> audio)
void LoadTorchWeights(Func<string, int[], double[]> read) Loads a transformers *ForCTC state dict straight from source.