Data2Vec2<T>

Models & Types · AiDotNet.Audio.Foundations

data2vec 2.0 self-supervised audio representation model.

Inheritance: NeuralNetworkBase<T> → AudioNeuralNetworkBase<T> → Data2Vec2<T>

Implements: IAudioFoundationModel<T>

For Beginners

data2vec 2.0 learns audio features by predicting its own hidden representations - like learning by explaining things to yourself. Unlike HuBERT which predicts discrete labels, data2vec predicts rich continuous features. It can be used as a powerful feature extractor for any downstream audio task.

Usage:

How It Works

data2vec 2.0 (Baevski et al., 2023, Meta) is a self-supervised framework that predicts contextualized latent representations rather than modality-specific targets. Version 2.0 is 16x faster than v1 through efficient data encoding and self-distillation. It achieves strong results on speech, vision, and language tasks with the same method.

Constructors

Constructor Summary
Data2Vec2(NeuralNetworkArchitecture<T> architecture, Data2Vec2Options options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) Creates a data2vec 2.0 model in native training mode.
Data2Vec2(NeuralNetworkArchitecture<T> architecture, string modelPath, Data2Vec2Options options) Creates a data2vec 2.0 model in native training mode.

Properties

Property Summary
int EmbeddingDimension { get; }
int NumLayers { get; }

Methods

Method Summary
ModelMetadata<T> GetModelMetadata()
Task<Tensor<T>> ExtractEmbeddingsAsync(Tensor<T> audio, CancellationToken cancellationToken)
Tensor<T> ExtractEmbeddings(Tensor<T> audio)
Tensor<T> ExtractLayerFeatures(Tensor<T> audio, int layerIndex)
Tensor<T> ExtractWeightedFeatures(Tensor<T> audio, T[] layerWeights)