Wav2Vec2Model<T>

Models & Types · AiDotNet.Audio.SpeechRecognition

Wav2Vec2 self-supervised speech recognition model.

Inheritance: NeuralNetworkBase<T> → AudioNeuralNetworkBase<T> → Wav2Vec2Model<T>

Implements: ISpeechRecognizer<T>

For Beginners

Wav2Vec2 works differently from traditional speech recognition:

  1. It processes raw audio directly (no mel spectrograms needed)
  2. It learns speech patterns from unlabeled audio data
  3. It can be fine-tuned with small amounts of labeled data

Architecture:

  • Convolutional feature encoder: Processes raw audio into features
  • Transformer encoder: Captures long-range dependencies in speech
  • CTC head: Aligns speech to text (Connectionist Temporal Classification)

Two ways to use this class:

  1. ONNX Mode: Load pretrained Wav2Vec2 models for fast inference
  2. Native Mode: Train your own speech recognition model from scratch

ONNX Mode Example:

Training Mode Example:

How It Works

Wav2Vec2 is a self-supervised learning model for speech recognition developed by Meta AI. It learns representations from raw audio through contrastive learning, then can be fine-tuned for speech recognition tasks.

Constructors

Constructor Summary
Wav2Vec2Model(NeuralNetworkArchitecture<T> architecture, Wav2Vec2ModelOptions options, string[] vocabulary, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer, ILossFunction<T> lossFunction) Creates a Wav2Vec2 network for training from scratch using native layers.
Wav2Vec2Model(NeuralNetworkArchitecture<T> architecture, string modelPath, Wav2Vec2ModelOptions options, string[] vocabulary, OnnxModelOptions onnxOptions) Creates a Wav2Vec2 network for training from scratch using native layers.

Properties

Property Summary
IReadOnlyList<string> SupportedLanguages { get; } Gets the list of languages supported by this model.
bool IsReady { get; } Gets whether the model is ready for inference.
bool SupportsStreaming { get; } Gets whether this model supports real-time streaming transcription.
bool SupportsWordTimestamps { get; } Gets whether this model can identify timestamps for each word.
int MaxAudioLengthSeconds { get; } Gets the maximum audio length in seconds.
string Language { get; } Gets the target language for transcription.

Methods

Method Summary
Dictionary<string, Tensor<T>> GetNamedLayerActivations(Tensor<T> input)
IReadOnlyDictionary<string, T> DetectLanguageProbabilities(Tensor<T> audio) Gets language detection probabilities for the audio.
IStreamingTranscriptionSession<T> StartStreamingSession(string language) Starts a streaming transcription session.
ModelMetadata<T> GetModelMetadata() Gets metadata about the model.
Task<TranscriptionResult<T>> TranscribeAsync(Tensor<T> audio, string language, bool? includeTimestamps, CancellationToken cancellationToken) Transcribes audio to text asynchronously.
Tensor<T> ForwardForTraining(Tensor<T> input)
TranscriptionResult<T> Transcribe(Tensor<T> audio, string language, bool? includeTimestamps) Transcribes audio to text.
string DetectLanguage(Tensor<T> audio) Detects the language spoken in the audio.
void LoadTorchWeights(Func<string, int[], double[]> read) Loads a transformers Wav2Vec2ForCTC state dict straight from source.