Wav2Vec2Model<T>
Models & Types · AiDotNet.Audio.SpeechRecognition
Wav2Vec2 self-supervised speech recognition model.
Inheritance: NeuralNetworkBase<T> → AudioNeuralNetworkBase<T> → Wav2Vec2Model<T>
Implements: ISpeechRecognizer<T>
For Beginners
Wav2Vec2 works differently from traditional speech recognition:
- It processes raw audio directly (no mel spectrograms needed)
- It learns speech patterns from unlabeled audio data
- It can be fine-tuned with small amounts of labeled data
Architecture:
- Convolutional feature encoder: Processes raw audio into features
- Transformer encoder: Captures long-range dependencies in speech
- CTC head: Aligns speech to text (Connectionist Temporal Classification)
Two ways to use this class:
- ONNX Mode: Load pretrained Wav2Vec2 models for fast inference
- Native Mode: Train your own speech recognition model from scratch
ONNX Mode Example:
Training Mode Example:
How It Works
Wav2Vec2 is a self-supervised learning model for speech recognition developed by Meta AI. It learns representations from raw audio through contrastive learning, then can be fine-tuned for speech recognition tasks.
Constructors
| Constructor | Summary |
|---|---|
Wav2Vec2Model(NeuralNetworkArchitecture<T> architecture, Wav2Vec2ModelOptions options, string[] vocabulary, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer, ILossFunction<T> lossFunction) |
Creates a Wav2Vec2 network for training from scratch using native layers. |
Wav2Vec2Model(NeuralNetworkArchitecture<T> architecture, string modelPath, Wav2Vec2ModelOptions options, string[] vocabulary, OnnxModelOptions onnxOptions) |
Creates a Wav2Vec2 network for training from scratch using native layers. |
Properties
| Property | Summary |
|---|---|
IReadOnlyList<string> SupportedLanguages { get; } |
Gets the list of languages supported by this model. |
bool IsReady { get; } |
Gets whether the model is ready for inference. |
bool SupportsStreaming { get; } |
Gets whether this model supports real-time streaming transcription. |
bool SupportsWordTimestamps { get; } |
Gets whether this model can identify timestamps for each word. |
int MaxAudioLengthSeconds { get; } |
Gets the maximum audio length in seconds. |
string Language { get; } |
Gets the target language for transcription. |
Methods
| Method | Summary |
|---|---|
Dictionary<string, Tensor<T>> GetNamedLayerActivations(Tensor<T> input) |
|
IReadOnlyDictionary<string, T> DetectLanguageProbabilities(Tensor<T> audio) |
Gets language detection probabilities for the audio. |
IStreamingTranscriptionSession<T> StartStreamingSession(string language) |
Starts a streaming transcription session. |
ModelMetadata<T> GetModelMetadata() |
Gets metadata about the model. |
Task<TranscriptionResult<T>> TranscribeAsync(Tensor<T> audio, string language, bool? includeTimestamps, CancellationToken cancellationToken) |
Transcribes audio to text asynchronously. |
Tensor<T> ForwardForTraining(Tensor<T> input) |
|
TranscriptionResult<T> Transcribe(Tensor<T> audio, string language, bool? includeTimestamps) |
Transcribes audio to text. |
string DetectLanguage(Tensor<T> audio) |
Detects the language spoken in the audio. |
void LoadTorchWeights(Func<string, int[], double[]> read) |
Loads a transformers Wav2Vec2ForCTC state dict straight from source. |