Wav2Vec2LanguageIdentifier<T>

Models & Types · AiDotNet.Audio.LanguageIdentification

Wav2Vec2 model fine-tuned for spoken language identification.

Inheritance: NeuralNetworkBase<T> → AudioNeuralNetworkBase<T> → Wav2Vec2LanguageIdentifier<T>

Implements: ILanguageIdentifier<T>

For Beginners

Wav2Vec2 is like a very attentive listener that:

  1. First breaks down the raw sound wave into small pieces (feature encoder)
  2. Then looks at how all these pieces relate to each other (transformer)
  3. Finally makes a decision about what language is being spoken (classifier)

Key advantages:

  • Works directly on raw audio (no need for handcrafted features like MFCCs)
  • Pre-trained on massive amounts of unlabeled speech data
  • Can recognize languages even with limited labeled training data

Example usage:

How It Works

Wav2Vec2 is Meta’s self-supervised speech representation learning model that learns powerful representations directly from raw audio waveforms. When fine-tuned for language identification, it achieves state-of-the-art performance on many benchmarks.

Architecture overview:

  • Feature Encoder: 7 temporal convolution layers that process raw waveform
  • Transformer Encoder: 12-24 transformer blocks for contextual representations
  • Classification Head: Linear projection to language classes

Constructors

Constructor Summary
Wav2Vec2LanguageIdentifier(NeuralNetworkArchitecture<T> architecture, IReadOnlyList<string> supportedLanguages, Wav2Vec2LidOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer, ILossFunction<T> lossFunction) Creates a Wav2Vec2 language identifier for native training.
Wav2Vec2LanguageIdentifier(NeuralNetworkArchitecture<T> architecture, string modelPath, Wav2Vec2LidOptions options) Creates a Wav2Vec2 language identifier with ONNX model for inference.

Properties

Property Summary
IReadOnlyList<string> SupportedLanguages { get; }
bool SupportsTraining { get; }
int HiddenSize { get; } Gets the hidden size of the transformer.

Methods

Method Summary
Dictionary<string, Tensor<T>> GetNamedLayerActivations(Tensor<T> input)
IReadOnlyDictionary<string, T> GetLanguageProbabilities(Tensor<T> audio)
IReadOnlyList<LanguageSegment<T>> IdentifyLanguageSegments(Tensor<T> audio, int windowSizeMs)
IReadOnlyList<ValueTuple<string, T>> GetTopLanguages(Tensor<T> audio, int topN)
LanguageResult<T> IdentifyLanguage(Tensor<T> audio)
ModelMetadata<T> GetModelMetadata()
Tensor<T> ForwardForTraining(Tensor<T> input)
ValueTuple<bool, T> AreSameLanguage(Tensor<T> audio1, Tensor<T> audio2)
string GetLanguageDisplayName(string languageCode)
void LoadTorchWeights(Func<string, int[], double[]> read) Loads a transformers Wav2Vec2ForSequenceClassification state dict straight from source.