AudioVisualCorrespondenceNetwork<T>

Models & Types · AiDotNet.NeuralNetworks

Audio-visual correspondence learning network for cross-modal understanding.

Inheritance: NeuralNetworkBase<T> → DeclaredModelLayoutBase<T> → MultimodalModelLayoutBase<T> → AudioVisualCorrespondenceNetwork<T>

Implements: IAudioVisualCorrespondenceModel<T>

For Beginners

This model learns to connect what it “hears” with what it “sees.”

For example, if you show it a video of someone playing guitar, it learns that the guitar sound corresponds to the guitar in the image. This enables:

  • Sound source localization: “Where in the image is the sound coming from?”
  • Audio-visual retrieval: “Find images that match this sound”
  • Scene understanding: “What objects are making sounds in this scene?”

Separate audio and visual input projections feed one shared Dense encoder. The paired training objective aligns their normalized embeddings; the feature-input Predict path returns two correspondence logits through the separate fusion stack.

How It Works

This network learns correspondences between audio and visual modalities, enabling sound source localization, audio-visual retrieval, and scene understanding.

Example

var architecture = new NeuralNetworkArchitecture<float>(
    inputType: InputType.OneDimensional,
    taskType: NeuralNetworkTaskType.BinaryClassification,
    inputSize: 512,
    outputSize: 2);
            
var trainX = Tensor<float>.CreateRandom(4, 512);
var trainY = Tensor<float>.CreateRandom(4, 128);
            
var result = new AiModelBuilder<float, Tensor<float>, Tensor<float>>()
    .ConfigureModel(new AudioVisualCorrespondenceNetwork<float>(architecture))
    .Build(trainX, trainY);
            
var inputTensor = Tensor<float>.CreateRandom(1, 512);
Tensor<float> embedding = result.Predict(inputTensor);

Constructors

Constructor Summary
AudioVisualCorrespondenceNetwork() Initializes a new instance with default architecture settings.
AudioVisualCorrespondenceNetwork(NeuralNetworkArchitecture<T> architecture, AudioVisualCorrespondenceOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer, ILossFunction<T> lossFunction) Creates a new audio-visual correspondence network.

Properties

Property Summary
double VideoFrameRate { get; }
int AudioSampleRate { get; }
int EmbeddingDimension { get; }

Methods

Method Summary
Dictionary<string, T> ClassifyScene(Tensor<T> audioWaveform, IEnumerable<Tensor<T>> frames, IEnumerable<string> sceneLabels)
IEnumerable<Tensor<T>> LocalizeSoundSource(Tensor<T> audioWaveform, IEnumerable<Tensor<T>> frames)
IEnumerable<ValueTuple<int, T>> RetrieveAudioFromVisuals(IEnumerable<Tensor<T>> frames, IEnumerable<Vector<T>> audioDatabase, int topK)
IEnumerable<ValueTuple<int, T>> RetrieveVisualsFromAudio(Tensor<T> audioWaveform, IEnumerable<Vector<T>> visualDatabase, int topK)
ModelMetadata<T> GetModelMetadata()
T ComputeCorrespondence(Tensor<T> audioWaveform, IEnumerable<Tensor<T>> frames)
Tensor<T> SeparateAudioByVisual(Tensor<T> mixedAudio, Tensor<T> targetVisual)
ValueTuple<double, T> CheckSynchronization(Tensor<T> audioWaveform, IEnumerable<Tensor<T>> frames)
Vector<T> GetAudioEmbedding(Tensor<T> audioWaveform, int sampleRate)
Vector<T> GetVisualEmbedding(IEnumerable<Tensor<T>> frames)
string DescribeExpectedAudio(IEnumerable<Tensor<T>> frames)
void LearnCorrespondence(IEnumerable<Tensor<T>> audioSamples, IEnumerable<IEnumerable<Tensor<T>>> visualSamples, int epochs)
void LearnScene(Tensor<T> audioWaveform, IEnumerable<Tensor<T>> frames, string label) Adds one labeled audio-visual example to that label’s scene prototype, the running mean of its joint embeddings.
void LearnSeparation(IEnumerable<Tensor<T>> mixedAudio, IEnumerable<Tensor<T>> targetVisuals, IEnumerable<Tensor<T>> targetMasks, int epochs) Trains the separation head, together with the encoders, adapters and fusion trunk beneath it, to predict each example’s per-mel-bin mask for the sound belonging to its visual.
void LearnSynchronization(IEnumerable<Tensor<T>> audioSamples, IEnumerable<IEnumerable<Tensor<T>>> visualSamples, IEnumerable<double> offsetsSeconds, int epochs) Trains the synchronization head, together with the encoders, adapters and fusion trunk beneath it, to predict each pair’s audio-to-visual offset in seconds.