IAudioVisualCorrespondenceModel<T>

Interfaces · AiDotNet.Interfaces

Defines the contract for audio-visual correspondence learning models.

Known implementations: AudioVisualCorrespondenceNetwork<T>

For Beginners

Teaching AI to connect sounds with visuals!

Key capabilities:

  • Sound source localization: Where in the image is the sound coming from?
  • Audio-visual synchronization: Are the audio and video in sync?
  • Cross-modal retrieval: Find images matching sounds and vice versa
  • Audio-visual scene understanding: What’s happening based on both modalities?

Examples:

  • A dog barking → The model highlights the dog in the image
  • Piano music → The model finds images of pianos
  • Clapping sound → The model locates hands in the video

How It Works

Audio-visual correspondence learning focuses on understanding the relationship between what we see and what we hear. This enables tasks like finding the source of a sound in a video, synchronizing audio and video, and understanding audio-visual events.

Properties

Property Summary
double VideoFrameRate { get; } Gets the expected video frame rate.
int AudioSampleRate { get; } Gets the expected audio sample rate.
int EmbeddingDimension { get; } Gets the embedding dimension for audio-visual features.

Methods

Method Summary
Dictionary<string, T> ClassifyScene(Tensor<T> audioWaveform, IEnumerable<Tensor<T>> frames, IEnumerable<string> sceneLabels) Classifies audio-visual scenes.
IEnumerable<Tensor<T>> LocalizeSoundSource(Tensor<T> audioWaveform, IEnumerable<Tensor<T>> frames) Localizes sound sources in video frames.
IEnumerable<ValueTuple<int, T>> RetrieveAudioFromVisuals(IEnumerable<Tensor<T>> frames, IEnumerable<Vector<T>> audioDatabase, int topK) Retrieves audio content matching visual input.
IEnumerable<ValueTuple<int, T>> RetrieveVisualsFromAudio(Tensor<T> audioWaveform, IEnumerable<Vector<T>> visualDatabase, int topK) Retrieves visual content matching audio.
T ComputeCorrespondence(Tensor<T> audioWaveform, IEnumerable<Tensor<T>> frames) Computes audio-visual correspondence score.
Tensor<T> SeparateAudioByVisual(Tensor<T> mixedAudio, Tensor<T> targetVisual) Separates audio sources based on visual guidance.
ValueTuple<double, T> CheckSynchronization(Tensor<T> audioWaveform, IEnumerable<Tensor<T>> frames) Checks audio-visual synchronization.
Vector<T> GetAudioEmbedding(Tensor<T> audioWaveform, int sampleRate) Computes audio embedding from waveform.
Vector<T> GetVisualEmbedding(IEnumerable<Tensor<T>> frames) Computes visual embedding from video frames.
string DescribeExpectedAudio(IEnumerable<Tensor<T>> frames) Generates audio description from visual content.
void LearnCorrespondence(IEnumerable<Tensor<T>> audioSamples, IEnumerable<IEnumerable<Tensor<T>>> visualSamples, int epochs) Learns correspondence from paired audio-visual data.