IAudioVisualCorrespondenceModel<T>
Interfaces · AiDotNet.Interfaces
Defines the contract for audio-visual correspondence learning models.
Known implementations: AudioVisualCorrespondenceNetwork<T>
For Beginners
Teaching AI to connect sounds with visuals!
Key capabilities:
- Sound source localization: Where in the image is the sound coming from?
- Audio-visual synchronization: Are the audio and video in sync?
- Cross-modal retrieval: Find images matching sounds and vice versa
- Audio-visual scene understanding: What’s happening based on both modalities?
Examples:
- A dog barking → The model highlights the dog in the image
- Piano music → The model finds images of pianos
- Clapping sound → The model locates hands in the video
How It Works
Audio-visual correspondence learning focuses on understanding the relationship between what we see and what we hear. This enables tasks like finding the source of a sound in a video, synchronizing audio and video, and understanding audio-visual events.
Properties
| Property | Summary |
|---|---|
double VideoFrameRate { get; } |
Gets the expected video frame rate. |
int AudioSampleRate { get; } |
Gets the expected audio sample rate. |
int EmbeddingDimension { get; } |
Gets the embedding dimension for audio-visual features. |
Methods
| Method | Summary |
|---|---|
Dictionary<string, T> ClassifyScene(Tensor<T> audioWaveform, IEnumerable<Tensor<T>> frames, IEnumerable<string> sceneLabels) |
Classifies audio-visual scenes. |
IEnumerable<Tensor<T>> LocalizeSoundSource(Tensor<T> audioWaveform, IEnumerable<Tensor<T>> frames) |
Localizes sound sources in video frames. |
IEnumerable<ValueTuple<int, T>> RetrieveAudioFromVisuals(IEnumerable<Tensor<T>> frames, IEnumerable<Vector<T>> audioDatabase, int topK) |
Retrieves audio content matching visual input. |
IEnumerable<ValueTuple<int, T>> RetrieveVisualsFromAudio(Tensor<T> audioWaveform, IEnumerable<Vector<T>> visualDatabase, int topK) |
Retrieves visual content matching audio. |
T ComputeCorrespondence(Tensor<T> audioWaveform, IEnumerable<Tensor<T>> frames) |
Computes audio-visual correspondence score. |
Tensor<T> SeparateAudioByVisual(Tensor<T> mixedAudio, Tensor<T> targetVisual) |
Separates audio sources based on visual guidance. |
ValueTuple<double, T> CheckSynchronization(Tensor<T> audioWaveform, IEnumerable<Tensor<T>> frames) |
Checks audio-visual synchronization. |
Vector<T> GetAudioEmbedding(Tensor<T> audioWaveform, int sampleRate) |
Computes audio embedding from waveform. |
Vector<T> GetVisualEmbedding(IEnumerable<Tensor<T>> frames) |
Computes visual embedding from video frames. |
string DescribeExpectedAudio(IEnumerable<Tensor<T>> frames) |
Generates audio description from visual content. |
void LearnCorrespondence(IEnumerable<Tensor<T>> audioSamples, IEnumerable<IEnumerable<Tensor<T>>> visualSamples, int epochs) |
Learns correspondence from paired audio-visual data. |