AudioVisualCorrespondenceNetwork<T>
Models & Types · AiDotNet.NeuralNetworks
Audio-visual correspondence learning network for cross-modal understanding.
Inheritance: NeuralNetworkBase<T> → DeclaredModelLayoutBase<T> → MultimodalModelLayoutBase<T> → AudioVisualCorrespondenceNetwork<T>
Implements: IAudioVisualCorrespondenceModel<T>
For Beginners
This model learns to connect what it “hears” with what it “sees.”
For example, if you show it a video of someone playing guitar, it learns that the guitar sound corresponds to the guitar in the image. This enables:
- Sound source localization: “Where in the image is the sound coming from?”
- Audio-visual retrieval: “Find images that match this sound”
- Scene understanding: “What objects are making sounds in this scene?”
Separate audio and visual input projections feed one shared Dense encoder. The paired training objective aligns their normalized embeddings; the feature-input Predict path returns two correspondence logits through the separate fusion stack.
How It Works
This network learns correspondences between audio and visual modalities, enabling sound source localization, audio-visual retrieval, and scene understanding.
Example
var architecture = new NeuralNetworkArchitecture<float>(
inputType: InputType.OneDimensional,
taskType: NeuralNetworkTaskType.BinaryClassification,
inputSize: 512,
outputSize: 2);
var trainX = Tensor<float>.CreateRandom(4, 512);
var trainY = Tensor<float>.CreateRandom(4, 128);
var result = new AiModelBuilder<float, Tensor<float>, Tensor<float>>()
.ConfigureModel(new AudioVisualCorrespondenceNetwork<float>(architecture))
.Build(trainX, trainY);
var inputTensor = Tensor<float>.CreateRandom(1, 512);
Tensor<float> embedding = result.Predict(inputTensor);
Constructors
| Constructor | Summary |
|---|---|
AudioVisualCorrespondenceNetwork() |
Initializes a new instance with default architecture settings. |
AudioVisualCorrespondenceNetwork(NeuralNetworkArchitecture<T> architecture, AudioVisualCorrespondenceOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer, ILossFunction<T> lossFunction) |
Creates a new audio-visual correspondence network. |
Properties
| Property | Summary |
|---|---|
double VideoFrameRate { get; } |
|
int AudioSampleRate { get; } |
|
int EmbeddingDimension { get; } |
Methods
| Method | Summary |
|---|---|
Dictionary<string, T> ClassifyScene(Tensor<T> audioWaveform, IEnumerable<Tensor<T>> frames, IEnumerable<string> sceneLabels) |
|
IEnumerable<Tensor<T>> LocalizeSoundSource(Tensor<T> audioWaveform, IEnumerable<Tensor<T>> frames) |
|
IEnumerable<ValueTuple<int, T>> RetrieveAudioFromVisuals(IEnumerable<Tensor<T>> frames, IEnumerable<Vector<T>> audioDatabase, int topK) |
|
IEnumerable<ValueTuple<int, T>> RetrieveVisualsFromAudio(Tensor<T> audioWaveform, IEnumerable<Vector<T>> visualDatabase, int topK) |
|
ModelMetadata<T> GetModelMetadata() |
|
T ComputeCorrespondence(Tensor<T> audioWaveform, IEnumerable<Tensor<T>> frames) |
|
Tensor<T> SeparateAudioByVisual(Tensor<T> mixedAudio, Tensor<T> targetVisual) |
|
ValueTuple<double, T> CheckSynchronization(Tensor<T> audioWaveform, IEnumerable<Tensor<T>> frames) |
|
Vector<T> GetAudioEmbedding(Tensor<T> audioWaveform, int sampleRate) |
|
Vector<T> GetVisualEmbedding(IEnumerable<Tensor<T>> frames) |
|
string DescribeExpectedAudio(IEnumerable<Tensor<T>> frames) |
|
void LearnCorrespondence(IEnumerable<Tensor<T>> audioSamples, IEnumerable<IEnumerable<Tensor<T>>> visualSamples, int epochs) |
|
void LearnScene(Tensor<T> audioWaveform, IEnumerable<Tensor<T>> frames, string label) |
Adds one labeled audio-visual example to that label’s scene prototype, the running mean of its joint embeddings. |
void LearnSeparation(IEnumerable<Tensor<T>> mixedAudio, IEnumerable<Tensor<T>> targetVisuals, IEnumerable<Tensor<T>> targetMasks, int epochs) |
Trains the separation head, together with the encoders, adapters and fusion trunk beneath it, to predict each example’s per-mel-bin mask for the sound belonging to its visual. |
void LearnSynchronization(IEnumerable<Tensor<T>> audioSamples, IEnumerable<IEnumerable<Tensor<T>>> visualSamples, IEnumerable<double> offsetsSeconds, int epochs) |
Trains the synchronization head, together with the encoders, adapters and fusion trunk beneath it, to predict each pair’s audio-to-visual offset in seconds. |