AudioVisualEventLocalizationNetwork<T>

Models & Types · AiDotNet.NeuralNetworks

Neural network for audio-visual event localization - identifying WHEN and WHERE events occur in video by jointly analyzing audio and visual streams with precise temporal boundaries.

Inheritance: NeuralNetworkBase<T> → DeclaredModelLayoutBase<T> → MultimodalModelLayoutBase<T> → AudioVisualEventLocalizationNetwork<T>

Implements: IAudioVisualEventLocalizationModel<T>

For Beginners

This model watches and listens to video simultaneously to find specific events. For example, in a concert video it can identify:

  • WHEN the guitar solo starts and ends (temporal localization)
  • WHERE on screen the guitar player is (spatial localization)

It works by processing audio and video frames in parallel, then using cross-modal attention to find moments where what’s heard matches what’s seen. This is useful for video surveillance, sports analysis, and content moderation.

How It Works

This network jointly analyzes audio and visual streams to identify when and where events occur in video, producing precise temporal boundaries for detected events.

Example

var architecture = new NeuralNetworkArchitecture<float>(
    inputType: InputType.OneDimensional,
    taskType: NeuralNetworkTaskType.ObjectDetection,
    inputSize: 512,
    outputSize: 128);
            
var trainX = Tensor<float>.CreateRandom(4, 512);
var trainY = Tensor<float>.CreateRandom(4, 128);
            
var result = new AiModelBuilder<float, Tensor<float>, Tensor<float>>()
    .ConfigureModel(new AudioVisualEventLocalizationNetwork<float>(architecture))
    .Build(trainX, trainY);
            
var inputTensor = Tensor<float>.CreateRandom(1, 512);
Tensor<float> detections = result.Predict(inputTensor);

Constructors

Constructor Summary
AudioVisualEventLocalizationNetwork() Initializes a new instance with default architecture settings.
AudioVisualEventLocalizationNetwork(NeuralNetworkArchitecture<T> architecture, AudioVisualEventLocalizationOptions options, IEnumerable<string> eventCategories, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer, ILossFunction<T> lossFunction, int? seed) Initializes a new instance of the AudioVisualEventLocalizationNetwork.

Properties

Property Summary
IReadOnlyList<string> SupportedEventCategories { get; }
double TemporalResolution { get; }

Methods

Method Summary
Dictionary<string, T> ClassifyEvent(Tensor<T> audioSegment, IEnumerable<Tensor<T>> frameSegment, IEnumerable<string> candidateLabels)
IEnumerable<AudioVisualEvent> DetectEvents(Tensor<T> audioWaveform, IEnumerable<Tensor<T>> frames, double frameRate)
IEnumerable<AudioVisualEvent> DetectSpecificEvents(Tensor<T> audioWaveform, IEnumerable<Tensor<T>> frames, IEnumerable<string> targetCategories, double frameRate)
IEnumerable<AudioVisualEvent> TrackEvent(Tensor<T> audioWaveform, IEnumerable<Tensor<T>> frames, AudioVisualEvent initialEvent, double frameRate)
IEnumerable<ValueTuple<double, double, T, string>> DetectAnomalies(Tensor<T> audioWaveform, IEnumerable<Tensor<T>> frames, double frameRate)
IEnumerable<ValueTuple<double, double, T, string>> DetectSyncEvents(Tensor<T> audioWaveform, IEnumerable<Tensor<T>> frames, double frameRate)
IEnumerable<ValueTuple<double, double, T>> GenerateProposals(Tensor<T> audioWaveform, IEnumerable<Tensor<T>> frames, double frameRate)
IEnumerable<ValueTuple<double, double, T>> LocalizeEventByDescription(Tensor<T> audioWaveform, IEnumerable<Tensor<T>> frames, string eventDescription, double frameRate)
IEnumerable<ValueTuple<double, double, string>> SegmentScenes(Tensor<T> audioWaveform, IEnumerable<Tensor<T>> frames, double frameRate)
IEnumerable<ValueTuple<double, string>> GenerateDenseCaptions(Tensor<T> audioWaveform, IEnumerable<Tensor<T>> frames, double frameRate)
ModelMetadata<T> GetModelMetadata()
Tensor<T> ForwardForTraining(Tensor<T> input)
ValueTuple<Tensor<T>, Tensor<T>> ComputeEventAttention(Tensor<T> audioSegment, IEnumerable<Tensor<T>> frameSegment)
ValueTuple<string, IEnumerable<ValueTuple<double, double>>> AnswerEventQuestion(Tensor<T> audioWaveform, IEnumerable<Tensor<T>> frames, string question, double frameRate)