IAudioVisualEventLocalizationModel<T>

Interfaces · AiDotNet.Interfaces

Defines the contract for audio-visual event localization models.

Known implementations: AudioVisualEventLocalizationNetwork<T>

For Beginners

Finding events in videos using sight AND sound!

Key capabilities:

  • Temporal localization: When does the dog bark? (2.3s - 4.1s)
  • Spatial localization: Where is the barking dog? (bounding box)
  • Event classification: What kind of event is it? (animal sound)
  • Multi-event detection: Find all events in a video

Use cases:

  • Video surveillance: Detect glass breaking sounds and locate the window
  • Sports analysis: Find and timestamp all goals using crowd cheering
  • Content moderation: Detect and locate inappropriate audio-visual content

How It Works

Audio-visual event localization identifies WHEN and WHERE events occur in video by jointly analyzing audio and visual streams. This goes beyond simple detection to provide precise temporal boundaries and spatial locations.

Properties

Property Summary
IReadOnlyList<string> SupportedEventCategories { get; } Gets the supported event categories.
double TemporalResolution { get; } Gets the temporal resolution in seconds.

Methods

Method Summary
Dictionary<string, T> ClassifyEvent(Tensor<T> audioSegment, IEnumerable<Tensor<T>> frameSegment, IEnumerable<string> candidateLabels) Classifies a pre-segmented event.
IEnumerable<AudioVisualEvent> DetectEvents(Tensor<T> audioWaveform, IEnumerable<Tensor<T>> frames, double frameRate) Detects and localizes all audio-visual events in a video.
IEnumerable<AudioVisualEvent> DetectSpecificEvents(Tensor<T> audioWaveform, IEnumerable<Tensor<T>> frames, IEnumerable<string> targetCategories, double frameRate) Detects events of specific categories.
IEnumerable<AudioVisualEvent> TrackEvent(Tensor<T> audioWaveform, IEnumerable<Tensor<T>> frames, AudioVisualEvent initialEvent, double frameRate) Tracks an event across time.
IEnumerable<ValueTuple<double, double, T, string>> DetectAnomalies(Tensor<T> audioWaveform, IEnumerable<Tensor<T>> frames, double frameRate) Detects anomalous events that don’t match expected patterns.
IEnumerable<ValueTuple<double, double, T, string>> DetectSyncEvents(Tensor<T> audioWaveform, IEnumerable<Tensor<T>> frames, double frameRate) Detects audio-visual synchronization events (e.g., lip sync).
IEnumerable<ValueTuple<double, double, T>> GenerateProposals(Tensor<T> audioWaveform, IEnumerable<Tensor<T>> frames, double frameRate) Generates temporal proposals for potential events.
IEnumerable<ValueTuple<double, double, T>> LocalizeEventByDescription(Tensor<T> audioWaveform, IEnumerable<Tensor<T>> frames, string eventDescription, double frameRate) Localizes a specific event described in text.
IEnumerable<ValueTuple<double, double, string>> SegmentScenes(Tensor<T> audioWaveform, IEnumerable<Tensor<T>> frames, double frameRate) Segments video into coherent audio-visual scenes.
IEnumerable<ValueTuple<double, string>> GenerateDenseCaptions(Tensor<T> audioWaveform, IEnumerable<Tensor<T>> frames, double frameRate) Generates dense event captions for the entire video.
ValueTuple<Tensor<T>, Tensor<T>> ComputeEventAttention(Tensor<T> audioSegment, IEnumerable<Tensor<T>> frameSegment) Computes event-level audio-visual attention.
ValueTuple<string, IEnumerable<ValueTuple<double, double>>> AnswerEventQuestion(Tensor<T> audioWaveform, IEnumerable<Tensor<T>> frames, string question, double frameRate) Answers questions about events in the video.