IAudioVisualEventLocalizationModel<T>
Interfaces · AiDotNet.Interfaces
Defines the contract for audio-visual event localization models.
Known implementations: AudioVisualEventLocalizationNetwork<T>
For Beginners
Finding events in videos using sight AND sound!
Key capabilities:
- Temporal localization: When does the dog bark? (2.3s - 4.1s)
- Spatial localization: Where is the barking dog? (bounding box)
- Event classification: What kind of event is it? (animal sound)
- Multi-event detection: Find all events in a video
Use cases:
- Video surveillance: Detect glass breaking sounds and locate the window
- Sports analysis: Find and timestamp all goals using crowd cheering
- Content moderation: Detect and locate inappropriate audio-visual content
How It Works
Audio-visual event localization identifies WHEN and WHERE events occur in video by jointly analyzing audio and visual streams. This goes beyond simple detection to provide precise temporal boundaries and spatial locations.
Properties
| Property | Summary |
|---|---|
IReadOnlyList<string> SupportedEventCategories { get; } |
Gets the supported event categories. |
double TemporalResolution { get; } |
Gets the temporal resolution in seconds. |
Methods
| Method | Summary |
|---|---|
Dictionary<string, T> ClassifyEvent(Tensor<T> audioSegment, IEnumerable<Tensor<T>> frameSegment, IEnumerable<string> candidateLabels) |
Classifies a pre-segmented event. |
IEnumerable<AudioVisualEvent> DetectEvents(Tensor<T> audioWaveform, IEnumerable<Tensor<T>> frames, double frameRate) |
Detects and localizes all audio-visual events in a video. |
IEnumerable<AudioVisualEvent> DetectSpecificEvents(Tensor<T> audioWaveform, IEnumerable<Tensor<T>> frames, IEnumerable<string> targetCategories, double frameRate) |
Detects events of specific categories. |
IEnumerable<AudioVisualEvent> TrackEvent(Tensor<T> audioWaveform, IEnumerable<Tensor<T>> frames, AudioVisualEvent initialEvent, double frameRate) |
Tracks an event across time. |
IEnumerable<ValueTuple<double, double, T, string>> DetectAnomalies(Tensor<T> audioWaveform, IEnumerable<Tensor<T>> frames, double frameRate) |
Detects anomalous events that don’t match expected patterns. |
IEnumerable<ValueTuple<double, double, T, string>> DetectSyncEvents(Tensor<T> audioWaveform, IEnumerable<Tensor<T>> frames, double frameRate) |
Detects audio-visual synchronization events (e.g., lip sync). |
IEnumerable<ValueTuple<double, double, T>> GenerateProposals(Tensor<T> audioWaveform, IEnumerable<Tensor<T>> frames, double frameRate) |
Generates temporal proposals for potential events. |
IEnumerable<ValueTuple<double, double, T>> LocalizeEventByDescription(Tensor<T> audioWaveform, IEnumerable<Tensor<T>> frames, string eventDescription, double frameRate) |
Localizes a specific event described in text. |
IEnumerable<ValueTuple<double, double, string>> SegmentScenes(Tensor<T> audioWaveform, IEnumerable<Tensor<T>> frames, double frameRate) |
Segments video into coherent audio-visual scenes. |
IEnumerable<ValueTuple<double, string>> GenerateDenseCaptions(Tensor<T> audioWaveform, IEnumerable<Tensor<T>> frames, double frameRate) |
Generates dense event captions for the entire video. |
ValueTuple<Tensor<T>, Tensor<T>> ComputeEventAttention(Tensor<T> audioSegment, IEnumerable<Tensor<T>> frameSegment) |
Computes event-level audio-visual attention. |
ValueTuple<string, IEnumerable<ValueTuple<double, double>>> AnswerEventQuestion(Tensor<T> audioWaveform, IEnumerable<Tensor<T>> frames, string question, double frameRate) |
Answers questions about events in the video. |