IVideoLanguageModel<T>

Interfaces · AiDotNet.VisionLanguage.Interfaces

Interface for video-language models that process video frames for temporal understanding and QA.

Implements: IGenerativeVisionLanguageModel<T>

Known implementations: LLaVANeXTVideo<T>, LLaVAVideo<T>, LongVILA<T>, PLLaVA<T>, SlowFastLLaVA<T>, VideoChat2<T>, VideoLLaMA2<T>, VideoLLaMA3<T>, VideoLLaVA<T>

How It Works

Video-language models extend image-based VLMs to handle temporal sequences of video frames. Architectures include frame averaging, spatial-temporal convolution, slow/fast pathways, and long-context approaches for processing hour+ videos.

Properties

Property Summary
int MaxFrames { get; } Gets the maximum number of video frames the model can process.
string LanguageModelName { get; } Gets the name of the language model backbone.

Methods

Method Summary
Tensor<T> GenerateFromVideo(IReadOnlyList<Tensor<T>> frames, string prompt) Generates output from a sequence of video frames, optionally conditioned on a text prompt.