VideoLLaVA<T>

Models & Types · AiDotNet.VisionLanguage.VideoLanguage

Video-LLaVA: united visual representation for video understanding.

Inheritance: NeuralNetworkBase<T> → VisionLanguageModelBase<T> → VideoLLaVA<T>

Implements: IVideoLanguageModel<T>

For Beginners

Video-LLaVA is a video-language model that aligns image and video features for unified visual understanding. Default values follow the original paper settings.

How It Works

Video-LLaVA (PKU, 2024) learns united visual representations for video understanding by aligning image and video features before projection into the language model. It processes both images and videos through separate encoders that are aligned into a shared feature space, enabling the model to leverage image-text knowledge for video understanding.

References:

  • Paper: “Video-LLaVA: Learning United Visual Representation by Alignment Before Projection” (PKU, 2024)

Example

// Create a Video-LLaVA model for unified video understanding
// with aligned visual representations before projection
var architecture = new NeuralNetworkArchitecture<double>(
    inputType: InputType.TwoDimensional,
    taskType: NeuralNetworkTaskType.ImageClassification,
    inputHeight: 224, inputWidth: 224, inputDepth: 3, outputSize: 512);
            
// ONNX inference mode with pre-trained model
var model = new VideoLLaVA<double>(architecture, "videollava.onnx");
            
// Training mode with native layers
var trainModel = new VideoLLaVA<double>(architecture, new VideoLLaVAOptions());

Constructors

Constructor Summary
VideoLLaVA(NeuralNetworkArchitecture<T> architecture, VideoLLaVAOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer)
VideoLLaVA(NeuralNetworkArchitecture<T> architecture, string modelPath, VideoLLaVAOptions options)

Properties

Property Summary
int DecoderEmbeddingDim { get; }
int EmbeddingDimension { get; }
int MaxFrames { get; }
int MaxGenerationLength { get; }
string LanguageModelName { get; }

Methods

Method Summary
ModelMetadata<T> GetModelMetadata()
Tensor<T> EncodeImage(Tensor<T> image)
Tensor<T> GenerateFromImage(Tensor<T> image, string prompt) Generates from a single image using Video-LLaVA’s alignment-before-projection approach.
Tensor<T> GenerateFromVideo(IReadOnlyList<Tensor<T>> frames, string prompt) Generates output from video frames using Video-LLaVA’s unified alignment before projection.