VideoLLaVA<T>
Models & Types · AiDotNet.VisionLanguage.VideoLanguage
Video-LLaVA: united visual representation for video understanding.
Inheritance: NeuralNetworkBase<T> → VisionLanguageModelBase<T> → VideoLLaVA<T>
Implements: IVideoLanguageModel<T>
For Beginners
Video-LLaVA is a video-language model that aligns image and video features for unified visual understanding. Default values follow the original paper settings.
How It Works
Video-LLaVA (PKU, 2024) learns united visual representations for video understanding by aligning image and video features before projection into the language model. It processes both images and videos through separate encoders that are aligned into a shared feature space, enabling the model to leverage image-text knowledge for video understanding.
References:
- Paper: “Video-LLaVA: Learning United Visual Representation by Alignment Before Projection” (PKU, 2024)
Example
// Create a Video-LLaVA model for unified video understanding
// with aligned visual representations before projection
var architecture = new NeuralNetworkArchitecture<double>(
inputType: InputType.TwoDimensional,
taskType: NeuralNetworkTaskType.ImageClassification,
inputHeight: 224, inputWidth: 224, inputDepth: 3, outputSize: 512);
// ONNX inference mode with pre-trained model
var model = new VideoLLaVA<double>(architecture, "videollava.onnx");
// Training mode with native layers
var trainModel = new VideoLLaVA<double>(architecture, new VideoLLaVAOptions());
Constructors
| Constructor | Summary |
|---|---|
VideoLLaVA(NeuralNetworkArchitecture<T> architecture, VideoLLaVAOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
|
VideoLLaVA(NeuralNetworkArchitecture<T> architecture, string modelPath, VideoLLaVAOptions options) |
Properties
| Property | Summary |
|---|---|
int DecoderEmbeddingDim { get; } |
|
int EmbeddingDimension { get; } |
|
int MaxFrames { get; } |
|
int MaxGenerationLength { get; } |
|
string LanguageModelName { get; } |
Methods
| Method | Summary |
|---|---|
ModelMetadata<T> GetModelMetadata() |
|
Tensor<T> EncodeImage(Tensor<T> image) |
|
Tensor<T> GenerateFromImage(Tensor<T> image, string prompt) |
Generates from a single image using Video-LLaVA’s alignment-before-projection approach. |
Tensor<T> GenerateFromVideo(IReadOnlyList<Tensor<T>> frames, string prompt) |
Generates output from video frames using Video-LLaVA’s unified alignment before projection. |