LongVILA<T>
Models & Types · AiDotNet.VisionLanguage.VideoLanguage
LongVILA: long-context visual language model for 1hr+ videos.
Inheritance: NeuralNetworkBase<T> → VisionLanguageModelBase<T> → LongVILA<T>
Implements: IVideoLanguageModel<T>
For Beginners
LongVILA is a video-language model from NVIDIA for understanding long videos of 1 hour or more. Default values follow the original paper settings.
How It Works
LongVILA (NVIDIA, 2024) scales long-context visual language models for processing videos exceeding one hour in duration. It uses multi-modal sequence parallelism to distribute long video frame sequences across multiple GPUs and extends the context window to handle thousands of frames for long-form video understanding and temporal reasoning.
References:
- Paper: “LongVILA: Scaling Long-Context Visual Language Models for Long Videos” (NVIDIA, 2024)
Example
// Create a LongVILA model for long-context video understanding
// capable of processing 1hr+ videos with long-context attention
var architecture = new NeuralNetworkArchitecture<double>(
inputType: InputType.TwoDimensional,
taskType: NeuralNetworkTaskType.ImageClassification,
inputHeight: 224, inputWidth: 224, inputDepth: 3, outputSize: 512);
// ONNX inference mode with pre-trained model
var model = new LongVILA<double>(architecture, "longvila.onnx");
// Training mode with native layers
var trainModel = new LongVILA<double>(architecture, new LongVILAOptions());
Constructors
| Constructor | Summary |
|---|---|
LongVILA(NeuralNetworkArchitecture<T> architecture, LongVILAOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
|
LongVILA(NeuralNetworkArchitecture<T> architecture, string modelPath, LongVILAOptions options) |
Properties
| Property | Summary |
|---|---|
int DecoderEmbeddingDim { get; } |
|
int EmbeddingDimension { get; } |
|
int MaxFrames { get; } |
|
int MaxGenerationLength { get; } |
|
string LanguageModelName { get; } |
Methods
| Method | Summary |
|---|---|
ModelMetadata<T> GetModelMetadata() |
|
Tensor<T> EncodeImage(Tensor<T> image) |
|
Tensor<T> GenerateFromImage(Tensor<T> image, string prompt) |
Generates from a single image using LongVILA’s MM-SP single-chunk processing. |
Tensor<T> GenerateFromVideo(IReadOnlyList<Tensor<T>> frames, string prompt) |
Generates output from video frames using LongVILA’s chunked multi-modal sequence processing for long videos (1hr+). |