LongVILA<T>

Models & Types · AiDotNet.VisionLanguage.VideoLanguage

LongVILA: long-context visual language model for 1hr+ videos.

Inheritance: NeuralNetworkBase<T> → VisionLanguageModelBase<T> → LongVILA<T>

Implements: IVideoLanguageModel<T>

For Beginners

LongVILA is a video-language model from NVIDIA for understanding long videos of 1 hour or more. Default values follow the original paper settings.

How It Works

LongVILA (NVIDIA, 2024) scales long-context visual language models for processing videos exceeding one hour in duration. It uses multi-modal sequence parallelism to distribute long video frame sequences across multiple GPUs and extends the context window to handle thousands of frames for long-form video understanding and temporal reasoning.

References:

  • Paper: “LongVILA: Scaling Long-Context Visual Language Models for Long Videos” (NVIDIA, 2024)

Example

// Create a LongVILA model for long-context video understanding
// capable of processing 1hr+ videos with long-context attention
var architecture = new NeuralNetworkArchitecture<double>(
    inputType: InputType.TwoDimensional,
    taskType: NeuralNetworkTaskType.ImageClassification,
    inputHeight: 224, inputWidth: 224, inputDepth: 3, outputSize: 512);
            
// ONNX inference mode with pre-trained model
var model = new LongVILA<double>(architecture, "longvila.onnx");
            
// Training mode with native layers
var trainModel = new LongVILA<double>(architecture, new LongVILAOptions());

Constructors

Constructor Summary
LongVILA(NeuralNetworkArchitecture<T> architecture, LongVILAOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer)
LongVILA(NeuralNetworkArchitecture<T> architecture, string modelPath, LongVILAOptions options)

Properties

Property Summary
int DecoderEmbeddingDim { get; }
int EmbeddingDimension { get; }
int MaxFrames { get; }
int MaxGenerationLength { get; }
string LanguageModelName { get; }

Methods

Method Summary
ModelMetadata<T> GetModelMetadata()
Tensor<T> EncodeImage(Tensor<T> image)
Tensor<T> GenerateFromImage(Tensor<T> image, string prompt) Generates from a single image using LongVILA’s MM-SP single-chunk processing.
Tensor<T> GenerateFromVideo(IReadOnlyList<Tensor<T>> frames, string prompt) Generates output from video frames using LongVILA’s chunked multi-modal sequence processing for long videos (1hr+).