VILA<T>

Models & Types · AiDotNet.VisionLanguage.InstructionTuned

VILA: VLM pre-trained with interleaved image-text data for enhanced in-context learning.

Inheritance: NeuralNetworkBase<T> → VisionLanguageModelBase<T> → VILA<T>

Implements: IInstructionTunedVLM<T>

For Beginners

VILA from NVIDIA explores what makes pre-training work best for vision-language models. Its key finding is that training with interleaved image-text data — where images and text alternate naturally like on a web page — gives the model much better in-context learning abilities. This means you can show VILA a few examples of a new task in the prompt (like “here are 3 examples of product descriptions from product photos”) and it will learn to do that task without any fine-tuning. It also found that keeping the language model unfrozen during pre-training is essential for this capability. Default values follow the original paper settings.

How It Works

VILA (NVIDIA, 2024) investigates pre-training strategies for vision-language models and demonstrates that pre-training with interleaved image-text data (where images and text alternate naturally, like web pages) significantly improves in-context learning capabilities. The model shows that unfreezing the language model during pre-training and using interleaved data together produce strong few-shot learning performance, where the model can learn new tasks from just a few examples shown in the prompt.

References:

  • Paper: “VILA: On Pre-training for Visual Language Models” (NVIDIA, 2024)

Example

// Create a VILA model for in-context visual learning
// pre-trained with interleaved image-text data
var architecture = new NeuralNetworkArchitecture<double>(
    inputType: InputType.TwoDimensional,
    taskType: NeuralNetworkTaskType.ImageClassification,
    inputHeight: 224, inputWidth: 224, inputDepth: 3, outputSize: 512);
            
// ONNX inference mode with pre-trained model
var model = new VILA<double>(architecture, "vila.onnx");
            
// Training mode with native layers
var trainModel = new VILA<double>(architecture, new VILAOptions());

Constructors

Constructor Summary
VILA(NeuralNetworkArchitecture<T> architecture, VILAOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer)
VILA(NeuralNetworkArchitecture<T> architecture, string modelPath, VILAOptions options)

Properties

Property Summary
int DecoderEmbeddingDim { get; }
int EmbeddingDimension { get; }
int MaxGenerationLength { get; }
string LanguageModelName { get; }

Methods

Method Summary
ModelMetadata<T> GetModelMetadata()
Tensor<T> Chat(Tensor<T> image, IEnumerable<ValueTuple<string, string>> conversationHistory, string userMessage)
Tensor<T> EncodeImage(Tensor<T> image)
Tensor<T> GenerateFromImage(Tensor<T> image, string prompt) Generates text using VILA’s interleaved pre-training with visual instruction tuning.