VILA<T>
Models & Types · AiDotNet.VisionLanguage.InstructionTuned
VILA: VLM pre-trained with interleaved image-text data for enhanced in-context learning.
Inheritance: NeuralNetworkBase<T> → VisionLanguageModelBase<T> → VILA<T>
Implements: IInstructionTunedVLM<T>
For Beginners
VILA from NVIDIA explores what makes pre-training work best for vision-language models. Its key finding is that training with interleaved image-text data — where images and text alternate naturally like on a web page — gives the model much better in-context learning abilities. This means you can show VILA a few examples of a new task in the prompt (like “here are 3 examples of product descriptions from product photos”) and it will learn to do that task without any fine-tuning. It also found that keeping the language model unfrozen during pre-training is essential for this capability. Default values follow the original paper settings.
How It Works
VILA (NVIDIA, 2024) investigates pre-training strategies for vision-language models and demonstrates that pre-training with interleaved image-text data (where images and text alternate naturally, like web pages) significantly improves in-context learning capabilities. The model shows that unfreezing the language model during pre-training and using interleaved data together produce strong few-shot learning performance, where the model can learn new tasks from just a few examples shown in the prompt.
References:
- Paper: “VILA: On Pre-training for Visual Language Models” (NVIDIA, 2024)
Example
// Create a VILA model for in-context visual learning
// pre-trained with interleaved image-text data
var architecture = new NeuralNetworkArchitecture<double>(
inputType: InputType.TwoDimensional,
taskType: NeuralNetworkTaskType.ImageClassification,
inputHeight: 224, inputWidth: 224, inputDepth: 3, outputSize: 512);
// ONNX inference mode with pre-trained model
var model = new VILA<double>(architecture, "vila.onnx");
// Training mode with native layers
var trainModel = new VILA<double>(architecture, new VILAOptions());
Constructors
| Constructor | Summary |
|---|---|
VILA(NeuralNetworkArchitecture<T> architecture, VILAOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
|
VILA(NeuralNetworkArchitecture<T> architecture, string modelPath, VILAOptions options) |
Properties
| Property | Summary |
|---|---|
int DecoderEmbeddingDim { get; } |
|
int EmbeddingDimension { get; } |
|
int MaxGenerationLength { get; } |
|
string LanguageModelName { get; } |
Methods
| Method | Summary |
|---|---|
ModelMetadata<T> GetModelMetadata() |
|
Tensor<T> Chat(Tensor<T> image, IEnumerable<ValueTuple<string, string>> conversationHistory, string userMessage) |
|
Tensor<T> EncodeImage(Tensor<T> image) |
|
Tensor<T> GenerateFromImage(Tensor<T> image, string prompt) |
Generates text using VILA’s interleaved pre-training with visual instruction tuning. |