IDEFICS3<T>
Models & Types · AiDotNet.VisionLanguage.Generative
IDEFICS3: state-of-the-art 8B VLM trained with Docmatix for document understanding.
Inheritance: NeuralNetworkBase<T> → VisionLanguageModelBase<T> → IDEFICS3<T>
Implements: IGenerativeVisionLanguageModel<T>
For Beginners
IDEFICS3 builds on IDEFICS2 with improved training data, including the Docmatix dataset for document understanding, and upgrades the language backbone to Llama 3.1. It achieves state-of-the-art results on document QA benchmarks while being trained exclusively on open datasets, making it one of the strongest open-source VLMs at the 8B parameter scale. Default values follow the original paper settings.
How It Works
IDEFICS3 (Laurencon et al., 2024) is an 8B parameter model that builds upon IDEFICS2 with improved training data including the Docmatix dataset for document understanding. It retains the SigLIP + perceiver + Llama 3.1 architecture but achieves state-of-the-art results on document QA benchmarks while being trained exclusively on open datasets.
References:
- Paper: “Building and better understanding vision-language models: insights and future directions” (Laurencon et al., 2024)
Example
var architecture = new NeuralNetworkArchitecture<double>(
inputType: InputType.TwoDimensional,
taskType: NeuralNetworkTaskType.ImageClassification,
inputHeight: 224, inputWidth: 224, inputDepth: 3, outputSize: 512);
var trainModel = new IDEFICS3<double>(architecture, new IDEFICS3Options());
Constructors
| Constructor | Summary |
|---|---|
IDEFICS3(NeuralNetworkArchitecture<T> architecture, IDEFICS3Options options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
|
IDEFICS3(NeuralNetworkArchitecture<T> architecture, string modelPath, IDEFICS3Options options) |
Properties
| Property | Summary |
|---|---|
int DecoderEmbeddingDim { get; } |
|
int EmbeddingDimension { get; } |
|
int MaxGenerationLength { get; } |
Methods
| Method | Summary |
|---|---|
ModelMetadata<T> GetModelMetadata() |
|
Tensor<T> EncodeImage(Tensor<T> image) |
|
Tensor<T> GenerateFromImage(Tensor<T> image, string prompt) |
Generates text using IDEFICS3’s SigLIP+Llama-3.1 architecture trained on Docmatix. |