IDEFICS3<T>

Models & Types · AiDotNet.VisionLanguage.Generative

IDEFICS3: state-of-the-art 8B VLM trained with Docmatix for document understanding.

Inheritance: NeuralNetworkBase<T> → VisionLanguageModelBase<T> → IDEFICS3<T>

Implements: IGenerativeVisionLanguageModel<T>

For Beginners

IDEFICS3 builds on IDEFICS2 with improved training data, including the Docmatix dataset for document understanding, and upgrades the language backbone to Llama 3.1. It achieves state-of-the-art results on document QA benchmarks while being trained exclusively on open datasets, making it one of the strongest open-source VLMs at the 8B parameter scale. Default values follow the original paper settings.

How It Works

IDEFICS3 (Laurencon et al., 2024) is an 8B parameter model that builds upon IDEFICS2 with improved training data including the Docmatix dataset for document understanding. It retains the SigLIP + perceiver + Llama 3.1 architecture but achieves state-of-the-art results on document QA benchmarks while being trained exclusively on open datasets.

References:

  • Paper: “Building and better understanding vision-language models: insights and future directions” (Laurencon et al., 2024)

Example

var architecture = new NeuralNetworkArchitecture<double>(
    inputType: InputType.TwoDimensional,
    taskType: NeuralNetworkTaskType.ImageClassification,
    inputHeight: 224, inputWidth: 224, inputDepth: 3, outputSize: 512);
var trainModel = new IDEFICS3<double>(architecture, new IDEFICS3Options());

Constructors

Constructor Summary
IDEFICS3(NeuralNetworkArchitecture<T> architecture, IDEFICS3Options options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer)
IDEFICS3(NeuralNetworkArchitecture<T> architecture, string modelPath, IDEFICS3Options options)

Properties

Property Summary
int DecoderEmbeddingDim { get; }
int EmbeddingDimension { get; }
int MaxGenerationLength { get; }

Methods

Method Summary
ModelMetadata<T> GetModelMetadata()
Tensor<T> EncodeImage(Tensor<T> image)
Tensor<T> GenerateFromImage(Tensor<T> image, string prompt) Generates text using IDEFICS3’s SigLIP+Llama-3.1 architecture trained on Docmatix.