IDEFICS2<T>
Models & Types · AiDotNet.VisionLanguage.Generative
IDEFICS2: 8B efficient VLM with SigLIP encoder and Mistral-7B decoder.
Inheritance: NeuralNetworkBase<T> → VisionLanguageModelBase<T> → IDEFICS2<T>
Implements: IGenerativeVisionLanguageModel<T>
For Beginners
IDEFICS2 is a much more efficient 8 billion parameter successor to the original 80B IDEFICS. It replaces the OpenCLIP encoder with SigLIP and uses Mistral-7B as the language backbone, introducing learned perceiver pooling and native resolution image processing that splits images into sub-images for better document understanding. Default values follow the original paper settings.
How It Works
IDEFICS2 (Laurencon et al., 2024) is an 8B parameter efficient VLM that replaces the OpenCLIP encoder with SigLIP and uses Mistral-7B as the language backbone. It introduces a learned perceiver pooling strategy and native resolution image processing via sub-image splitting for improved document understanding.
References:
- Paper: “What matters when building vision-language models?” (Laurencon et al., 2024)
Example
var architecture = new NeuralNetworkArchitecture<double>(
inputType: InputType.TwoDimensional,
taskType: NeuralNetworkTaskType.ImageClassification,
inputHeight: 224, inputWidth: 224, inputDepth: 3, outputSize: 512);
var trainModel = new IDEFICS2<double>(architecture, new IDEFICS2Options());
Constructors
| Constructor | Summary |
|---|---|
IDEFICS2(NeuralNetworkArchitecture<T> architecture, IDEFICS2Options options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
|
IDEFICS2(NeuralNetworkArchitecture<T> architecture, string modelPath, IDEFICS2Options options) |
Properties
| Property | Summary |
|---|---|
int DecoderEmbeddingDim { get; } |
|
int EmbeddingDimension { get; } |
|
int MaxGenerationLength { get; } |
Methods
| Method | Summary |
|---|---|
ModelMetadata<T> GetModelMetadata() |
|
Tensor<T> EncodeImage(Tensor<T> image) |
|
Tensor<T> GenerateFromImage(Tensor<T> image, string prompt) |
Generates text using IDEFICS2’s efficient 8B SigLIP+Mistral architecture. |