IDEFICS2<T>

Models & Types · AiDotNet.VisionLanguage.Generative

IDEFICS2: 8B efficient VLM with SigLIP encoder and Mistral-7B decoder.

Inheritance: NeuralNetworkBase<T> → VisionLanguageModelBase<T> → IDEFICS2<T>

Implements: IGenerativeVisionLanguageModel<T>

For Beginners

IDEFICS2 is a much more efficient 8 billion parameter successor to the original 80B IDEFICS. It replaces the OpenCLIP encoder with SigLIP and uses Mistral-7B as the language backbone, introducing learned perceiver pooling and native resolution image processing that splits images into sub-images for better document understanding. Default values follow the original paper settings.

How It Works

IDEFICS2 (Laurencon et al., 2024) is an 8B parameter efficient VLM that replaces the OpenCLIP encoder with SigLIP and uses Mistral-7B as the language backbone. It introduces a learned perceiver pooling strategy and native resolution image processing via sub-image splitting for improved document understanding.

References:

  • Paper: “What matters when building vision-language models?” (Laurencon et al., 2024)

Example

var architecture = new NeuralNetworkArchitecture<double>(
    inputType: InputType.TwoDimensional,
    taskType: NeuralNetworkTaskType.ImageClassification,
    inputHeight: 224, inputWidth: 224, inputDepth: 3, outputSize: 512);
var trainModel = new IDEFICS2<double>(architecture, new IDEFICS2Options());

Constructors

Constructor Summary
IDEFICS2(NeuralNetworkArchitecture<T> architecture, IDEFICS2Options options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer)
IDEFICS2(NeuralNetworkArchitecture<T> architecture, string modelPath, IDEFICS2Options options)

Properties

Property Summary
int DecoderEmbeddingDim { get; }
int EmbeddingDimension { get; }
int MaxGenerationLength { get; }

Methods

Method Summary
ModelMetadata<T> GetModelMetadata()
Tensor<T> EncodeImage(Tensor<T> image)
Tensor<T> GenerateFromImage(Tensor<T> image, string prompt) Generates text using IDEFICS2’s efficient 8B SigLIP+Mistral architecture.