BLIP3<T>

Models & Types · AiDotNet.VisionLanguage.Generative

BLIP-3 (xGen-MM): scaled vision-language model with interleaved data and any-to-any generation.

Inheritance: NeuralNetworkBase<T> → VisionLanguageModelBase<T> → BLIP3<T>

Implements: IGenerativeVisionLanguageModel<T>

For Beginners

BLIP-3 (also called xGen-MM) is Salesforce’s scaled multimodal model that can understand images, answer questions about them, and generate both text and images from interleaved image-text inputs. It builds on BLIP-2’s Q-Former design with larger capacity and training on web-scraped interleaved data. Default values follow the original paper settings.

How It Works

BLIP-3/xGen-MM (Salesforce, 2024) scales the BLIP-2 architecture with interleaved image-text data (OBELICS), larger Q-Former capacity, and any-to-any generation capabilities.

References:

  • Paper: “xGen-MM (BLIP-3): A Family of Open Large Multimodal Models” (Salesforce, 2024)

Architecture layout: Triple-stream — vision encoder lives in Layers (so default Predict / TrainWithTape walk only it), Q-Former lives in a private auxiliary stream, decoder lives in another. Image+text generation goes through GenerateFromImage which walks all three streams; raw Predict returns the vision-only forward (matching the IVisualEncoder contract).

Example

var architecture = new NeuralNetworkArchitecture<double>(
    inputType: InputType.TwoDimensional,
    taskType: NeuralNetworkTaskType.ImageClassification,
    inputHeight: 224, inputWidth: 224, inputDepth: 3, outputSize: 512);
var trainModel = new BLIP3<double>(architecture, new BLIP3Options());

Constructors

Constructor Summary
BLIP3(NeuralNetworkArchitecture<T> architecture, BLIP3Options options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer)
BLIP3(NeuralNetworkArchitecture<T> architecture, string modelPath, BLIP3Options options)

Properties

Property Summary
int DecoderEmbeddingDim { get; }
int EmbeddingDimension { get; }
int MaxGenerationLength { get; }

Methods

Method Summary
ModelMetadata<T> GetModelMetadata()
Tensor<T> EncodeImage(Tensor<T> image)
Tensor<T> GenerateFromImage(Tensor<T> image, string prompt) Generates text using BLIP-3/xGen-MM’s scaled Q-Former architecture.