BLIP3<T>
Models & Types · AiDotNet.VisionLanguage.Generative
BLIP-3 (xGen-MM): scaled vision-language model with interleaved data and any-to-any generation.
Inheritance: NeuralNetworkBase<T> → VisionLanguageModelBase<T> → BLIP3<T>
Implements: IGenerativeVisionLanguageModel<T>
For Beginners
BLIP-3 (also called xGen-MM) is Salesforce’s scaled multimodal model that can understand images, answer questions about them, and generate both text and images from interleaved image-text inputs. It builds on BLIP-2’s Q-Former design with larger capacity and training on web-scraped interleaved data. Default values follow the original paper settings.
How It Works
BLIP-3/xGen-MM (Salesforce, 2024) scales the BLIP-2 architecture with interleaved image-text data (OBELICS), larger Q-Former capacity, and any-to-any generation capabilities.
References:
- Paper: “xGen-MM (BLIP-3): A Family of Open Large Multimodal Models” (Salesforce, 2024)
Architecture layout: Triple-stream — vision encoder lives in
Layers (so default Predict / TrainWithTape walk only it),
Q-Former lives in a private auxiliary stream, decoder lives in another. Image+text generation
goes through GenerateFromImage which walks all three streams; raw
Predict returns the vision-only forward (matching the IVisualEncoder contract).
Example
var architecture = new NeuralNetworkArchitecture<double>(
inputType: InputType.TwoDimensional,
taskType: NeuralNetworkTaskType.ImageClassification,
inputHeight: 224, inputWidth: 224, inputDepth: 3, outputSize: 512);
var trainModel = new BLIP3<double>(architecture, new BLIP3Options());
Constructors
| Constructor | Summary |
|---|---|
BLIP3(NeuralNetworkArchitecture<T> architecture, BLIP3Options options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
|
BLIP3(NeuralNetworkArchitecture<T> architecture, string modelPath, BLIP3Options options) |
Properties
| Property | Summary |
|---|---|
int DecoderEmbeddingDim { get; } |
|
int EmbeddingDimension { get; } |
|
int MaxGenerationLength { get; } |
Methods
| Method | Summary |
|---|---|
ModelMetadata<T> GetModelMetadata() |
|
Tensor<T> EncodeImage(Tensor<T> image) |
|
Tensor<T> GenerateFromImage(Tensor<T> image, string prompt) |
Generates text using BLIP-3/xGen-MM’s scaled Q-Former architecture. |