Blip2NeuralNetwork<T>
Models & Types · AiDotNet.NeuralNetworks
BLIP-2 (Bootstrapped Language-Image Pre-training 2) neural network for vision-language tasks.
Inheritance: NeuralNetworkBase<T> → DeclaredModelLayoutBase<T> → MultimodalModelLayoutBase<T> → Blip2NeuralNetwork<T>
Implements: IBlip2Model<T>
For Beginners
BLIP-2 is the next evolution of vision-language models!
Architecture overview:
- Frozen Image Encoder (ViT-G): Extracts image patch features
- Q-Former: Small trainable transformer that bridges vision and language
- Uses 32 learnable “query” tokens
- Queries attend to image features via cross-attention
- Output goes to the language model
- Frozen LLM (OPT/Flan-T5): Generates text from visual features
Why this architecture is brilliant:
- Only trains the small Q-Former (~188M parameters)
- Image encoder stays frozen (no GPU memory for gradients)
- LLM stays frozen (can use huge 66B+ models)
- Much cheaper to train than end-to-end models
Training stages:
- Vision-Language Representation Learning (Q-Former + ViT)
- Vision-to-Language Generative Learning (Q-Former + LLM)
How It Works
BLIP-2 uses a Q-Former (Querying Transformer) to efficiently bridge frozen image encoders with frozen large language models. The Q-Former uses learnable query tokens that interact with frozen image features through cross-attention layers.
Example
var architecture = new NeuralNetworkArchitecture<float>(inputFeatures: 8, outputSize: 4);
var image = Tensor<float>.CreateRandom(new[] { 1, 3, 224, 224 });
var trainX = Tensor<float>.CreateRandom(4, 8);
var trainY = Tensor<float>.CreateRandom(4, 2);
var result = new AiModelBuilder<float, Tensor<float>, Tensor<float>>()
.ConfigureModel(new Blip2NeuralNetwork<float>(architecture))
.Build(trainX, trainY);
var output = result.Predict(image);
Constructors
| Constructor | Summary |
|---|---|
Blip2NeuralNetwork(NeuralNetworkArchitecture<T> architecture, Blip2Options options, ITokenizer tokenizer, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer, ILossFunction<T> lossFunction) |
Creates a BLIP-2 network using native library layers. |
Blip2NeuralNetwork(NeuralNetworkArchitecture<T> architecture, string visionEncoderPath, string qformerPath, string languageModelPath, ITokenizer tokenizer, Blip2Options options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer, ILossFunction<T> lossFunction) |
Creates a BLIP-2 network using pretrained ONNX models. |
Properties
| Property | Summary |
|---|---|
LanguageModelBackbone LanguageModelBackbone { get; } |
|
int EmbeddingDimension { get; } |
|
int ImageSize { get; } |
|
int MaxSequenceLength { get; } |
|
int NumQueryTokens { get; } |
Gets the configured visual-query count, validated against an image-query export. |
Methods
| Method | Summary |
|---|---|
Dictionary<string, T> ZeroShotClassify(Tensor<T> image, IEnumerable<string> classLabels) |
|
Dictionary<string, T> ZeroShotClassify(Tensor<T> image, IEnumerable<string> classLabels, bool useItm) |
Performs zero-shot image classification with optional ITM scoring. |
Dictionary<string, T> ZeroShotClassify(double[] imageData, IEnumerable<string> labels) |
|
IEnumerable<ValueTuple<int, T>> RetrieveImages(string query, IEnumerable<Tensor<T>> imageFeatures, int topK, bool useItmReranking, int rerankTopN) |
|
IEnumerable<ValueTuple<string, T>> GenerateCaptions(Tensor<T> image, int numCaptions, string prompt, int maxLength, double temperature, double topP) |
|
IEnumerable<Vector<T>> GetImageEmbeddings(IEnumerable<Tensor<T>> images) |
|
IEnumerable<Vector<T>> GetTextEmbeddings(IEnumerable<string> texts) |
|
Matrix<T> EncodeImageBatch(IEnumerable<double[]> imageDataBatch) |
|
Matrix<T> EncodeTextBatch(IEnumerable<string> texts) |
|
ModelMetadata<T> GetModelMetadata() |
|
T ComputeContrastiveSimilarity(Tensor<T> image, string text) |
|
T ComputeImageTextMatch(Tensor<T> image, string text) |
|
T ComputeSimilarity(Vector<T> textEmbedding, Vector<T> imageEmbedding) |
|
Task<Matrix<T>> EmbedBatchAsync(IEnumerable<string> texts) |
|
Task<Vector<T>> EmbedAsync(string text) |
|
Tensor<T> ExtractQFormerFeatures(Tensor<T> image) |
|
Tensor<T> Forward(Tensor<T> input) |
Forward pass through Q-Former and vision encoder. |
Tensor<T> ForwardForTraining(Tensor<T> input) |
|
Vector<T> EncodeImage(double[] imageData) |
|
Vector<T> EncodeText(string text) |
|
Vector<T> GetImageEmbedding(Tensor<T> image) |
|
Vector<T> GetTextEmbedding(string text) |
|
Vector<T> GroundText(Tensor<T> image, string description) |
|
string AnswerQuestion(Tensor<T> image, string question, int maxLength) |
|
string GenerateCaption(Tensor<T> image, string prompt, int maxLength, int numBeams, double temperature) |
|
string GenerateWithInstruction(Tensor<T> image, string instruction, int maxLength) |