Blip2NeuralNetwork<T>

Models & Types · AiDotNet.NeuralNetworks

BLIP-2 (Bootstrapped Language-Image Pre-training 2) neural network for vision-language tasks.

Inheritance: NeuralNetworkBase<T> → DeclaredModelLayoutBase<T> → MultimodalModelLayoutBase<T> → Blip2NeuralNetwork<T>

Implements: IBlip2Model<T>

For Beginners

BLIP-2 is the next evolution of vision-language models!

Architecture overview:

  1. Frozen Image Encoder (ViT-G): Extracts image patch features
  2. Q-Former: Small trainable transformer that bridges vision and language
  • Uses 32 learnable “query” tokens
  • Queries attend to image features via cross-attention
  • Output goes to the language model
  1. Frozen LLM (OPT/Flan-T5): Generates text from visual features

Why this architecture is brilliant:

  • Only trains the small Q-Former (~188M parameters)
  • Image encoder stays frozen (no GPU memory for gradients)
  • LLM stays frozen (can use huge 66B+ models)
  • Much cheaper to train than end-to-end models

Training stages:

  1. Vision-Language Representation Learning (Q-Former + ViT)
  2. Vision-to-Language Generative Learning (Q-Former + LLM)

How It Works

BLIP-2 uses a Q-Former (Querying Transformer) to efficiently bridge frozen image encoders with frozen large language models. The Q-Former uses learnable query tokens that interact with frozen image features through cross-attention layers.

Example

var architecture = new NeuralNetworkArchitecture<float>(inputFeatures: 8, outputSize: 4);
var image = Tensor<float>.CreateRandom(new[] { 1, 3, 224, 224 });
var trainX = Tensor<float>.CreateRandom(4, 8);
var trainY = Tensor<float>.CreateRandom(4, 2);
var result = new AiModelBuilder<float, Tensor<float>, Tensor<float>>()
    .ConfigureModel(new Blip2NeuralNetwork<float>(architecture))
    .Build(trainX, trainY);
var output = result.Predict(image);

Constructors

Constructor Summary
Blip2NeuralNetwork(NeuralNetworkArchitecture<T> architecture, Blip2Options options, ITokenizer tokenizer, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer, ILossFunction<T> lossFunction) Creates a BLIP-2 network using native library layers.
Blip2NeuralNetwork(NeuralNetworkArchitecture<T> architecture, string visionEncoderPath, string qformerPath, string languageModelPath, ITokenizer tokenizer, Blip2Options options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer, ILossFunction<T> lossFunction) Creates a BLIP-2 network using pretrained ONNX models.

Properties

Property Summary
LanguageModelBackbone LanguageModelBackbone { get; }
int EmbeddingDimension { get; }
int ImageSize { get; }
int MaxSequenceLength { get; }
int NumQueryTokens { get; } Gets the configured visual-query count, validated against an image-query export.

Methods

Method Summary
Dictionary<string, T> ZeroShotClassify(Tensor<T> image, IEnumerable<string> classLabels)
Dictionary<string, T> ZeroShotClassify(Tensor<T> image, IEnumerable<string> classLabels, bool useItm) Performs zero-shot image classification with optional ITM scoring.
Dictionary<string, T> ZeroShotClassify(double[] imageData, IEnumerable<string> labels)
IEnumerable<ValueTuple<int, T>> RetrieveImages(string query, IEnumerable<Tensor<T>> imageFeatures, int topK, bool useItmReranking, int rerankTopN)
IEnumerable<ValueTuple<string, T>> GenerateCaptions(Tensor<T> image, int numCaptions, string prompt, int maxLength, double temperature, double topP)
IEnumerable<Vector<T>> GetImageEmbeddings(IEnumerable<Tensor<T>> images)
IEnumerable<Vector<T>> GetTextEmbeddings(IEnumerable<string> texts)
Matrix<T> EncodeImageBatch(IEnumerable<double[]> imageDataBatch)
Matrix<T> EncodeTextBatch(IEnumerable<string> texts)
ModelMetadata<T> GetModelMetadata()
T ComputeContrastiveSimilarity(Tensor<T> image, string text)
T ComputeImageTextMatch(Tensor<T> image, string text)
T ComputeSimilarity(Vector<T> textEmbedding, Vector<T> imageEmbedding)
Task<Matrix<T>> EmbedBatchAsync(IEnumerable<string> texts)
Task<Vector<T>> EmbedAsync(string text)
Tensor<T> ExtractQFormerFeatures(Tensor<T> image)
Tensor<T> Forward(Tensor<T> input) Forward pass through Q-Former and vision encoder.
Tensor<T> ForwardForTraining(Tensor<T> input)
Vector<T> EncodeImage(double[] imageData)
Vector<T> EncodeText(string text)
Vector<T> GetImageEmbedding(Tensor<T> image)
Vector<T> GetTextEmbedding(string text)
Vector<T> GroundText(Tensor<T> image, string description)
string AnswerQuestion(Tensor<T> image, string question, int maxLength)
string GenerateCaption(Tensor<T> image, string prompt, int maxLength, int numBeams, double temperature)
string GenerateWithInstruction(Tensor<T> image, string instruction, int maxLength)