GIT<T>

Models & Types · AiDotNet.VisionLanguage.Generative

GIT (Generative Image-to-Text): simple ViT encoder + autoregressive text decoder.

Inheritance: NeuralNetworkBase<T> → VisionLanguageModelBase<T> → GIT<T>

Implements: IGenerativeVisionLanguageModel<T>

For Beginners

GIT (Generative Image-to-Text) is a deliberately simple model that connects a pre-trained image encoder (ViT) to a text decoder through just a linear projection layer. Despite its simplicity — no Q-Former, perceiver, or cross-attention — it achieves strong results on captioning and VQA by scaling data and model size. Default values follow the original paper settings.

How It Works

GIT (Wang et al., TMLR 2022) uses a simple architecture: a contrastive pre-trained image encoder (ViT) connected to a text decoder via a linear projection. The decoder generates captions autoregressively, conditioned on the projected visual features.

References:

  • Paper: “GIT: A Generative Image-to-text Transformer for Vision and Language” (Wang et al., TMLR 2022)

Example

// Create a GIT model for simple image-to-text generation
// using ViT encoder + linear projection + autoregressive decoder
var architecture = new NeuralNetworkArchitecture<double>(
    inputType: InputType.TwoDimensional,
    taskType: NeuralNetworkTaskType.ImageClassification,
    inputHeight: 224, inputWidth: 224, inputDepth: 3, outputSize: 512);
            
// ONNX inference mode with pre-trained model
var model = new GIT<double>(architecture, "git.onnx");
            
// Training mode with native layers
var trainModel = new GIT<double>(architecture, new GITOptions());

Constructors

Constructor Summary
GIT(NeuralNetworkArchitecture<T> architecture, GITOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer)
GIT(NeuralNetworkArchitecture<T> architecture, string modelPath, GITOptions options)

Properties

Property Summary
int DecoderEmbeddingDim { get; }
int EmbeddingDimension { get; }
int MaxGenerationLength { get; }

Methods

Method Summary
ModelMetadata<T> GetModelMetadata()
Tensor<T> EncodeImage(Tensor<T> image)
Tensor<T> GenerateFromImage(Tensor<T> image, string prompt) Generates text using GIT’s simple concatenation architecture.