GIT<T>
Models & Types · AiDotNet.VisionLanguage.Generative
GIT (Generative Image-to-Text): simple ViT encoder + autoregressive text decoder.
Inheritance: NeuralNetworkBase<T> → VisionLanguageModelBase<T> → GIT<T>
Implements: IGenerativeVisionLanguageModel<T>
For Beginners
GIT (Generative Image-to-Text) is a deliberately simple model that connects a pre-trained image encoder (ViT) to a text decoder through just a linear projection layer. Despite its simplicity — no Q-Former, perceiver, or cross-attention — it achieves strong results on captioning and VQA by scaling data and model size. Default values follow the original paper settings.
How It Works
GIT (Wang et al., TMLR 2022) uses a simple architecture: a contrastive pre-trained image encoder (ViT) connected to a text decoder via a linear projection. The decoder generates captions autoregressively, conditioned on the projected visual features.
References:
- Paper: “GIT: A Generative Image-to-text Transformer for Vision and Language” (Wang et al., TMLR 2022)
Example
// Create a GIT model for simple image-to-text generation
// using ViT encoder + linear projection + autoregressive decoder
var architecture = new NeuralNetworkArchitecture<double>(
inputType: InputType.TwoDimensional,
taskType: NeuralNetworkTaskType.ImageClassification,
inputHeight: 224, inputWidth: 224, inputDepth: 3, outputSize: 512);
// ONNX inference mode with pre-trained model
var model = new GIT<double>(architecture, "git.onnx");
// Training mode with native layers
var trainModel = new GIT<double>(architecture, new GITOptions());
Constructors
| Constructor | Summary |
|---|---|
GIT(NeuralNetworkArchitecture<T> architecture, GITOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
|
GIT(NeuralNetworkArchitecture<T> architecture, string modelPath, GITOptions options) |
Properties
| Property | Summary |
|---|---|
int DecoderEmbeddingDim { get; } |
|
int EmbeddingDimension { get; } |
|
int MaxGenerationLength { get; } |
Methods
| Method | Summary |
|---|---|
ModelMetadata<T> GetModelMetadata() |
|
Tensor<T> EncodeImage(Tensor<T> image) |
|
Tensor<T> GenerateFromImage(Tensor<T> image, string prompt) |
Generates text using GIT’s simple concatenation architecture. |