KOSMOS2<T>
Models & Types · AiDotNet.VisionLanguage.Generative
KOSMOS-2: grounded multimodal large language model with text spans linked to bounding boxes.
Inheritance: NeuralNetworkBase<T> → VisionLanguageModelBase<T> → KOSMOS2<T>
Implements: IGenerativeVisionLanguageModel<T>
For Beginners
KOSMOS-2 extends KOSMOS-1 with visual grounding — the ability
to link words in generated text to specific bounding box locations in the image. It uses
special location tokens to encode bounding box coordinates, enabling the model to output
phrases like “the dog
How It Works
KOSMOS-2 (Peng et al., 2023) extends KOSMOS-1 with grounding capabilities by linking text spans to bounding box locations in the image. Special location tokens encode bounding box coordinates, enabling the model to output referring expressions grounded in the visual input. The architecture retains the causal multimodal LM design with visual tokens embedded directly in the sequence.
References:
- Paper: “Kosmos-2: Grounding Multimodal Large Language Models to the World” (Peng et al., 2023)
Example
var architecture = new NeuralNetworkArchitecture<double>(
inputType: InputType.TwoDimensional,
taskType: NeuralNetworkTaskType.ImageClassification,
inputHeight: 224, inputWidth: 224, inputDepth: 3, outputSize: 512);
var trainModel = new KOSMOS2<double>(architecture, new KOSMOS2Options());
Constructors
| Constructor | Summary |
|---|---|
KOSMOS2(NeuralNetworkArchitecture<T> architecture, KOSMOS2Options options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
Creates a trainable native model. |
KOSMOS2(NeuralNetworkArchitecture<T> architecture, string modelPath, KOSMOS2Options options) |
Creates a trainable native model. |
Properties
| Property | Summary |
|---|---|
int DecoderEmbeddingDim { get; } |
|
int EmbeddingDimension { get; } |
|
int MaxGenerationLength { get; } |
Methods
| Method | Summary |
|---|---|
ModelMetadata<T> GetModelMetadata() |
|
Tensor<T> EncodeImage(Tensor<T> image) |
The NumImageTokens image embeddings the decoder reads, [NumImageTokens, DecoderDim]. |
Tensor<T> GenerateFromImage(Tensor<T> image, string prompt) |
Greedy generation after <s> <image> [image] </image> and the prompt; returns the generated token ids (ending at EOS or after MaxGenerationLength tokens). |
Tensor<T> PredictTokens(Tensor<T> image, IReadOnlyList<int> promptIds) |
Next-token logits [NumImageTokens + 3 + prompt, VocabSize] for an image and prompt token ids. |
void TrainCaption(Tensor<T> image, IReadOnlyList<int> captionIds) |
Trains on one image and its caption token ids (next-token cross-entropy on the caption). |