KOSMOS2<T>

Models & Types · AiDotNet.VisionLanguage.Generative

KOSMOS-2: grounded multimodal large language model with text spans linked to bounding boxes.

Inheritance: NeuralNetworkBase<T> → VisionLanguageModelBase<T> → KOSMOS2<T>

Implements: IGenerativeVisionLanguageModel<T>

For Beginners

KOSMOS-2 extends KOSMOS-1 with visual grounding — the ability to link words in generated text to specific bounding box locations in the image. It uses special location tokens to encode bounding box coordinates, enabling the model to output phrases like “the dog x1,y1,x2,y2” that point to objects in the image. Default values follow the original paper settings.

How It Works

KOSMOS-2 (Peng et al., 2023) extends KOSMOS-1 with grounding capabilities by linking text spans to bounding box locations in the image. Special location tokens encode bounding box coordinates, enabling the model to output referring expressions grounded in the visual input. The architecture retains the causal multimodal LM design with visual tokens embedded directly in the sequence.

References:

  • Paper: “Kosmos-2: Grounding Multimodal Large Language Models to the World” (Peng et al., 2023)

Example

var architecture = new NeuralNetworkArchitecture<double>(
    inputType: InputType.TwoDimensional,
    taskType: NeuralNetworkTaskType.ImageClassification,
    inputHeight: 224, inputWidth: 224, inputDepth: 3, outputSize: 512);
var trainModel = new KOSMOS2<double>(architecture, new KOSMOS2Options());

Constructors

Constructor Summary
KOSMOS2(NeuralNetworkArchitecture<T> architecture, KOSMOS2Options options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) Creates a trainable native model.
KOSMOS2(NeuralNetworkArchitecture<T> architecture, string modelPath, KOSMOS2Options options) Creates a trainable native model.

Properties

Property Summary
int DecoderEmbeddingDim { get; }
int EmbeddingDimension { get; }
int MaxGenerationLength { get; }

Methods

Method Summary
ModelMetadata<T> GetModelMetadata()
Tensor<T> EncodeImage(Tensor<T> image) The NumImageTokens image embeddings the decoder reads, [NumImageTokens, DecoderDim].
Tensor<T> GenerateFromImage(Tensor<T> image, string prompt) Greedy generation after <s> <image> [image] </image> and the prompt; returns the generated token ids (ending at EOS or after MaxGenerationLength tokens).
Tensor<T> PredictTokens(Tensor<T> image, IReadOnlyList<int> promptIds) Next-token logits [NumImageTokens + 3 + prompt, VocabSize] for an image and prompt token ids.
void TrainCaption(Tensor<T> image, IReadOnlyList<int> captionIds) Trains on one image and its caption token ids (next-token cross-entropy on the caption).