KOSMOS1<T>

Models & Types · AiDotNet.VisionLanguage.Generative

KOSMOS-1: multimodal large language model with visual tokens embedded in causal LM.

Inheritance: NeuralNetworkBase<T> → VisionLanguageModelBase<T> → KOSMOS1<T>

Implements: IGenerativeVisionLanguageModel<T>

For Beginners

KOSMOS-1 from Microsoft embeds image features directly into a causal language model as if they were text tokens. Image patches from a CLIP ViT are linearly projected into the same embedding space as text, then the combined image-text sequence is processed by a standard causal transformer for unified multimodal understanding and generation. Default values follow the original paper settings.

How It Works

KOSMOS-1 (Huang et al., 2023) is a multimodal large language model that embeds visual tokens directly into a causal language model. Image features from a CLIP ViT are linearly projected into the same embedding space as text tokens, then the combined sequence is processed by a causal transformer decoder for unified multimodal understanding and generation.

References:

  • Paper: “Language Is Not All You Need: Aligning Perception with Language Models” (Huang et al., 2023)

Architecture layout: the vision encoder, the image resampler and the causal MAGNETO decoder are all in Layers, so each is trained, serialized and counted. They are not a sequential pass: Predict runs them directly and returns the next-token logits after the image block, and GenerateFromImage decodes a caption.

Example

var architecture = new NeuralNetworkArchitecture<double>(
    inputType: InputType.TwoDimensional,
    taskType: NeuralNetworkTaskType.ImageClassification,
    inputHeight: 224, inputWidth: 224, inputDepth: 3, outputSize: 512);
var trainModel = new KOSMOS1<double>(architecture, new KOSMOS1Options());

Constructors

Constructor Summary
KOSMOS1(NeuralNetworkArchitecture<T> architecture, KOSMOS1Options options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) Creates a trainable native model.
KOSMOS1(NeuralNetworkArchitecture<T> architecture, string modelPath, KOSMOS1Options options) Creates a trainable native model.

Properties

Property Summary
int DecoderEmbeddingDim { get; }
int EmbeddingDimension { get; }
int MaxGenerationLength { get; }

Methods

Method Summary
ModelMetadata<T> GetModelMetadata()
Tensor<T> EncodeImage(Tensor<T> image) The NumImageTokens image embeddings the decoder reads, [NumImageTokens, DecoderDim].
Tensor<T> GenerateFromImage(Tensor<T> image, string prompt) Greedy generation after <s> <image> [image] </image> and the prompt; returns the generated token ids (ending at EOS or after MaxGenerationLength tokens).
Tensor<T> PredictTokens(Tensor<T> image, IReadOnlyList<int> promptIds) Next-token logits [NumImageTokens + 3 + prompt, VocabSize] for an image and prompt token ids.
void TrainCaption(Tensor<T> image, IReadOnlyList<int> captionIds) Trains on one image and its caption token ids (next-token cross-entropy on the caption).