KOSMOS1<T>
Models & Types · AiDotNet.VisionLanguage.Generative
KOSMOS-1: multimodal large language model with visual tokens embedded in causal LM.
Inheritance: NeuralNetworkBase<T> → VisionLanguageModelBase<T> → KOSMOS1<T>
Implements: IGenerativeVisionLanguageModel<T>
For Beginners
KOSMOS-1 from Microsoft embeds image features directly into a causal language model as if they were text tokens. Image patches from a CLIP ViT are linearly projected into the same embedding space as text, then the combined image-text sequence is processed by a standard causal transformer for unified multimodal understanding and generation. Default values follow the original paper settings.
How It Works
KOSMOS-1 (Huang et al., 2023) is a multimodal large language model that embeds visual tokens directly into a causal language model. Image features from a CLIP ViT are linearly projected into the same embedding space as text tokens, then the combined sequence is processed by a causal transformer decoder for unified multimodal understanding and generation.
References:
- Paper: “Language Is Not All You Need: Aligning Perception with Language Models” (Huang et al., 2023)
Architecture layout: the vision encoder, the image resampler and the causal MAGNETO decoder are all in
Layers, so each is trained, serialized and counted. They are not a sequential
pass: Predict runs them directly and returns the next-token logits after the image block, and
GenerateFromImage decodes a caption.
Example
var architecture = new NeuralNetworkArchitecture<double>(
inputType: InputType.TwoDimensional,
taskType: NeuralNetworkTaskType.ImageClassification,
inputHeight: 224, inputWidth: 224, inputDepth: 3, outputSize: 512);
var trainModel = new KOSMOS1<double>(architecture, new KOSMOS1Options());
Constructors
| Constructor | Summary |
|---|---|
KOSMOS1(NeuralNetworkArchitecture<T> architecture, KOSMOS1Options options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
Creates a trainable native model. |
KOSMOS1(NeuralNetworkArchitecture<T> architecture, string modelPath, KOSMOS1Options options) |
Creates a trainable native model. |
Properties
| Property | Summary |
|---|---|
int DecoderEmbeddingDim { get; } |
|
int EmbeddingDimension { get; } |
|
int MaxGenerationLength { get; } |
Methods
| Method | Summary |
|---|---|
ModelMetadata<T> GetModelMetadata() |
|
Tensor<T> EncodeImage(Tensor<T> image) |
The NumImageTokens image embeddings the decoder reads, [NumImageTokens, DecoderDim]. |
Tensor<T> GenerateFromImage(Tensor<T> image, string prompt) |
Greedy generation after <s> <image> [image] </image> and the prompt; returns the generated token ids (ending at EOS or after MaxGenerationLength tokens). |
Tensor<T> PredictTokens(Tensor<T> image, IReadOnlyList<int> promptIds) |
Next-token logits [NumImageTokens + 3 + prompt, VocabSize] for an image and prompt token ids. |
void TrainCaption(Tensor<T> image, IReadOnlyList<int> captionIds) |
Trains on one image and its caption token ids (next-token cross-entropy on the caption). |