KosmosImageResamplerLayer<T>

Layers · AiDotNet.NeuralNetworks.Layers

Turns image tokens into a fixed number of image embeddings for the language model.

Inheritance: LayerBase<T> → KosmosImageResamplerLayer<T>

Implements: IShapeContract

For Beginners

A fixed set of learned “questions” reads the image and summarizes it into the same small number of vectors every time, so an image always takes the same room in the text.

How It Works

There are two forms:

  • KOSMOS-2 (perceiver = false; reference Kosmos2ImageToTextProjection): project the

tokens to the text width. Then numLatents learned queries attend once, with no residual and no LayerNorm, to [image tokens; queries].

  • KOSMOS-1 (perceiver = true): a Flamingo perceiver resampler (Alayrac et al. 2022). Each of

depth blocks runs latents += Attn(LN(latents), LN([x; latents])), then latents += FFN(LN(latents)) with a 4x GELU MLP. A final LayerNorm follows.

Constructors

Constructor Summary
KosmosImageResamplerLayer(int visionDim, int dim, int numLatents, int numHeads, int depth, bool perceiver) Creates the resampler.

Properties

Property Summary
bool SupportsTraining { get; }

Methods

Method Summary
Dictionary<string, string> GetParameterRoles() Returns parameter roles for per-role optimizer configuration (e.g., weight decay exemption for biases).
IReadOnlyList<ILayer<T>> GetSubLayers()
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank)
IReadOnlyList<Tensor<T>> GetTrainableParameters() Returns all trainable parameter tensors marked with [TrainableParameter].
void ResetState()
void SetTrainableParameters(IReadOnlyList<Tensor<T>> parameters) Replaces trainable parameter tensors (e.g., with ParameterBuffer views).
void ZeroGrad() Clears all gradient fields discovered by convention ({paramName}Gradient).