KosmosImageResamplerLayer<T>
Layers · AiDotNet.NeuralNetworks.Layers
Turns image tokens into a fixed number of image embeddings for the language model.
Inheritance: LayerBase<T> → KosmosImageResamplerLayer<T>
Implements: IShapeContract
For Beginners
A fixed set of learned “questions” reads the image and summarizes it into the same small number of vectors every time, so an image always takes the same room in the text.
How It Works
There are two forms:
- KOSMOS-2 (
perceiver = false; referenceKosmos2ImageToTextProjection): project the
tokens to the text width. Then numLatents learned queries attend once, with no residual and no
LayerNorm, to [image tokens; queries].
- KOSMOS-1 (
perceiver = true): a Flamingo perceiver resampler (Alayrac et al. 2022). Each of
depth blocks runs latents += Attn(LN(latents), LN([x; latents])), then
latents += FFN(LN(latents)) with a 4x GELU MLP. A final LayerNorm follows.
Constructors
| Constructor | Summary |
|---|---|
KosmosImageResamplerLayer(int visionDim, int dim, int numLatents, int numHeads, int depth, bool perceiver) |
Creates the resampler. |
Properties
| Property | Summary |
|---|---|
bool SupportsTraining { get; } |
Methods
| Method | Summary |
|---|---|
Dictionary<string, string> GetParameterRoles() |
Returns parameter roles for per-role optimizer configuration (e.g., weight decay exemption for biases). |
IReadOnlyList<ILayer<T>> GetSubLayers() |
|
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank) |
|
IReadOnlyList<Tensor<T>> GetTrainableParameters() |
Returns all trainable parameter tensors marked with [TrainableParameter]. |
void ResetState() |
|
void SetTrainableParameters(IReadOnlyList<Tensor<T>> parameters) |
Replaces trainable parameter tensors (e.g., with ParameterBuffer views). |
void ZeroGrad() |
Clears all gradient fields discovered by convention ({paramName}Gradient). |