ClipVisionTransformerLayer<T>

Layers · AiDotNet.NeuralNetworks.Layers

CLIP’s vision transformer (Radford et al. 2021) as used by KOSMOS (reference Kosmos2VisionTransformer):

  • A patch x patch stride-patch convolution, plus a class token and learned position

embeddings.

  • pre_layrnorm, then pre-LN encoder blocks: attention and a QuickGELU MLP at 4x width, each with a

residual.

  • post_layernorm over ALL tokens, then L2 normalization per token. KOSMOS feeds every token, not

only the pooled class token, to its resampler.

Inheritance: LayerBase<T> → ClipVisionTransformerLayer<T>

Implements: IShapeContract

For Beginners

This cuts the image into small squares, turns each into a vector, and lets them exchange information, so each vector describes its square in the context of the whole image.

How It Works

Input: one image [3, H, W] (or [1, 3, H, W]) with H = W = imageSize. Output: [1 + (imageSize / patch)^2, hidden]. The patch convolution here carries a bias; CLIP’s has none.

Constructors

Constructor Summary
ClipVisionTransformerLayer(int hidden, int numLayers, int numHeads, int patchSize, int imageSize) Creates the encoder.

Properties

Property Summary
bool SupportsTraining { get; }

Methods

Method Summary
Dictionary<string, string> GetParameterRoles() Returns parameter roles for per-role optimizer configuration (e.g., weight decay exemption for biases).
IReadOnlyList<ILayer<T>> GetSubLayers()
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank)
IReadOnlyList<Tensor<T>> GetTrainableParameters() Returns all trainable parameter tensors marked with [TrainableParameter].
void ResetState()
void SetTrainableParameters(IReadOnlyList<Tensor<T>> parameters) Replaces trainable parameter tensors (e.g., with ParameterBuffer views).
void ZeroGrad() Clears all gradient fields discovered by convention ({paramName}Gradient).