ClipVisionTransformerLayer<T>
Layers · AiDotNet.NeuralNetworks.Layers
CLIP’s vision transformer (Radford et al. 2021) as used by KOSMOS (reference Kosmos2VisionTransformer):
- A
patch x patchstride-patchconvolution, plus a class token and learned position
embeddings.
- pre_layrnorm, then pre-LN encoder blocks: attention and a QuickGELU MLP at 4x width, each with a
residual.
- post_layernorm over ALL tokens, then L2 normalization per token. KOSMOS feeds every token, not
only the pooled class token, to its resampler.
Inheritance: LayerBase<T> → ClipVisionTransformerLayer<T>
Implements: IShapeContract
For Beginners
This cuts the image into small squares, turns each into a vector, and lets them exchange information, so each vector describes its square in the context of the whole image.
How It Works
Input: one image [3, H, W] (or [1, 3, H, W]) with H = W = imageSize. Output:
[1 + (imageSize / patch)^2, hidden]. The patch convolution here carries a bias; CLIP’s has none.
Constructors
| Constructor | Summary |
|---|---|
ClipVisionTransformerLayer(int hidden, int numLayers, int numHeads, int patchSize, int imageSize) |
Creates the encoder. |
Properties
| Property | Summary |
|---|---|
bool SupportsTraining { get; } |
Methods
| Method | Summary |
|---|---|
Dictionary<string, string> GetParameterRoles() |
Returns parameter roles for per-role optimizer configuration (e.g., weight decay exemption for biases). |
IReadOnlyList<ILayer<T>> GetSubLayers() |
|
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank) |
|
IReadOnlyList<Tensor<T>> GetTrainableParameters() |
Returns all trainable parameter tensors marked with [TrainableParameter]. |
void ResetState() |
|
void SetTrainableParameters(IReadOnlyList<Tensor<T>> parameters) |
Replaces trainable parameter tensors (e.g., with ParameterBuffer views). |
void ZeroGrad() |
Clears all gradient fields discovered by convention ({paramName}Gradient). |