ViT<T>
Models & Types · AiDotNet.VisionLanguage.Encoders
Vision Transformer (ViT) that splits images into patches and processes them with a standard Transformer.
Inheritance: NeuralNetworkBase<T> → VisionLanguageModelBase<T> → ViT<T>
Implements: IVisualEncoder<T>
For Beginners
ViT (Vision Transformer) applies the Transformer architecture — originally designed for text — directly to images. It splits an image into a grid of fixed-size patches (e.g., 16x16 pixels), treats each patch like a “word”, and processes the sequence of patches through standard Transformer encoder layers. A special [CLS] token collects information from all patches for classification. Default values follow the original paper settings.
How It Works
ViT (Dosovitskiy et al., ICLR 2021) demonstrated that a pure Transformer applied directly to sequences of image patches can perform very well on image classification. An image is split into fixed-size patches (e.g., 16x16), each linearly embedded and processed by Transformer encoder layers. A learnable [CLS] token aggregates information for the final representation.
References:
- Paper: “An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale” (Dosovitskiy et al., ICLR 2021)
Example
// Create a ViT model for image classification via patch-based Transformer
// splitting images into 16x16 patches processed by Transformer layers
var architecture = new NeuralNetworkArchitecture<double>(
inputType: InputType.TwoDimensional,
taskType: NeuralNetworkTaskType.ImageClassification,
inputHeight: 224, inputWidth: 224, inputDepth: 3, outputSize: 512);
// ONNX inference mode with pre-trained model
var model = new ViT<double>(architecture, "vit.onnx");
// Training mode with native layers
var trainModel = new ViT<double>(architecture, new ViTOptions());
Constructors
| Constructor | Summary |
|---|---|
ViT(NeuralNetworkArchitecture<T> architecture, ViTOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
|
ViT(NeuralNetworkArchitecture<T> architecture, string modelPath, ViTOptions options) |
Properties
| Property | Summary |
|---|---|
int EmbeddingDimension { get; } |
Methods
| Method | Summary |
|---|---|
ModelMetadata<T> GetModelMetadata() |
|
Tensor<T> EncodeImage(Tensor<T> image) |
|
Tensor<T> ForwardForTraining(Tensor<T> input) |