ViT<T>

Models & Types · AiDotNet.VisionLanguage.Encoders

Vision Transformer (ViT) that splits images into patches and processes them with a standard Transformer.

Inheritance: NeuralNetworkBase<T> → VisionLanguageModelBase<T> → ViT<T>

Implements: IVisualEncoder<T>

For Beginners

ViT (Vision Transformer) applies the Transformer architecture — originally designed for text — directly to images. It splits an image into a grid of fixed-size patches (e.g., 16x16 pixels), treats each patch like a “word”, and processes the sequence of patches through standard Transformer encoder layers. A special [CLS] token collects information from all patches for classification. Default values follow the original paper settings.

How It Works

ViT (Dosovitskiy et al., ICLR 2021) demonstrated that a pure Transformer applied directly to sequences of image patches can perform very well on image classification. An image is split into fixed-size patches (e.g., 16x16), each linearly embedded and processed by Transformer encoder layers. A learnable [CLS] token aggregates information for the final representation.

References:

  • Paper: “An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale” (Dosovitskiy et al., ICLR 2021)

Example

// Create a ViT model for image classification via patch-based Transformer
// splitting images into 16x16 patches processed by Transformer layers
var architecture = new NeuralNetworkArchitecture<double>(
    inputType: InputType.TwoDimensional,
    taskType: NeuralNetworkTaskType.ImageClassification,
    inputHeight: 224, inputWidth: 224, inputDepth: 3, outputSize: 512);
            
// ONNX inference mode with pre-trained model
var model = new ViT<double>(architecture, "vit.onnx");
            
// Training mode with native layers
var trainModel = new ViT<double>(architecture, new ViTOptions());

Constructors

Constructor Summary
ViT(NeuralNetworkArchitecture<T> architecture, ViTOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer)
ViT(NeuralNetworkArchitecture<T> architecture, string modelPath, ViTOptions options)

Properties

Property Summary
int EmbeddingDimension { get; }

Methods

Method Summary
ModelMetadata<T> GetModelMetadata()
Tensor<T> EncodeImage(Tensor<T> image)
Tensor<T> ForwardForTraining(Tensor<T> input)