VisionTransformer<T>

Models & Types · AiDotNet.NeuralNetworks

Implements the Vision Transformer (ViT) architecture for image classification tasks.

Inheritance: NeuralNetworkBase<T> → DeclaredModelLayoutBase<T> → ImageClassifierModelLayoutBase<T> → VisionTransformer<T>

For Beginners

The Vision Transformer (ViT) is a modern approach to understanding images using transformers.

Unlike traditional neural networks that process images pixel by pixel or with sliding windows (convolutions), ViT treats an image like a sentence of words:

  • First, it cuts the image into small square patches (like breaking a sentence into words)
  • Each patch gets converted to a numerical representation (like word embeddings)
  • Position information is added so the model knows where each patch came from
  • A special classification token is added to gather information about the whole image
  • Transformer layers process all patches together, learning relationships between them
  • Finally, the classification token’s output is used to predict the image class

This approach has been very successful and often outperforms traditional convolutional neural networks, especially when trained on large datasets.

How It Works

The Vision Transformer applies transformer architecture, originally designed for natural language processing, to computer vision tasks. It divides images into fixed-size patches, linearly embeds them, adds positional embeddings, and processes the sequence through transformer encoder layers.

Example

var image = Tensor<float>.CreateRandom(new[] { 1, 3, 224, 224 });
var trainX = Tensor<float>.CreateRandom(4, 8);
var trainY = Tensor<float>.CreateRandom(4, 2);
var result = new AiModelBuilder<float, Tensor<float>, Tensor<float>>()
    .ConfigureModel(new VisionTransformer<float>())
    .Build(trainX, trainY);
var output = result.Predict(image);

Constructors

Constructor Summary
VisionTransformer() Creates a new Vision Transformer with the specified configuration.
VisionTransformer(NeuralNetworkArchitecture<T> architecture, int imageHeight, int imageWidth, int channels, int patchSize, int numClasses, ILossFunction<T> lossFunction, VisionTransformerOptions options)

Properties

Property Summary
bool SupportsTraining { get; } Indicates whether this network supports training.

Methods

Method Summary
ModelMetadata<T> GetModelMetadata() Gets the model metadata.
Tensor<T> ForwardForTraining(Tensor<T> input) Tape-tracked forward pass for training.