VisionTransformer<T>
Models & Types · AiDotNet.NeuralNetworks
Implements the Vision Transformer (ViT) architecture for image classification tasks.
Inheritance: NeuralNetworkBase<T> → DeclaredModelLayoutBase<T> → ImageClassifierModelLayoutBase<T> → VisionTransformer<T>
For Beginners
The Vision Transformer (ViT) is a modern approach to understanding images using transformers.
Unlike traditional neural networks that process images pixel by pixel or with sliding windows (convolutions), ViT treats an image like a sentence of words:
- First, it cuts the image into small square patches (like breaking a sentence into words)
- Each patch gets converted to a numerical representation (like word embeddings)
- Position information is added so the model knows where each patch came from
- A special classification token is added to gather information about the whole image
- Transformer layers process all patches together, learning relationships between them
- Finally, the classification token’s output is used to predict the image class
This approach has been very successful and often outperforms traditional convolutional neural networks, especially when trained on large datasets.
How It Works
The Vision Transformer applies transformer architecture, originally designed for natural language processing, to computer vision tasks. It divides images into fixed-size patches, linearly embeds them, adds positional embeddings, and processes the sequence through transformer encoder layers.
Example
var image = Tensor<float>.CreateRandom(new[] { 1, 3, 224, 224 });
var trainX = Tensor<float>.CreateRandom(4, 8);
var trainY = Tensor<float>.CreateRandom(4, 2);
var result = new AiModelBuilder<float, Tensor<float>, Tensor<float>>()
.ConfigureModel(new VisionTransformer<float>())
.Build(trainX, trainY);
var output = result.Predict(image);
Constructors
| Constructor | Summary |
|---|---|
VisionTransformer() |
Creates a new Vision Transformer with the specified configuration. |
VisionTransformer(NeuralNetworkArchitecture<T> architecture, int imageHeight, int imageWidth, int channels, int patchSize, int numClasses, ILossFunction<T> lossFunction, VisionTransformerOptions options) |
Properties
| Property | Summary |
|---|---|
bool SupportsTraining { get; } |
Indicates whether this network supports training. |
Methods
| Method | Summary |
|---|---|
ModelMetadata<T> GetModelMetadata() |
Gets the model metadata. |
Tensor<T> ForwardForTraining(Tensor<T> input) |
Tape-tracked forward pass for training. |