PerceptionEncoder<T>

Models & Types · AiDotNet.VisionLanguage.Encoders

Meta’s Perception Encoder for multimodal alignment tasks with global and dense features.

Inheritance: NeuralNetworkBase<T> → VisionLanguageModelBase<T> → PerceptionEncoder<T>

Implements: IVisualEncoder<T>

For Beginners

Meta’s Perception Encoder is a vision backbone that produces both global features (a single vector summarizing the whole image) and dense spatial features (one vector per image region). This makes it versatile — the global features work for classification and retrieval, while the spatial features enable detection and segmentation. Default values follow the original paper settings.

How It Works

Perception Encoder (Meta, 2025) is a vision encoder combining contrastive learning with dense prediction objectives. It produces both global features (via CLS token) and spatial features (via patch tokens), making it suitable for classification, detection, and segmentation tasks in multimodal systems.

References:

  • Paper: “Perception Encoder: The best satisfies all” (Meta, 2025)

Example

// Create a Perception Encoder for global and dense visual features
// combining contrastive and dense prediction objectives
var architecture = new NeuralNetworkArchitecture<double>(
    inputType: InputType.TwoDimensional,
    taskType: NeuralNetworkTaskType.ImageClassification,
    inputHeight: 224, inputWidth: 224, inputDepth: 3, outputSize: 512);
            
// ONNX inference mode with pre-trained model
var model = new PerceptionEncoder<double>(architecture, "perception_encoder.onnx");
            
// Training mode with native layers
var trainModel = new PerceptionEncoder<double>(architecture, new PerceptionEncoderOptions());

Constructors

Constructor Summary
PerceptionEncoder(NeuralNetworkArchitecture<T> architecture, PerceptionEncoderOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer)
PerceptionEncoder(NeuralNetworkArchitecture<T> architecture, string modelPath, PerceptionEncoderOptions options)

Properties

Property Summary
int EmbeddingDimension { get; }

Methods

Method Summary
ModelMetadata<T> GetModelMetadata()
Tensor<T> EncodeImage(Tensor<T> image)