VisionLanguageModelBase<T>

Base Classes · AiDotNet.VisionLanguage

Base class for vision-language neural networks that can operate in both ONNX inference and native training modes.

Inheritance: NeuralNetworkBase<T> → VisionLanguageModelBase<T>

Implements: IShapeContract

Known derived types: ALIGN<T>, AquilaVL<T>, Aria<T>, BASIC<T>, BLIP3<T>, BiomedCLIP<T>, BridgeTower<T>, CLIPA<T>, Cambrian1<T>, Chameleon<T>, ClaudeVision<T>, CoCa<T>, CogVLM2<T>, CogVLM<T>, DFNCLIP<T>, DINOX<T>, DINOv2<T>, DINOv3<T>, DeCLIP<T>, DeepSeekVL2<T>, DeepSeekVL<T>, DocPedia<T>, DragonflyMed<T>, Dragonfly<T>, EVACLIP<T>, Eagle25<T>, Eagle<T>, Emu2<T>, Emu3<T>, Emu<T>, FLIP<T>, FerretV2<T>, Ferret<T>, Florence2<T>, Fuyu<T>, GIT<T>, GLaMM<T>, GOTOCR2<T>, GPT4Point<T>, GR00TN1<T>, GeminiVision<T>, Gemma3<T>, GeoChat<T>, GrokVision<T>, Groma<T>, GroundedSAM2<T>, GroundingDINO15<T>, GroundingDINO<T>, Helix<T>, IDEFICS2<T>, IDEFICS3<T>, IDEFICS<T>, InstructBLIP<T>, InternVL25<T>, InternVL2<T>, InternVL3<T>, InternVL<T>, InternViT<T>, JanusPro<T>, Janus<T> (+103 more)

For Beginners

Vision-language models process both images and text together. This base class provides:

  • Support for pre-trained ONNX models (fast inference with existing models)
  • Full training capability from scratch (like other neural networks)
  • Image preprocessing utilities (normalization, resizing)
  • Dual-encoder architecture support (separate image and text encoders)

You can use this class in two ways:

  1. Load a pre-trained ONNX model for quick inference
  2. Build and train a new model from scratch

How It Works

This class extends NeuralNetworkBase to provide vision-language-specific functionality while maintaining full integration with the AiDotNet neural network infrastructure.

Properties

Property Summary
ILossFunction<T> DefaultLossFunction { get; } Gets the default loss function for this model.
bool IsOnnxMode { get; } Gets whether this model is running in ONNX inference mode.
bool SupportsTraining { get; } Gets whether this network supports training.
int EmbeddingDim { get; set; } Gets the embedding dimensionality for this model.
int ImageChannels { get; set; } Gets the number of image channels expected (typically 3 for RGB).
int ImageSize { get; set; } Gets the expected input image size (height = width in pixels).

Methods

Method Summary
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank) The family’s default output law: one pooled EmbeddingDim-wide embedding per sample.