Vision Language

395 public types in this namespace, organized by kind.

Models & Types (178)

Type Summary
ALIGN<T> ALIGN (A Large-scale ImaGe and Noisy-text embedding) model for zero-shot classification and cross-modal retrieval using EfficientNet as the vision encoder.
AquilaVL<T> AquilaVL: bilingual VLM aligned with Chinese and English visual-language understanding.
Aria<T> Aria: mixture-of-experts multimodal model with efficient expert routing.
BASIC<T> BASIC (Combined Scaling for Zero-shot Transfer Learning) model using CoAtNet vision encoder.
BLIP3<T> BLIP-3 (xGen-MM): scaled vision-language model with interleaved data and any-to-any generation.
BiomedCLIP<T> BiomedCLIP model fine-tuned on 15M biomedical image-text pairs from PubMed Central.
BridgeTower<T> BridgeTower: cross-modal alignment through bridge layers connecting vision and text encoder layers.
CLIPA<T> CLIPA (CLIP with Inverse scaling law and Accelerated training) model using progressive resolution.
Cambrian1<T> Cambrian-1: Spatial Vision Aggregator with 35+ vision encoder combinations.
Chameleon<T> Chameleon: early fusion with discrete tokens for all modalities.
ClaudeVision<T> Claude Vision: reference implementation of Anthropic’s multimodal reasoning model.
CoCa<T> CoCa (Contrastive Captioners): dual-loss model combining contrastive and captioning objectives.
CogVLM2<T> CogVLM2: improved visual expert architecture with GLM-4 backbone and video understanding.
CogVLM<T> CogVLM: deep fusion via visual expert module in every LLM layer.
DFNCLIP<T> DFN-CLIP (Data Filtering Networks for CLIP) model using filtered high-quality training data.
DINOX<T> DINO-X: strongest open-world perception model.
DINOv2<T> DINOv2 self-supervised vision encoder producing universal visual features without labels.
DINOv3<T> DINOv3 self-supervised vision encoder scaled to 7B parameters on 1.7B images.
DeCLIP<T> DeCLIP (Data-efficient CLIP) model using self-supervised and nearest-neighbor supervision for data efficiency.
DeepSeekVL2<T> DeepSeek-VL2: MoE vision-language model with dynamic tiling and multi-head latent attention.
DeepSeekVL<T> DeepSeek-VL: real-world vision-language understanding with hybrid SigLIP + SAM-B encoder.
DocPedia<T> DocPedia: frequency-domain document understanding model.
DragonflyMed<T> Dragonfly-Med: medical image understanding model.
Dragonfly<T> Dragonfly: multi-resolution visual encoding VLM for fine-grained understanding.
EVACLIP<T> EVA-CLIP model combining EVA-02 masked image modeling backbone with CLIP contrastive training.
Eagle25<T> Eagle 2.5: extended context VLM with video understanding capabilities.
Eagle<T> Eagle: NVIDIA data-centric VLM with high-quality training data.
Emu2<T> Emu2: scaled 37B unified understanding and generation model.
Emu3<T> Emu3: next-token prediction unifies understanding and generation in a single model.
EmuEdit<T> Emu Edit - precise image editing via recognition and generation tasks.
Emu<T> Emu: unified VQA, captioning, and image generation via EVA-CLIP + LLM + regression head.
FLIP<T> FLIP (Fast Language-Image Pre-training) model that accelerates CLIP training via random masking of image patches.
FerretV2<T> Ferret-v2: improved referring and grounding with enhanced spatial understanding.
Ferret<T> Ferret: spatial-aware visual sampler for free-form region referring and grounding.
Florence2Region One region a Florence-2 task answer located: its label and its pixel box.
Florence2TaskResult A parsed Florence-2 task answer: its text, plus any regions written as location tokens.
Florence2<T> Florence-2 unified vision foundation model for captioning, detection, grounding, and OCR.
Fuyu<T> Fuyu: no vision encoder; raw patches directly into transformer.
GIT<T> GIT (Generative Image-to-Text): simple ViT encoder + autoregressive text decoder.
GLaMM<T> GLaMM: Pixel Grounding Large Multimodal Model.
GOTOCR2<T> GOT-OCR2: 580M unified OCR model for text, tables, charts, equations, and music scores.
GPT4Point<T> GPT4Point: unified point-language understanding and generation.
GR00TFlowMatchingActionHead<T> Flow-matching action head used by GR00T N1 to denoise continuous joint commands conditioned on the System-2 vision-language latent.
GR00TN1<T> GR00T N1: NVIDIA’s open foundation model for generalist humanoid robots, combining a SigLIP + Eagle-2 vision-language System-2 reasoner with a flow-matching DiT action head as System 1 (NVIDIA 2025, arXiv:2503.14734).
GeminiVision<T> Gemini Vision: reference implementation of Google’s native multimodal Mixture of Experts model.
Gemma3<T> Gemma 3: Google’s 3B-72B VLM with Native Dynamic Resolution ViT and 128k context.
GeoChat<T> GeoChat: grounded VLM for satellite imagery understanding.
GlammAnswer<T> GLaMM’s answer: the text and a mask for every [SEG] in it.
GlammSample<T> A GLaMM training sample: an image, a conversation about it, the regions its <bbox> tokens name, and a mask for every [SEG] in the answers.
GlammTurn One turn of a GLaMM conversation.
GrokVision<T> Grok Vision: reference implementation of xAI’s real-time multimodal model.
Groma<T> Groma: localized visual tokenization for grounded understanding.
GroundedSAM2<T> Grounded SAM 2: Grounding DINO finds the boxes a caption names, and SAM 2 segments each box.
GroundedSegment<T> One region a caption names, with its SAM 2 mask.
GroundingBox A box that a caption’s phrase names, for training a grounding model.
GroundingDINO15<T> Grounding DINO 1.5: enhanced open-set detection with improved architecture.
GroundingDINO<T> Grounding DINO: open-set detection combining DINO with grounded pre-training.
GroundingDetection<T> A region a grounding model found for a caption.
HelixDualSystemRunner<T> Streaming controller that coordinates Helix’s fast System-1 visuomotor policy (200 Hz) and slow System-2 VLM (7–9 Hz) per Figure AI’s dual-rate dual-system architecture.
HelixSystem2Latent<T> Conditioning signal that Helix’s slow System-2 VLM produces for the fast System-1 controller.
Helix<T> Helix: dual-system vision-language-action model for full-body humanoid control (Figure AI, 2025, arXiv:2502.07092).
IDEFICS2<T> IDEFICS2: 8B efficient VLM with SigLIP encoder and Mistral-7B decoder.
IDEFICS3<T> IDEFICS3: state-of-the-art 8B VLM trained with Docmatix for document understanding.
IDEFICS<T> IDEFICS: 80B open reproduction of Flamingo for interleaved image-text generation.
InstructBLIP<T> InstructBLIP: instruction-tuned BLIP-2 for zero-shot generalization across vision-language tasks.
InternVL25<T> InternVL2.5: improved training data and strategy over InternVL2 with InternLM2.5 backbone.
InternVL2<T> InternVL2: dynamic resolution with pixel shuffle and InternLM2 backbone.
InternVL3<T> InternVL3: 78B SOTA open-source model with advanced training and test-time recipes.
InternVL<T> InternVL: 6B InternViT with progressive alignment and LLaMA language backbone.
InternViT<T> InternViT, a 6B-parameter ViT used as the vision encoder in the InternVL series.
JanusPro<T> Janus-Pro: unified multimodal understanding and generation with decoupled vision encoders (Chen et al., DeepSeek 2025, arXiv:2501.17811).
Janus<T> Janus: decoupled visual encoding for understanding vs generation.
KOSMOS1<T> KOSMOS-1: multimodal large language model with visual tokens embedded in causal LM.
KOSMOS2<T> KOSMOS-2: grounded multimodal large language model with text spans linked to bounding boxes.
KimiVLThinking<T> Kimi-VL-Thinking: long chain-of-thought reasoning with RL alignment.
KimiVL<T> Kimi-VL: MoE VLM with MoonViT and long-context processing.
LEOVL<T> LEO-VL: efficient 3D scene representation from multi-view RGB-D.
LLM2CLIP<T> LLM2CLIP model enhancing CLIP’s text encoder with LLM embeddings for richer semantic understanding.
LLaVA15<T> LLaVA-1.5: improved baselines with visual instruction tuning using MLP cross-modal connector.
LLaVACoT<T> LLaVA-CoT: chain-of-thought visual reasoning with structured output.
LLaVAMed<T> LLaVA-Med: biomedical VLM with GPT-4-level visual understanding.
LLaVANeXTVideo<T> LLaVA-NeXT-Video: average pooling for frame token reduction.
LLaVANeXT<T> LLaVA-NeXT: improved reasoning, OCR, and world knowledge via dynamic high-resolution AnyRes.
LLaVAOneVision15<T> LLaVA-OneVision-1.5: fully open model outperforming Qwen2.5-VL on 18/27 benchmarks.
LLaVAOneVision<T> LLaVA-OneVision: single model for images, multi-image, and videos with easy visual task transfer.
LLaVAVideo<T> LLaVA-Video: synthetic dataset-trained video instruction model.
LXMERT<T> LXMERT (Learning Cross-Modality Encoder Representations from Transformers) with three-encoder architecture.
LiT<T> LiT (Locked-image Tuning) model that freezes a pre-trained image encoder and only trains the text encoder for efficient contrastive image-text alignment.
Llama32Vision<T> Llama 3.2 Vision: Meta’s 11B/90B vision models for edge/mobile deployment.
LongVILA<T> LongVILA: long-context visual language model for 1hr+ videos.
METER<T> METER (Multimodal End-to-end TransformER) with systematic VLP component study.
MGIE<T> MGIE - instruction-based image editing guided by a multimodal large language model.
MPLUGDocOwl15<T> mPLUG-DocOwl 1.5: unified structure learning achieving SOTA on 10 document benchmarks.
MPLUGDocOwl2<T> mPLUG-DocOwl 2: high-res compressing for multi-page document understanding.
MPLUGDocOwl<T> mPLUG-DocOwl: modular MLLM for document understanding with visual abstractor.
MPLUGOwl2<T> mPLUG-Owl2: improved modular design for multi-image understanding.
MPLUGOwl3<T> mPLUG-Owl3: enhanced with hyper-attention for long visual sequences.
MPLUGOwl<T> mPLUG-Owl: modular VLM with visual abstractor for vision-language alignment.
Mantis<T> Mantis: interleaved multi-image VLM for complex visual reasoning.
Maya<T> Maya: multilingual multimodal VLM supporting 8 languages.
MedCLIP<T> MedCLIP model using decoupled semantic matching for medical image-text alignment.
MedFlamingo<T> Med-Flamingo: few-shot medical visual question answering via Flamingo architecture.
MetaCLIP<T> MetaCLIP (Demystifying CLIP Data) reproduces and improves CLIP via a fully transparent data curation pipeline.
MiniCPMV<T> MiniCPM-V: efficient on-device VLM with strong OCR and multilingual capabilities.
MiniCPMo<T> MiniCPM-o: omnimodal VLM with speech and real-time streaming support.
MiniGPT4<T> MiniGPT-4: ViT + Q-Former aligned with Vicuna via single projection layer.
MiniGPTv2<T> MiniGPT-v2: unified interface for multi-task VL learning with task-specific tokens.
Molmo<T> Molmo: open VLM with pointing and grounding capabilities.
Monkey<T> Monkey: high-resolution VLM with multi-level description generation.
Moondream<T> Moondream: tiny yet capable VLM for resource-constrained environments.
NVLM<T> NVLM 1.0: NVIDIA’s cross-attention + decoder-only hybrid VLM that retains text performance.
OWLViT<T> OWL-ViT: open-vocabulary object detection via ViT + CLIP alignment.
OWLv2<T> OWLv2: self-training for scaling open-vocabulary detection.
Octo<T> Octo: open-source generalist robot policy trained on 800K demonstrations.
OmniGen2<T> OmniGen2: dual-path architecture with parameter decoupling.
OpenCLIP<T> OpenCLIP (Open Contrastive Language-Image Pre-training) model for zero-shot classification and cross-modal retrieval using open-source training data.
OpenFlamingo<T> OpenFlamingo: open-source reproduction of Flamingo with perceiver resampler.
Oscar<T> Oscar (Object-Semantics Aligned pre-training) using object tags as anchor points.
Ovis<T> Ovis: structural embedding alignment VLM with visual token scaling.
PLLaVA<T> PLLaVA: parameter-free pooling extension from images to video.
PaLI3<T> PaLI-3: efficient PaLI with SigLIP ViT encoder, smaller and better.
PaLIX<T> PaLI-X: scaled PaLI to 55B with improved component fine-tuning.
PaLI<T> PaLI (Pathways Language and Image model): large-scale ViT-e vision encoder + mT5 text decoder.
PaLME<T> PaLM-E: 562B embodied multimodal language model for robotic planning.
PathVLM<T> PathVLM: histopathology-specific vision-language model.
PerceptionEncoder<T> Meta’s Perception Encoder for multimodal alignment tasks with global and dense features.
Phi3Vision<T> Phi-3-Vision: compact 4.2B parameter model with strong vision capabilities via curated data.
Phi4Multimodal<T> Phi-4-Multimodal: unified vision + audio + text in a single framework.
PiZero<T> pi-zero: PaliGemma VLM with action expert for 8 robot embodiments.
PixtralLarge<T> Pixtral Large: Mistral’s 124B decoder + 1B vision encoder.
Pixtral<T> Pixtral: Mistral’s 12B decoder + 400M vision encoder.
PointLLM<T> PointLLM: LLM understanding of colored 3D point clouds.
QVQ72B<T> QVQ-72B: first open-source multimodal reasoning model from Qwen.
Qwen25VL<T> Qwen2.5-VL: visual agent with 1hr+ video, bounding box/point localization capabilities.
Qwen2VL<T> Qwen2-VL: Naive Dynamic Resolution with M-RoPE for multimodal positional encoding.
Qwen3VL<T> Qwen3-VL: latest series with 2B/4B/8B/32B variants and cross-attention resampler.
QwenVL<T> Qwen-VL: visual window attention, multi-resolution, bounding box output via cross-attention resampler.
RADIOv25<T> RADIOv2.5 agglomerative vision foundation model distilling DINOv2, SAM, SigLIP, and CLIP.
RSGPT<T> RSGPT: remote sensing GPT based on InstructBLIP architecture.
RT2ActionTokenizer<T> RT-2’s action-as-text tokenizer.
RT2<T> RT-2: vision-language-action model that transfers web knowledge to robotic control (Brohan et al., Google DeepMind, 2023, arXiv:2307.15818).
RadFM<T> RadFM: 3D ViT with perceiver for radiology report generation and VQA.
RegionCLIP<T> RegionCLIP model extending CLIP to learn region-level (object-level) visual representations.
RemoteCLIP<T> RemoteCLIP model adapted for remote sensing and satellite imagery understanding.
SAM<T> Segment Anything Model (SAM) vision encoder for promptable image segmentation.
SEEDX<T> SEED-X: multi-granularity comprehension and generation model.
SceneLLM<T> Scene-LLM: voxel-based 3D scene understanding with language models.
Shikra<T> Shikra: referential dialogue model with coordinate output for grounding.
ShowO2<T> Show-o2: improved native unified multimodal model.
ShowO<T> Show-o: single transformer for unified understanding and generation.
SigLIP2<T> SigLIP 2 (Multilingual Vision-Language Encoders with Improved Semantic Understanding) for contrastive encoding, zero-shot classification, and captioning.
SigLIPSO<T> SigLIP-SO (Shape-Optimized SigLIP) 400M vision encoder widely used as a VLM backbone.
SigLIP<T> SigLIP (Sigmoid Loss for Language-Image Pre-training) model for zero-shot classification and cross-modal retrieval with improved batch scaling.
SkyEyeGPT<T> SkyEyeGPT: unified remote sensing vision-language tasks with 968K instruction samples.
SkyworkR1V2<T> Skywork R1V2: hybrid RL (MPO + GRPO) for multimodal reasoning SOTA.
SkyworkR1V<T> Skywork R1V: cross-modal transfer of reasoning LLMs to vision.
SlowFastLLaVA<T> SlowFast-LLaVA: token-efficient slow/fast pathways for long video.
SmartEdit<T> SmartEdit - complex instruction-based image editing with a multimodal LLM.
SmolVLM<T> SmolVLM: compact and efficient VLM designed for edge deployment.
Surya<T> Surya: multi-language OCR with layout analysis support.
TextMonkey<T> TextMonkey: OCR-free text understanding with shifted window attention.
ThreeDGraphLLM<T> 3DGraphLLM: 3D scene graph as LLM input for spatial reasoning.
ThreeDLLM<T> 3D-LLM: injects 3D spatial features into large language models.
ThreeDVLA<T> 3D-VLA: connects vision-language-action to 3D world via generative world model.
Transfusion<T> Transfusion: combined autoregressive and diffusion loss in single transformer.
UNITER<T> UNITER (Universal Image-TExt Representation) with conditional masking pre-training.
UReader<T> UReader: universal OCR-free visually-situated language model.
VILAU<T> VILA-U: unified VLM with autoregressive visual generation and understanding.
VILA<T> VILA: VLM pre-trained with interleaved image-text data for enhanced in-context learning.
ViLBERT<T> ViLBERT (Vision-and-Language BERT) with co-attention between parallel vision and language streams.
ViLT<T> ViLT (Vision-and-Language Transformer) with minimal architecture using patch embeddings.
ViT<T> Vision Transformer (ViT) that splits images into patches and processes them with a standard Transformer.
VideoChat2<T> VideoChat2: progressive video training with diverse data.
VideoLLaMA2<T> VideoLLaMA 2: spatial-temporal convolution for video tokens.
VideoLLaMA3<T> VideoLLaMA 3: frontier multimodal for image and video.
VideoLLaVA<T> Video-LLaVA: united visual representation for video understanding.
VinVL<T> VinVL (Visual Features in Vision-Language) with enhanced object detection features.
VisualBERT<T> VisualBERT: single-stream transformer that concatenates visual and text tokens.

Base Classes (1)

Type Summary
VisionLanguageModelBase<T> Base class for vision-language neural networks that can operate in both ONNX inference and native training modes.

Interfaces (19)

Type Summary
IContrastiveVisionLanguageModel<T> Interface for contrastive vision-language models that align image and text embeddings in a shared space for zero-shot classification and cross-modal retrieval.
IDocumentUnderstandingModel<T> Interface for document understanding models that process text-heavy images, documents, charts, and tables.
IGenerativeVisionLanguageModel<T> Interface for generative vision-language models that produce text output from visual input.
IGroundedConversationTrainable<T> A model that trains on grounded conversations: an image, a conversation about it, the regions its questions name, and a mask for every [SEG] in its answers.
IGroundingTrainable<T> A grounding model that trains on grounded phrases: an image, a caption, and the boxes the caption’s phrases name.
IImageEditingVLM<T> Interface for VLMs that edit images based on natural language instructions.
IInstructionTunedVLM<T> Interface for instruction-tuned vision-language models that generate conversational responses from visual input conditioned on user instructions.
IMedicalVLM<T> Interface for medical domain vision-language models specializing in biomedical image understanding.
IProprietaryVLM<T> Interface for reference implementations of proprietary VLM architectures.
IReasoningVLM<T> Interface for vision-language models with chain-of-thought reasoning capabilities.
IRemoteSensingVLM<T> Interface for remote sensing vision-language models specializing in satellite and aerial imagery.
ITextEncoder<T> Interface for text encoders that extract feature representations from text.
IThreeDVisionLanguageModel<T> Interface for 3D vision-language models that understand point clouds, 3D scenes, and spatial relationships.
IUnifiedVisionModel<T> Interface for unified vision models that can both understand and generate images.
IVideoLanguageModel<T> Interface for video-language models that process video frames for temporal understanding and QA.
IVisionLanguageAction<T> Interface for Vision-Language-Action (VLA) models that connect visual understanding and language reasoning to physical robotic actions.
IVisionLanguageFusionModel<T> Interface for vision-language fusion models that combine image and text features for tasks like VQA, image-text matching, and cross-modal retrieval.
IVisualEncoder<T> Interface for vision encoders that extract feature representations from images.
IVisualGroundingModel<T> Interface for visual grounding models that localize objects or regions from natural language descriptions.

Enums (14)

Type Summary
ContrastiveLossType Specifies the contrastive loss function type.
DomainSpecialization Specifies the domain specialization for a CLIP-family model.
Florence2ModelSize Specifies the Florence-2 model size variant.
Florence2Task The Florence-2 tasks.
FusionType Specifies how vision and language features are fused in a VLM.
GenerativeArchitectureType Specifies the architecture type for generative vision-language models.
InstructionTunedArchitectureType Specifies the architecture type for instruction-tuned vision-language models.
ModelPrecision Specifies the precision mode for model inference and training.
PoolingStrategy Specifies the global feature pooling strategy for vision encoders.
PositionalEmbeddingType Specifies the positional embedding type for vision transformers.
PretrainingDataset Specifies the pre-training dataset used for the model.
TextEncoderVariant Specifies the text encoder architecture.
ViTVariant Specifies the Vision Transformer (ViT) variant used as the image encoder.
VisualFeatureType Specifies the visual feature extraction method for foundational VLMs.

Options & Configuration (183)

Type Summary
ALIGNOptions Configuration options for the ALIGN (A Large-scale ImaGe and Noisy-text embedding) model.
AquilaVLOptions Options for Aquila-VL (BAAI LLaVA-style VLM with Qwen2.5 + SigLIP).
AriaOptions Options for Aria (Rhymes AI MoE VLM with 3.9B active params and 64K context multimodal).
BASICOptions Configuration options for the BASIC (Batch-wise Alignment of Scaled Image-text Contrastive) model.
BLIP3Options Options for BLIP-3 (xGen-MM) with interleaved data and any-to-any generation.
BiomedCLIPOptions Configuration options for the BiomedCLIP model.
BridgeTowerOptions Configuration options for BridgeTower cross-modal alignment model.
CLIPAOptions Configuration options for the CLIPA (CLIP with Inverse scaling law and Accelerated training) model.
Cambrian1Options Options for Cambrian-1 (Spatial Vision Aggregator with 35+ vision encoder combinations).
ChameleonOptions Configuration options for Chameleon: early fusion with discrete tokens for all modalities.
ClaudeVisionOptions Configuration options for Claude Vision.
CoCaOptions Options for CoCa (Contrastive Captioners) with dual loss.
CogVLM2Options Options for CogVLM2 (improved architecture with video understanding, GLM-4/LLaMA-3).
CogVLMOptions Options for CogVLM (deep fusion via visual expert module in every LLM layer).
ContrastiveEncoderOptions Base configuration options for contrastive vision-language encoders (CLIP-family models).
DFNCLIPOptions Configuration options for the DFN-CLIP (Data Filtering Networks for CLIP) model.
DINOXOptions Configuration options for DINO-X: strongest open-world perception model.
DINOv2Options Configuration options for the DINOv2 self-supervised vision encoder.
DINOv3Options Configuration options for DINOv3, the next evolution of self-supervised vision encoders.
DeCLIPOptions Configuration options for the DeCLIP (Data-efficient CLIP) model.
DeepSeekVL2Options Options for DeepSeek-VL2 (MoE, dynamic tiling, multi-head latent attention).
DeepSeekVLOptions Options for DeepSeek-VL (hybrid vision encoder with SigLIP + SAM-B).
DocPediaOptions Configuration options for DocPedia: frequency-domain document understanding model.
DocumentVLMOptions Base configuration options for document understanding vision-language models.
DragonflyMedOptions Configuration options for Dragonfly-Med.
DragonflyOptions Options for Dragonfly (efficient multi-resolution visual encoding from Together.ai).
EVACLIPOptions Configuration options for the EVA-CLIP model.
Eagle25Options Options for Eagle 2.5 (NVIDIA long-context multimodal for video + high-res images).
EagleOptions Options for Eagle (NVIDIA data-centric VLM with learned data mixing).
EditingVLMOptions Base configuration options for image editing VLMs.
Emu2Options Options for Emu2 (scaled to 37B; enhanced understanding + generation).
Emu3Options Options for Emu3 (next-token prediction unifies understanding + generation).
EmuEditOptions Configuration options for Emu Edit: precise image editing via recognition and generation tasks.
EmuOptions Options for Emu (unified VQA, captioning, generation via EVA-CLIP + LLM + diffusion).
FLIPOptions Configuration options for the FLIP (Fast Language-Image Pre-training) model.
FerretOptions Configuration options for Ferret: spatial-aware visual sampler for free-form region referring and grounding.
FerretV2Options Configuration options for Ferret-v2: improved referring and grounding with enhanced spatial understanding.
Florence2Options Configuration options for Florence-2, Microsoft’s unified vision foundation model.
FoundationalVLMOptions Base configuration options for foundational vision-language fusion models.
FuyuOptions Options for Fuyu (no vision encoder; raw patches directly into transformer).
GITOptions Options for GIT (Generative Image-to-Text) with simple ViT + text decoder.
GLaMMOptions Configuration options for GLaMM (Rasheed et al.
GOTOCR2Options Configuration options for GOT-OCR2, the General OCR Theory model (Wei et al., 2024).
GPT4PointOptions Configuration options for GPT4Point.
GR00TN1Options Configuration options for GR00T N1.
GeminiVisionOptions Configuration options for Gemini Vision.
Gemma3Options Options for Gemma 3 (3B-72B, Native Dynamic Resolution ViT, 128k context, 29 languages).
GenerativeVLMOptions Base configuration options for generative vision-language models.
GeoChatOptions Configuration options for GeoChat.
GrokVisionOptions Configuration options for Grok Vision.
GromaOptions Configuration options for Groma: localized visual tokenization for grounded understanding.
GroundedSAM2Options Configuration options for Grounded SAM 2: Grounding DINO finds the boxes a caption names, and SAM 2 segments each box.
GroundingDINO15Options Configuration options for Grounding DINO 1.5: enhanced open-set detection with improved architecture.
GroundingDINOOptions Configuration options for Grounding DINO (Liu et al.
GroundingVLMOptions Base configuration options for visual grounding models.
HelixOptions Configuration options for Helix.
IDEFICS2Options Options for IDEFICS2 (8B; SigLIP + Mistral-7B with Perceiver pooling).
IDEFICS3Options Options for IDEFICS3 (state-of-the-art 8B trained on open datasets + Docmatix).
IDEFICSOptions Options for IDEFICS (80B open reproduction of Flamingo).
InstructBLIPOptions Options for InstructBLIP (instruction-tuned Q-Former for zero-shot generalization).
InstructionTunedVLMOptions Base configuration options for instruction-tuned vision-language models.
InternVL25Options Options for InternVL2.5 (improved training data and strategy over InternVL2).
InternVL2Options Options for InternVL2 (dynamic resolution, pixel shuffle, InternLM2 backbone).
InternVL3Options Options for InternVL3 (78B, MMMU 72.2 SOTA among open-source).
InternVLOptions Options for InternVL (6B InternViT + LLaMA with progressive alignment).
InternViTOptions Configuration options for InternViT, the vision encoder used in the InternVL series.
JanusOptions Configuration options for Janus: decoupled visual encoding for understanding vs generation.
JanusProOptions Configuration options for Janus-Pro: scaled data and model with optimized training strategy.
KOSMOS1Options Options for KOSMOS-1 (Huang et al.
KOSMOS2Options Options for KOSMOS-2 (grounded multimodal with text spans linked to bounding boxes; Peng et al.
KimiVLOptions Configuration options for Kimi-VL: MoE VLM with MoonViT and long-context processing.
KimiVLThinkingOptions Configuration options for Kimi-VL-Thinking: long chain-of-thought reasoning with RL alignment.
KosmosOptions Hyperparameters shared by KOSMOS-1 and KOSMOS-2: - A CLIP ViT-L/14 image encoder: 224 px, 1024 wide, 24 layers, 16 heads, patch 14.
LEOVLOptions Configuration options for LEO-VL.
LLM2CLIPOptions Configuration options for the LLM2CLIP model.
LLaVA15Options Options for LLaVA-1.5 (MLP cross-modal connector, higher-res CLIP, academic task SOTA).
LLaVACoTOptions Configuration options for LLaVA-CoT: chain-of-thought visual reasoning with structured output.
LLaVAMedOptions Configuration options for LLaVA-Med.
LLaVANeXTOptions Options for LLaVA-NeXT (dynamic high-resolution via AnyRes, stronger reasoning).
LLaVANeXTVideoOptions Configuration options for LLaVA-NeXT-Video: average pooling for frame token reduction.
LLaVAOneVision15Options Options for LLaVA-OneVision-1.5 (fully open, outperforms Qwen2.5-VL on 18/27 benchmarks).
LLaVAOneVisionOptions Options for LLaVA-OneVision (single model for images, multi-image, and videos).
LLaVAVideoOptions Configuration options for LLaVA-Video: synthetic dataset-trained video instruction model.
LXMERTOptions Configuration options for LXMERT cross-modal encoder.
LiTOptions Configuration options for the LiT (Locked-image Tuning) model.
Llama32VisionOptions Options for Llama 3.2 Vision (11B/90B vision models for edge/mobile deployment).
LongVILAOptions Configuration options for LongVILA: long-context visual language model for 1hr+ videos.
METEROptions Configuration options for METER (Multimodal End-to-end TransformER).
MGIEOptions Configuration options for MGIE: MLLM-guided image editing with LLaVA-based instruction understanding.
MPLUGDocOwl15Options Configuration options for mPLUG-DocOwl 1.5: unified structure learning achieving SOTA on 10 document benchmarks.
MPLUGDocOwl2Options Configuration options for mPLUG-DocOwl 2: high-res compressing for multi-page document understanding.
MPLUGDocOwlOptions Configuration options for mPLUG-DocOwl: modular MLLM for document understanding with visual abstractor.
MPLUGOwl2Options Options for mPLUG-Owl2 (improved modular design for multi-image understanding).
MPLUGOwl3Options Options for mPLUG-Owl3 (enhanced with hyper-attention for long visual sequences).
MPLUGOwlOptions Options for mPLUG-Owl (modular VLM with visual abstractor).
MantisOptions Options for Mantis (multi-image reasoning with interleaved data).
MayaOptions Options for Maya (multilingual VLM for low-resource languages).
MedCLIPOptions Configuration options for the MedCLIP model.
MedFlamingoOptions Configuration options for Med-Flamingo.
MedicalVLMOptions Base configuration options for medical domain vision-language models.
MetaCLIPOptions Configuration options for the MetaCLIP model.
MiniCPMVOptions Options for MiniCPM-V (8B model achieving GPT-4V-level on select benchmarks).
MiniCPMoOptions Options for MiniCPM-o (omni-modal with real-time speech + vision).
MiniGPT4Options Options for MiniGPT-4 (ViT + Q-Former aligned with Vicuna via single projection layer).
MiniGPTv2Options Options for MiniGPT-v2 (multi-task with task-specific tokens, LLaMA-2 backbone).
MolmoOptions Options for Molmo (AI2 open VLM with diverse data mixture and pointing capability).
MonkeyOptions Options for Monkey (multi-level description generation with high resolution).
MoondreamOptions Options for Moondream (1.8B; SigLIP + Phi-1.5; ideal for edge/mobile).
NVLMOptions Options for NVLM 1.0 (cross-attention + decoder-only hybrid that retains text performance).
OWLViTOptions Configuration options for OWL-ViT: open-vocabulary object detection via ViT + CLIP alignment.
OWLv2Options Configuration options for OWLv2: self-training for scaling open-vocabulary detection.
OctoOptions Configuration options for Octo.
OmniGen2Options Configuration options for OmniGen2: dual-path architecture with parameter decoupling.
OpenCLIPOptions Configuration options for the OpenCLIP model.
OpenFlamingoOptions Options for OpenFlamingo (open-source Flamingo with perceiver resampler).
OscarOptions Configuration options for Oscar (Object-Semantics Aligned pre-training).
OvisOptions Options for Ovis (structural embedding alignment for visual tokens).
PLLaVAOptions Configuration options for PLLaVA: parameter-free pooling extension from images to video.
PaLI3Options Options for PaLI-3 (efficient PaLI with SigLIP ViT, smaller and better).
PaLIOptions Options for PaLI (Pathways Language and Image model) with ViT-e + mT5.
PaLIXOptions Options for PaLI-X (scaled PaLI to 55B with improved fine-tuning).
PaLMEOptions Configuration options for PaLM-E.
PathVLMOptions Configuration options for PathVLM.
PerceptionEncoderOptions Configuration options for Meta’s Perception Encoder vision foundation model.
Phi3VisionOptions Options for Phi-3-Vision (compact 4.2B with strong vision via curated data).
Phi4MultimodalOptions Options for Phi-4-Multimodal (unified vision + audio + text in single framework).
PiZeroOptions Configuration options for pi-zero.
PixtralLargeOptions Options for Pixtral Large (124B decoder + 1B vision encoder from Mistral).
PixtralOptions Options for Pixtral (12B decoder + 400M vision encoder from Mistral).
PointLLMOptions Configuration options for PointLLM.
ProprietaryVLMOptions Base configuration options for proprietary VLM reference implementations.
QVQ72BOptions Configuration options for QVQ-72B: first open-source multimodal reasoning model from Qwen.
Qwen25VLOptions Options for Qwen2.5-VL (visual agent, 1hr+ video, bounding box/point localization).
Qwen2VLOptions Options for Qwen2-VL (Naive Dynamic Resolution, M-RoPE for multimodal position).
Qwen3VLOptions Options for Qwen3-VL (latest series with 2B/4B/8B/32B variants).
QwenVLOptions Options for Qwen-VL (visual window attention, multi-resolution, bounding box output).
RADIOv25Options Configuration options for RADIOv2.5, NVIDIA’s agglomerative vision foundation model.
RSGPTOptions Configuration options for RSGPT.
RT2Options Configuration options for RT-2.
RadFMOptions Configuration options for RadFM.
ReasoningVLMOptions Base configuration options for reasoning vision-language models with chain-of-thought capabilities.
RegionCLIPOptions Configuration options for the RegionCLIP model.
RemoteCLIPOptions Configuration options for the RemoteCLIP model.
RemoteSensingVLMOptions Base configuration options for remote sensing vision-language models.
SAMOptions Configuration options for the Segment Anything Model (SAM) vision encoder.
SEEDXOptions Configuration options for SEED-X: multi-granularity comprehension and generation model.
SceneLLMOptions Configuration options for Scene-LLM.
ShikraOptions Configuration options for Shikra: referential dialogue model with coordinate output for grounding.
ShowO2Options Configuration options for Show-o2: improved native unified multimodal model.
ShowOOptions Configuration options for Show-o: single transformer for unified understanding and generation.
SigLIP2Options Configuration options for SigLIP 2 (Multilingual Vision-Language Encoders with Improved Semantic Understanding).
SigLIPOptions Configuration options for the SigLIP (Sigmoid Loss for Language-Image Pre-training) model.
SigLIPSOOptions Configuration options for SigLIP-SO (Shape-Optimized SigLIP), a 400M vision encoder.
SkyEyeGPTOptions Configuration options for SkyEyeGPT.
SkyworkR1V2Options Configuration options for Skywork R1V2: hybrid RL (MPO + GRPO) for multimodal reasoning SOTA.
SkyworkR1VOptions Configuration options for Skywork R1V: cross-modal transfer of reasoning LLMs to vision.
SlowFastLLaVAOptions Configuration options for SlowFast-LLaVA: token-efficient slow/fast pathways for long video.
SmartEditOptions Configuration options for SmartEdit: enhanced instruction understanding for complex image editing.
SmolVLMOptions Options for SmolVLM (tiny efficient VLMs: 256M/500M/2.2B from HuggingFace).
SuryaOptions Configuration options for Surya: multi-language OCR with layout analysis support.
TextMonkeyOptions Configuration options for TextMonkey: OCR-free text understanding with shifted window attention.
ThreeDGraphLLMOptions Configuration options for 3DGraphLLM.
ThreeDLLMOptions Configuration options for 3D-LLM.
ThreeDVLAOptions Configuration options for 3D-VLA.
ThreeDVLMOptions Base configuration options for 3D vision-language models.
TransfusionOptions Configuration options for Transfusion: combined autoregressive and diffusion loss in single transformer.
UNITEROptions Configuration options for UNITER (Universal Image-TExt Representation).
UReaderOptions Configuration options for UReader: universal OCR-free visually-situated language model.
UnifiedVisionOptions Base configuration options for unified understanding + generation vision models.
VILAOptions Options for VILA (NVIDIA visual language models with joint pre-training on interleaved data).
VILAUOptions Options for VILA-U (NVIDIA unified understanding + generation via shared visual embedding).
ViLBERTOptions Configuration options for ViLBERT (Vision-and-Language BERT) with co-attention.
ViLTOptions Configuration options for ViLT (Vision-and-Language Transformer).
ViTOptions Configuration options for the Vision Transformer (ViT) model.
VideoChat2Options Configuration options for VideoChat2: progressive video training with diverse data.
VideoLLaMA2Options Configuration options for VideoLLaMA 2: spatial-temporal convolution for video tokens.
VideoLLaMA3Options Configuration options for VideoLLaMA 3: frontier multimodal for image and video.
VideoLLaVAOptions Configuration options for Video-LLaVA: united visual representation for video understanding.
VideoLanguageOptions Base configuration options for video-language models.
VinVLOptions Configuration options for VinVL (Visual Features in Vision-Language).
VisionEncoderOptions Base configuration options for standalone vision encoder models.
VisionLanguageActionOptions Base configuration options for Vision-Language-Action (VLA) models.
VisualBERTOptions Configuration options for VisualBERT single-stream fusion model.