Vision Language
395 public types in this namespace, organized by kind.
Models & Types (178)
| Type | Summary |
|---|---|
ALIGN<T> |
ALIGN (A Large-scale ImaGe and Noisy-text embedding) model for zero-shot classification and cross-modal retrieval using EfficientNet as the vision encoder. |
AquilaVL<T> |
AquilaVL: bilingual VLM aligned with Chinese and English visual-language understanding. |
Aria<T> |
Aria: mixture-of-experts multimodal model with efficient expert routing. |
BASIC<T> |
BASIC (Combined Scaling for Zero-shot Transfer Learning) model using CoAtNet vision encoder. |
BLIP3<T> |
BLIP-3 (xGen-MM): scaled vision-language model with interleaved data and any-to-any generation. |
BiomedCLIP<T> |
BiomedCLIP model fine-tuned on 15M biomedical image-text pairs from PubMed Central. |
BridgeTower<T> |
BridgeTower: cross-modal alignment through bridge layers connecting vision and text encoder layers. |
CLIPA<T> |
CLIPA (CLIP with Inverse scaling law and Accelerated training) model using progressive resolution. |
Cambrian1<T> |
Cambrian-1: Spatial Vision Aggregator with 35+ vision encoder combinations. |
Chameleon<T> |
Chameleon: early fusion with discrete tokens for all modalities. |
ClaudeVision<T> |
Claude Vision: reference implementation of Anthropic’s multimodal reasoning model. |
CoCa<T> |
CoCa (Contrastive Captioners): dual-loss model combining contrastive and captioning objectives. |
CogVLM2<T> |
CogVLM2: improved visual expert architecture with GLM-4 backbone and video understanding. |
CogVLM<T> |
CogVLM: deep fusion via visual expert module in every LLM layer. |
DFNCLIP<T> |
DFN-CLIP (Data Filtering Networks for CLIP) model using filtered high-quality training data. |
DINOX<T> |
DINO-X: strongest open-world perception model. |
DINOv2<T> |
DINOv2 self-supervised vision encoder producing universal visual features without labels. |
DINOv3<T> |
DINOv3 self-supervised vision encoder scaled to 7B parameters on 1.7B images. |
DeCLIP<T> |
DeCLIP (Data-efficient CLIP) model using self-supervised and nearest-neighbor supervision for data efficiency. |
DeepSeekVL2<T> |
DeepSeek-VL2: MoE vision-language model with dynamic tiling and multi-head latent attention. |
DeepSeekVL<T> |
DeepSeek-VL: real-world vision-language understanding with hybrid SigLIP + SAM-B encoder. |
DocPedia<T> |
DocPedia: frequency-domain document understanding model. |
DragonflyMed<T> |
Dragonfly-Med: medical image understanding model. |
Dragonfly<T> |
Dragonfly: multi-resolution visual encoding VLM for fine-grained understanding. |
EVACLIP<T> |
EVA-CLIP model combining EVA-02 masked image modeling backbone with CLIP contrastive training. |
Eagle25<T> |
Eagle 2.5: extended context VLM with video understanding capabilities. |
Eagle<T> |
Eagle: NVIDIA data-centric VLM with high-quality training data. |
Emu2<T> |
Emu2: scaled 37B unified understanding and generation model. |
Emu3<T> |
Emu3: next-token prediction unifies understanding and generation in a single model. |
EmuEdit<T> |
Emu Edit - precise image editing via recognition and generation tasks. |
Emu<T> |
Emu: unified VQA, captioning, and image generation via EVA-CLIP + LLM + regression head. |
FLIP<T> |
FLIP (Fast Language-Image Pre-training) model that accelerates CLIP training via random masking of image patches. |
FerretV2<T> |
Ferret-v2: improved referring and grounding with enhanced spatial understanding. |
Ferret<T> |
Ferret: spatial-aware visual sampler for free-form region referring and grounding. |
Florence2Region |
One region a Florence-2 task answer located: its label and its pixel box. |
Florence2TaskResult |
A parsed Florence-2 task answer: its text, plus any regions written as location tokens. |
Florence2<T> |
Florence-2 unified vision foundation model for captioning, detection, grounding, and OCR. |
Fuyu<T> |
Fuyu: no vision encoder; raw patches directly into transformer. |
GIT<T> |
GIT (Generative Image-to-Text): simple ViT encoder + autoregressive text decoder. |
GLaMM<T> |
GLaMM: Pixel Grounding Large Multimodal Model. |
GOTOCR2<T> |
GOT-OCR2: 580M unified OCR model for text, tables, charts, equations, and music scores. |
GPT4Point<T> |
GPT4Point: unified point-language understanding and generation. |
GR00TFlowMatchingActionHead<T> |
Flow-matching action head used by GR00T N1 to denoise continuous joint commands conditioned on the System-2 vision-language latent. |
GR00TN1<T> |
GR00T N1: NVIDIA’s open foundation model for generalist humanoid robots, combining a SigLIP + Eagle-2 vision-language System-2 reasoner with a flow-matching DiT action head as System 1 (NVIDIA 2025, arXiv:2503.14734). |
GeminiVision<T> |
Gemini Vision: reference implementation of Google’s native multimodal Mixture of Experts model. |
Gemma3<T> |
Gemma 3: Google’s 3B-72B VLM with Native Dynamic Resolution ViT and 128k context. |
GeoChat<T> |
GeoChat: grounded VLM for satellite imagery understanding. |
GlammAnswer<T> |
GLaMM’s answer: the text and a mask for every [SEG] in it. |
GlammSample<T> |
A GLaMM training sample: an image, a conversation about it, the regions its <bbox> tokens name, and a mask for every [SEG] in the answers. |
GlammTurn |
One turn of a GLaMM conversation. |
GrokVision<T> |
Grok Vision: reference implementation of xAI’s real-time multimodal model. |
Groma<T> |
Groma: localized visual tokenization for grounded understanding. |
GroundedSAM2<T> |
Grounded SAM 2: Grounding DINO finds the boxes a caption names, and SAM 2 segments each box. |
GroundedSegment<T> |
One region a caption names, with its SAM 2 mask. |
GroundingBox |
A box that a caption’s phrase names, for training a grounding model. |
GroundingDINO15<T> |
Grounding DINO 1.5: enhanced open-set detection with improved architecture. |
GroundingDINO<T> |
Grounding DINO: open-set detection combining DINO with grounded pre-training. |
GroundingDetection<T> |
A region a grounding model found for a caption. |
HelixDualSystemRunner<T> |
Streaming controller that coordinates Helix’s fast System-1 visuomotor policy (200 Hz) and slow System-2 VLM (7–9 Hz) per Figure AI’s dual-rate dual-system architecture. |
HelixSystem2Latent<T> |
Conditioning signal that Helix’s slow System-2 VLM produces for the fast System-1 controller. |
Helix<T> |
Helix: dual-system vision-language-action model for full-body humanoid control (Figure AI, 2025, arXiv:2502.07092). |
IDEFICS2<T> |
IDEFICS2: 8B efficient VLM with SigLIP encoder and Mistral-7B decoder. |
IDEFICS3<T> |
IDEFICS3: state-of-the-art 8B VLM trained with Docmatix for document understanding. |
IDEFICS<T> |
IDEFICS: 80B open reproduction of Flamingo for interleaved image-text generation. |
InstructBLIP<T> |
InstructBLIP: instruction-tuned BLIP-2 for zero-shot generalization across vision-language tasks. |
InternVL25<T> |
InternVL2.5: improved training data and strategy over InternVL2 with InternLM2.5 backbone. |
InternVL2<T> |
InternVL2: dynamic resolution with pixel shuffle and InternLM2 backbone. |
InternVL3<T> |
InternVL3: 78B SOTA open-source model with advanced training and test-time recipes. |
InternVL<T> |
InternVL: 6B InternViT with progressive alignment and LLaMA language backbone. |
InternViT<T> |
InternViT, a 6B-parameter ViT used as the vision encoder in the InternVL series. |
JanusPro<T> |
Janus-Pro: unified multimodal understanding and generation with decoupled vision encoders (Chen et al., DeepSeek 2025, arXiv:2501.17811). |
Janus<T> |
Janus: decoupled visual encoding for understanding vs generation. |
KOSMOS1<T> |
KOSMOS-1: multimodal large language model with visual tokens embedded in causal LM. |
KOSMOS2<T> |
KOSMOS-2: grounded multimodal large language model with text spans linked to bounding boxes. |
KimiVLThinking<T> |
Kimi-VL-Thinking: long chain-of-thought reasoning with RL alignment. |
KimiVL<T> |
Kimi-VL: MoE VLM with MoonViT and long-context processing. |
LEOVL<T> |
LEO-VL: efficient 3D scene representation from multi-view RGB-D. |
LLM2CLIP<T> |
LLM2CLIP model enhancing CLIP’s text encoder with LLM embeddings for richer semantic understanding. |
LLaVA15<T> |
LLaVA-1.5: improved baselines with visual instruction tuning using MLP cross-modal connector. |
LLaVACoT<T> |
LLaVA-CoT: chain-of-thought visual reasoning with structured output. |
LLaVAMed<T> |
LLaVA-Med: biomedical VLM with GPT-4-level visual understanding. |
LLaVANeXTVideo<T> |
LLaVA-NeXT-Video: average pooling for frame token reduction. |
LLaVANeXT<T> |
LLaVA-NeXT: improved reasoning, OCR, and world knowledge via dynamic high-resolution AnyRes. |
LLaVAOneVision15<T> |
LLaVA-OneVision-1.5: fully open model outperforming Qwen2.5-VL on 18/27 benchmarks. |
LLaVAOneVision<T> |
LLaVA-OneVision: single model for images, multi-image, and videos with easy visual task transfer. |
LLaVAVideo<T> |
LLaVA-Video: synthetic dataset-trained video instruction model. |
LXMERT<T> |
LXMERT (Learning Cross-Modality Encoder Representations from Transformers) with three-encoder architecture. |
LiT<T> |
LiT (Locked-image Tuning) model that freezes a pre-trained image encoder and only trains the text encoder for efficient contrastive image-text alignment. |
Llama32Vision<T> |
Llama 3.2 Vision: Meta’s 11B/90B vision models for edge/mobile deployment. |
LongVILA<T> |
LongVILA: long-context visual language model for 1hr+ videos. |
METER<T> |
METER (Multimodal End-to-end TransformER) with systematic VLP component study. |
MGIE<T> |
MGIE - instruction-based image editing guided by a multimodal large language model. |
MPLUGDocOwl15<T> |
mPLUG-DocOwl 1.5: unified structure learning achieving SOTA on 10 document benchmarks. |
MPLUGDocOwl2<T> |
mPLUG-DocOwl 2: high-res compressing for multi-page document understanding. |
MPLUGDocOwl<T> |
mPLUG-DocOwl: modular MLLM for document understanding with visual abstractor. |
MPLUGOwl2<T> |
mPLUG-Owl2: improved modular design for multi-image understanding. |
MPLUGOwl3<T> |
mPLUG-Owl3: enhanced with hyper-attention for long visual sequences. |
MPLUGOwl<T> |
mPLUG-Owl: modular VLM with visual abstractor for vision-language alignment. |
Mantis<T> |
Mantis: interleaved multi-image VLM for complex visual reasoning. |
Maya<T> |
Maya: multilingual multimodal VLM supporting 8 languages. |
MedCLIP<T> |
MedCLIP model using decoupled semantic matching for medical image-text alignment. |
MedFlamingo<T> |
Med-Flamingo: few-shot medical visual question answering via Flamingo architecture. |
MetaCLIP<T> |
MetaCLIP (Demystifying CLIP Data) reproduces and improves CLIP via a fully transparent data curation pipeline. |
MiniCPMV<T> |
MiniCPM-V: efficient on-device VLM with strong OCR and multilingual capabilities. |
MiniCPMo<T> |
MiniCPM-o: omnimodal VLM with speech and real-time streaming support. |
MiniGPT4<T> |
MiniGPT-4: ViT + Q-Former aligned with Vicuna via single projection layer. |
MiniGPTv2<T> |
MiniGPT-v2: unified interface for multi-task VL learning with task-specific tokens. |
Molmo<T> |
Molmo: open VLM with pointing and grounding capabilities. |
Monkey<T> |
Monkey: high-resolution VLM with multi-level description generation. |
Moondream<T> |
Moondream: tiny yet capable VLM for resource-constrained environments. |
NVLM<T> |
NVLM 1.0: NVIDIA’s cross-attention + decoder-only hybrid VLM that retains text performance. |
OWLViT<T> |
OWL-ViT: open-vocabulary object detection via ViT + CLIP alignment. |
OWLv2<T> |
OWLv2: self-training for scaling open-vocabulary detection. |
Octo<T> |
Octo: open-source generalist robot policy trained on 800K demonstrations. |
OmniGen2<T> |
OmniGen2: dual-path architecture with parameter decoupling. |
OpenCLIP<T> |
OpenCLIP (Open Contrastive Language-Image Pre-training) model for zero-shot classification and cross-modal retrieval using open-source training data. |
OpenFlamingo<T> |
OpenFlamingo: open-source reproduction of Flamingo with perceiver resampler. |
Oscar<T> |
Oscar (Object-Semantics Aligned pre-training) using object tags as anchor points. |
Ovis<T> |
Ovis: structural embedding alignment VLM with visual token scaling. |
PLLaVA<T> |
PLLaVA: parameter-free pooling extension from images to video. |
PaLI3<T> |
PaLI-3: efficient PaLI with SigLIP ViT encoder, smaller and better. |
PaLIX<T> |
PaLI-X: scaled PaLI to 55B with improved component fine-tuning. |
PaLI<T> |
PaLI (Pathways Language and Image model): large-scale ViT-e vision encoder + mT5 text decoder. |
PaLME<T> |
PaLM-E: 562B embodied multimodal language model for robotic planning. |
PathVLM<T> |
PathVLM: histopathology-specific vision-language model. |
PerceptionEncoder<T> |
Meta’s Perception Encoder for multimodal alignment tasks with global and dense features. |
Phi3Vision<T> |
Phi-3-Vision: compact 4.2B parameter model with strong vision capabilities via curated data. |
Phi4Multimodal<T> |
Phi-4-Multimodal: unified vision + audio + text in a single framework. |
PiZero<T> |
pi-zero: PaliGemma VLM with action expert for 8 robot embodiments. |
PixtralLarge<T> |
Pixtral Large: Mistral’s 124B decoder + 1B vision encoder. |
Pixtral<T> |
Pixtral: Mistral’s 12B decoder + 400M vision encoder. |
PointLLM<T> |
PointLLM: LLM understanding of colored 3D point clouds. |
QVQ72B<T> |
QVQ-72B: first open-source multimodal reasoning model from Qwen. |
Qwen25VL<T> |
Qwen2.5-VL: visual agent with 1hr+ video, bounding box/point localization capabilities. |
Qwen2VL<T> |
Qwen2-VL: Naive Dynamic Resolution with M-RoPE for multimodal positional encoding. |
Qwen3VL<T> |
Qwen3-VL: latest series with 2B/4B/8B/32B variants and cross-attention resampler. |
QwenVL<T> |
Qwen-VL: visual window attention, multi-resolution, bounding box output via cross-attention resampler. |
RADIOv25<T> |
RADIOv2.5 agglomerative vision foundation model distilling DINOv2, SAM, SigLIP, and CLIP. |
RSGPT<T> |
RSGPT: remote sensing GPT based on InstructBLIP architecture. |
RT2ActionTokenizer<T> |
RT-2’s action-as-text tokenizer. |
RT2<T> |
RT-2: vision-language-action model that transfers web knowledge to robotic control (Brohan et al., Google DeepMind, 2023, arXiv:2307.15818). |
RadFM<T> |
RadFM: 3D ViT with perceiver for radiology report generation and VQA. |
RegionCLIP<T> |
RegionCLIP model extending CLIP to learn region-level (object-level) visual representations. |
RemoteCLIP<T> |
RemoteCLIP model adapted for remote sensing and satellite imagery understanding. |
SAM<T> |
Segment Anything Model (SAM) vision encoder for promptable image segmentation. |
SEEDX<T> |
SEED-X: multi-granularity comprehension and generation model. |
SceneLLM<T> |
Scene-LLM: voxel-based 3D scene understanding with language models. |
Shikra<T> |
Shikra: referential dialogue model with coordinate output for grounding. |
ShowO2<T> |
Show-o2: improved native unified multimodal model. |
ShowO<T> |
Show-o: single transformer for unified understanding and generation. |
SigLIP2<T> |
SigLIP 2 (Multilingual Vision-Language Encoders with Improved Semantic Understanding) for contrastive encoding, zero-shot classification, and captioning. |
SigLIPSO<T> |
SigLIP-SO (Shape-Optimized SigLIP) 400M vision encoder widely used as a VLM backbone. |
SigLIP<T> |
SigLIP (Sigmoid Loss for Language-Image Pre-training) model for zero-shot classification and cross-modal retrieval with improved batch scaling. |
SkyEyeGPT<T> |
SkyEyeGPT: unified remote sensing vision-language tasks with 968K instruction samples. |
SkyworkR1V2<T> |
Skywork R1V2: hybrid RL (MPO + GRPO) for multimodal reasoning SOTA. |
SkyworkR1V<T> |
Skywork R1V: cross-modal transfer of reasoning LLMs to vision. |
SlowFastLLaVA<T> |
SlowFast-LLaVA: token-efficient slow/fast pathways for long video. |
SmartEdit<T> |
SmartEdit - complex instruction-based image editing with a multimodal LLM. |
SmolVLM<T> |
SmolVLM: compact and efficient VLM designed for edge deployment. |
Surya<T> |
Surya: multi-language OCR with layout analysis support. |
TextMonkey<T> |
TextMonkey: OCR-free text understanding with shifted window attention. |
ThreeDGraphLLM<T> |
3DGraphLLM: 3D scene graph as LLM input for spatial reasoning. |
ThreeDLLM<T> |
3D-LLM: injects 3D spatial features into large language models. |
ThreeDVLA<T> |
3D-VLA: connects vision-language-action to 3D world via generative world model. |
Transfusion<T> |
Transfusion: combined autoregressive and diffusion loss in single transformer. |
UNITER<T> |
UNITER (Universal Image-TExt Representation) with conditional masking pre-training. |
UReader<T> |
UReader: universal OCR-free visually-situated language model. |
VILAU<T> |
VILA-U: unified VLM with autoregressive visual generation and understanding. |
VILA<T> |
VILA: VLM pre-trained with interleaved image-text data for enhanced in-context learning. |
ViLBERT<T> |
ViLBERT (Vision-and-Language BERT) with co-attention between parallel vision and language streams. |
ViLT<T> |
ViLT (Vision-and-Language Transformer) with minimal architecture using patch embeddings. |
ViT<T> |
Vision Transformer (ViT) that splits images into patches and processes them with a standard Transformer. |
VideoChat2<T> |
VideoChat2: progressive video training with diverse data. |
VideoLLaMA2<T> |
VideoLLaMA 2: spatial-temporal convolution for video tokens. |
VideoLLaMA3<T> |
VideoLLaMA 3: frontier multimodal for image and video. |
VideoLLaVA<T> |
Video-LLaVA: united visual representation for video understanding. |
VinVL<T> |
VinVL (Visual Features in Vision-Language) with enhanced object detection features. |
VisualBERT<T> |
VisualBERT: single-stream transformer that concatenates visual and text tokens. |
Base Classes (1)
| Type | Summary |
|---|---|
VisionLanguageModelBase<T> |
Base class for vision-language neural networks that can operate in both ONNX inference and native training modes. |
Interfaces (19)
| Type | Summary |
|---|---|
IContrastiveVisionLanguageModel<T> |
Interface for contrastive vision-language models that align image and text embeddings in a shared space for zero-shot classification and cross-modal retrieval. |
IDocumentUnderstandingModel<T> |
Interface for document understanding models that process text-heavy images, documents, charts, and tables. |
IGenerativeVisionLanguageModel<T> |
Interface for generative vision-language models that produce text output from visual input. |
IGroundedConversationTrainable<T> |
A model that trains on grounded conversations: an image, a conversation about it, the regions its questions name, and a mask for every [SEG] in its answers. |
IGroundingTrainable<T> |
A grounding model that trains on grounded phrases: an image, a caption, and the boxes the caption’s phrases name. |
IImageEditingVLM<T> |
Interface for VLMs that edit images based on natural language instructions. |
IInstructionTunedVLM<T> |
Interface for instruction-tuned vision-language models that generate conversational responses from visual input conditioned on user instructions. |
IMedicalVLM<T> |
Interface for medical domain vision-language models specializing in biomedical image understanding. |
IProprietaryVLM<T> |
Interface for reference implementations of proprietary VLM architectures. |
IReasoningVLM<T> |
Interface for vision-language models with chain-of-thought reasoning capabilities. |
IRemoteSensingVLM<T> |
Interface for remote sensing vision-language models specializing in satellite and aerial imagery. |
ITextEncoder<T> |
Interface for text encoders that extract feature representations from text. |
IThreeDVisionLanguageModel<T> |
Interface for 3D vision-language models that understand point clouds, 3D scenes, and spatial relationships. |
IUnifiedVisionModel<T> |
Interface for unified vision models that can both understand and generate images. |
IVideoLanguageModel<T> |
Interface for video-language models that process video frames for temporal understanding and QA. |
IVisionLanguageAction<T> |
Interface for Vision-Language-Action (VLA) models that connect visual understanding and language reasoning to physical robotic actions. |
IVisionLanguageFusionModel<T> |
Interface for vision-language fusion models that combine image and text features for tasks like VQA, image-text matching, and cross-modal retrieval. |
IVisualEncoder<T> |
Interface for vision encoders that extract feature representations from images. |
IVisualGroundingModel<T> |
Interface for visual grounding models that localize objects or regions from natural language descriptions. |
Enums (14)
| Type | Summary |
|---|---|
ContrastiveLossType |
Specifies the contrastive loss function type. |
DomainSpecialization |
Specifies the domain specialization for a CLIP-family model. |
Florence2ModelSize |
Specifies the Florence-2 model size variant. |
Florence2Task |
The Florence-2 tasks. |
FusionType |
Specifies how vision and language features are fused in a VLM. |
GenerativeArchitectureType |
Specifies the architecture type for generative vision-language models. |
InstructionTunedArchitectureType |
Specifies the architecture type for instruction-tuned vision-language models. |
ModelPrecision |
Specifies the precision mode for model inference and training. |
PoolingStrategy |
Specifies the global feature pooling strategy for vision encoders. |
PositionalEmbeddingType |
Specifies the positional embedding type for vision transformers. |
PretrainingDataset |
Specifies the pre-training dataset used for the model. |
TextEncoderVariant |
Specifies the text encoder architecture. |
ViTVariant |
Specifies the Vision Transformer (ViT) variant used as the image encoder. |
VisualFeatureType |
Specifies the visual feature extraction method for foundational VLMs. |
Options & Configuration (183)
| Type | Summary |
|---|---|
ALIGNOptions |
Configuration options for the ALIGN (A Large-scale ImaGe and Noisy-text embedding) model. |
AquilaVLOptions |
Options for Aquila-VL (BAAI LLaVA-style VLM with Qwen2.5 + SigLIP). |
AriaOptions |
Options for Aria (Rhymes AI MoE VLM with 3.9B active params and 64K context multimodal). |
BASICOptions |
Configuration options for the BASIC (Batch-wise Alignment of Scaled Image-text Contrastive) model. |
BLIP3Options |
Options for BLIP-3 (xGen-MM) with interleaved data and any-to-any generation. |
BiomedCLIPOptions |
Configuration options for the BiomedCLIP model. |
BridgeTowerOptions |
Configuration options for BridgeTower cross-modal alignment model. |
CLIPAOptions |
Configuration options for the CLIPA (CLIP with Inverse scaling law and Accelerated training) model. |
Cambrian1Options |
Options for Cambrian-1 (Spatial Vision Aggregator with 35+ vision encoder combinations). |
ChameleonOptions |
Configuration options for Chameleon: early fusion with discrete tokens for all modalities. |
ClaudeVisionOptions |
Configuration options for Claude Vision. |
CoCaOptions |
Options for CoCa (Contrastive Captioners) with dual loss. |
CogVLM2Options |
Options for CogVLM2 (improved architecture with video understanding, GLM-4/LLaMA-3). |
CogVLMOptions |
Options for CogVLM (deep fusion via visual expert module in every LLM layer). |
ContrastiveEncoderOptions |
Base configuration options for contrastive vision-language encoders (CLIP-family models). |
DFNCLIPOptions |
Configuration options for the DFN-CLIP (Data Filtering Networks for CLIP) model. |
DINOXOptions |
Configuration options for DINO-X: strongest open-world perception model. |
DINOv2Options |
Configuration options for the DINOv2 self-supervised vision encoder. |
DINOv3Options |
Configuration options for DINOv3, the next evolution of self-supervised vision encoders. |
DeCLIPOptions |
Configuration options for the DeCLIP (Data-efficient CLIP) model. |
DeepSeekVL2Options |
Options for DeepSeek-VL2 (MoE, dynamic tiling, multi-head latent attention). |
DeepSeekVLOptions |
Options for DeepSeek-VL (hybrid vision encoder with SigLIP + SAM-B). |
DocPediaOptions |
Configuration options for DocPedia: frequency-domain document understanding model. |
DocumentVLMOptions |
Base configuration options for document understanding vision-language models. |
DragonflyMedOptions |
Configuration options for Dragonfly-Med. |
DragonflyOptions |
Options for Dragonfly (efficient multi-resolution visual encoding from Together.ai). |
EVACLIPOptions |
Configuration options for the EVA-CLIP model. |
Eagle25Options |
Options for Eagle 2.5 (NVIDIA long-context multimodal for video + high-res images). |
EagleOptions |
Options for Eagle (NVIDIA data-centric VLM with learned data mixing). |
EditingVLMOptions |
Base configuration options for image editing VLMs. |
Emu2Options |
Options for Emu2 (scaled to 37B; enhanced understanding + generation). |
Emu3Options |
Options for Emu3 (next-token prediction unifies understanding + generation). |
EmuEditOptions |
Configuration options for Emu Edit: precise image editing via recognition and generation tasks. |
EmuOptions |
Options for Emu (unified VQA, captioning, generation via EVA-CLIP + LLM + diffusion). |
FLIPOptions |
Configuration options for the FLIP (Fast Language-Image Pre-training) model. |
FerretOptions |
Configuration options for Ferret: spatial-aware visual sampler for free-form region referring and grounding. |
FerretV2Options |
Configuration options for Ferret-v2: improved referring and grounding with enhanced spatial understanding. |
Florence2Options |
Configuration options for Florence-2, Microsoft’s unified vision foundation model. |
FoundationalVLMOptions |
Base configuration options for foundational vision-language fusion models. |
FuyuOptions |
Options for Fuyu (no vision encoder; raw patches directly into transformer). |
GITOptions |
Options for GIT (Generative Image-to-Text) with simple ViT + text decoder. |
GLaMMOptions |
Configuration options for GLaMM (Rasheed et al. |
GOTOCR2Options |
Configuration options for GOT-OCR2, the General OCR Theory model (Wei et al., 2024). |
GPT4PointOptions |
Configuration options for GPT4Point. |
GR00TN1Options |
Configuration options for GR00T N1. |
GeminiVisionOptions |
Configuration options for Gemini Vision. |
Gemma3Options |
Options for Gemma 3 (3B-72B, Native Dynamic Resolution ViT, 128k context, 29 languages). |
GenerativeVLMOptions |
Base configuration options for generative vision-language models. |
GeoChatOptions |
Configuration options for GeoChat. |
GrokVisionOptions |
Configuration options for Grok Vision. |
GromaOptions |
Configuration options for Groma: localized visual tokenization for grounded understanding. |
GroundedSAM2Options |
Configuration options for Grounded SAM 2: Grounding DINO finds the boxes a caption names, and SAM 2 segments each box. |
GroundingDINO15Options |
Configuration options for Grounding DINO 1.5: enhanced open-set detection with improved architecture. |
GroundingDINOOptions |
Configuration options for Grounding DINO (Liu et al. |
GroundingVLMOptions |
Base configuration options for visual grounding models. |
HelixOptions |
Configuration options for Helix. |
IDEFICS2Options |
Options for IDEFICS2 (8B; SigLIP + Mistral-7B with Perceiver pooling). |
IDEFICS3Options |
Options for IDEFICS3 (state-of-the-art 8B trained on open datasets + Docmatix). |
IDEFICSOptions |
Options for IDEFICS (80B open reproduction of Flamingo). |
InstructBLIPOptions |
Options for InstructBLIP (instruction-tuned Q-Former for zero-shot generalization). |
InstructionTunedVLMOptions |
Base configuration options for instruction-tuned vision-language models. |
InternVL25Options |
Options for InternVL2.5 (improved training data and strategy over InternVL2). |
InternVL2Options |
Options for InternVL2 (dynamic resolution, pixel shuffle, InternLM2 backbone). |
InternVL3Options |
Options for InternVL3 (78B, MMMU 72.2 SOTA among open-source). |
InternVLOptions |
Options for InternVL (6B InternViT + LLaMA with progressive alignment). |
InternViTOptions |
Configuration options for InternViT, the vision encoder used in the InternVL series. |
JanusOptions |
Configuration options for Janus: decoupled visual encoding for understanding vs generation. |
JanusProOptions |
Configuration options for Janus-Pro: scaled data and model with optimized training strategy. |
KOSMOS1Options |
Options for KOSMOS-1 (Huang et al. |
KOSMOS2Options |
Options for KOSMOS-2 (grounded multimodal with text spans linked to bounding boxes; Peng et al. |
KimiVLOptions |
Configuration options for Kimi-VL: MoE VLM with MoonViT and long-context processing. |
KimiVLThinkingOptions |
Configuration options for Kimi-VL-Thinking: long chain-of-thought reasoning with RL alignment. |
KosmosOptions |
Hyperparameters shared by KOSMOS-1 and KOSMOS-2: - A CLIP ViT-L/14 image encoder: 224 px, 1024 wide, 24 layers, 16 heads, patch 14. |
LEOVLOptions |
Configuration options for LEO-VL. |
LLM2CLIPOptions |
Configuration options for the LLM2CLIP model. |
LLaVA15Options |
Options for LLaVA-1.5 (MLP cross-modal connector, higher-res CLIP, academic task SOTA). |
LLaVACoTOptions |
Configuration options for LLaVA-CoT: chain-of-thought visual reasoning with structured output. |
LLaVAMedOptions |
Configuration options for LLaVA-Med. |
LLaVANeXTOptions |
Options for LLaVA-NeXT (dynamic high-resolution via AnyRes, stronger reasoning). |
LLaVANeXTVideoOptions |
Configuration options for LLaVA-NeXT-Video: average pooling for frame token reduction. |
LLaVAOneVision15Options |
Options for LLaVA-OneVision-1.5 (fully open, outperforms Qwen2.5-VL on 18/27 benchmarks). |
LLaVAOneVisionOptions |
Options for LLaVA-OneVision (single model for images, multi-image, and videos). |
LLaVAVideoOptions |
Configuration options for LLaVA-Video: synthetic dataset-trained video instruction model. |
LXMERTOptions |
Configuration options for LXMERT cross-modal encoder. |
LiTOptions |
Configuration options for the LiT (Locked-image Tuning) model. |
Llama32VisionOptions |
Options for Llama 3.2 Vision (11B/90B vision models for edge/mobile deployment). |
LongVILAOptions |
Configuration options for LongVILA: long-context visual language model for 1hr+ videos. |
METEROptions |
Configuration options for METER (Multimodal End-to-end TransformER). |
MGIEOptions |
Configuration options for MGIE: MLLM-guided image editing with LLaVA-based instruction understanding. |
MPLUGDocOwl15Options |
Configuration options for mPLUG-DocOwl 1.5: unified structure learning achieving SOTA on 10 document benchmarks. |
MPLUGDocOwl2Options |
Configuration options for mPLUG-DocOwl 2: high-res compressing for multi-page document understanding. |
MPLUGDocOwlOptions |
Configuration options for mPLUG-DocOwl: modular MLLM for document understanding with visual abstractor. |
MPLUGOwl2Options |
Options for mPLUG-Owl2 (improved modular design for multi-image understanding). |
MPLUGOwl3Options |
Options for mPLUG-Owl3 (enhanced with hyper-attention for long visual sequences). |
MPLUGOwlOptions |
Options for mPLUG-Owl (modular VLM with visual abstractor). |
MantisOptions |
Options for Mantis (multi-image reasoning with interleaved data). |
MayaOptions |
Options for Maya (multilingual VLM for low-resource languages). |
MedCLIPOptions |
Configuration options for the MedCLIP model. |
MedFlamingoOptions |
Configuration options for Med-Flamingo. |
MedicalVLMOptions |
Base configuration options for medical domain vision-language models. |
MetaCLIPOptions |
Configuration options for the MetaCLIP model. |
MiniCPMVOptions |
Options for MiniCPM-V (8B model achieving GPT-4V-level on select benchmarks). |
MiniCPMoOptions |
Options for MiniCPM-o (omni-modal with real-time speech + vision). |
MiniGPT4Options |
Options for MiniGPT-4 (ViT + Q-Former aligned with Vicuna via single projection layer). |
MiniGPTv2Options |
Options for MiniGPT-v2 (multi-task with task-specific tokens, LLaMA-2 backbone). |
MolmoOptions |
Options for Molmo (AI2 open VLM with diverse data mixture and pointing capability). |
MonkeyOptions |
Options for Monkey (multi-level description generation with high resolution). |
MoondreamOptions |
Options for Moondream (1.8B; SigLIP + Phi-1.5; ideal for edge/mobile). |
NVLMOptions |
Options for NVLM 1.0 (cross-attention + decoder-only hybrid that retains text performance). |
OWLViTOptions |
Configuration options for OWL-ViT: open-vocabulary object detection via ViT + CLIP alignment. |
OWLv2Options |
Configuration options for OWLv2: self-training for scaling open-vocabulary detection. |
OctoOptions |
Configuration options for Octo. |
OmniGen2Options |
Configuration options for OmniGen2: dual-path architecture with parameter decoupling. |
OpenCLIPOptions |
Configuration options for the OpenCLIP model. |
OpenFlamingoOptions |
Options for OpenFlamingo (open-source Flamingo with perceiver resampler). |
OscarOptions |
Configuration options for Oscar (Object-Semantics Aligned pre-training). |
OvisOptions |
Options for Ovis (structural embedding alignment for visual tokens). |
PLLaVAOptions |
Configuration options for PLLaVA: parameter-free pooling extension from images to video. |
PaLI3Options |
Options for PaLI-3 (efficient PaLI with SigLIP ViT, smaller and better). |
PaLIOptions |
Options for PaLI (Pathways Language and Image model) with ViT-e + mT5. |
PaLIXOptions |
Options for PaLI-X (scaled PaLI to 55B with improved fine-tuning). |
PaLMEOptions |
Configuration options for PaLM-E. |
PathVLMOptions |
Configuration options for PathVLM. |
PerceptionEncoderOptions |
Configuration options for Meta’s Perception Encoder vision foundation model. |
Phi3VisionOptions |
Options for Phi-3-Vision (compact 4.2B with strong vision via curated data). |
Phi4MultimodalOptions |
Options for Phi-4-Multimodal (unified vision + audio + text in single framework). |
PiZeroOptions |
Configuration options for pi-zero. |
PixtralLargeOptions |
Options for Pixtral Large (124B decoder + 1B vision encoder from Mistral). |
PixtralOptions |
Options for Pixtral (12B decoder + 400M vision encoder from Mistral). |
PointLLMOptions |
Configuration options for PointLLM. |
ProprietaryVLMOptions |
Base configuration options for proprietary VLM reference implementations. |
QVQ72BOptions |
Configuration options for QVQ-72B: first open-source multimodal reasoning model from Qwen. |
Qwen25VLOptions |
Options for Qwen2.5-VL (visual agent, 1hr+ video, bounding box/point localization). |
Qwen2VLOptions |
Options for Qwen2-VL (Naive Dynamic Resolution, M-RoPE for multimodal position). |
Qwen3VLOptions |
Options for Qwen3-VL (latest series with 2B/4B/8B/32B variants). |
QwenVLOptions |
Options for Qwen-VL (visual window attention, multi-resolution, bounding box output). |
RADIOv25Options |
Configuration options for RADIOv2.5, NVIDIA’s agglomerative vision foundation model. |
RSGPTOptions |
Configuration options for RSGPT. |
RT2Options |
Configuration options for RT-2. |
RadFMOptions |
Configuration options for RadFM. |
ReasoningVLMOptions |
Base configuration options for reasoning vision-language models with chain-of-thought capabilities. |
RegionCLIPOptions |
Configuration options for the RegionCLIP model. |
RemoteCLIPOptions |
Configuration options for the RemoteCLIP model. |
RemoteSensingVLMOptions |
Base configuration options for remote sensing vision-language models. |
SAMOptions |
Configuration options for the Segment Anything Model (SAM) vision encoder. |
SEEDXOptions |
Configuration options for SEED-X: multi-granularity comprehension and generation model. |
SceneLLMOptions |
Configuration options for Scene-LLM. |
ShikraOptions |
Configuration options for Shikra: referential dialogue model with coordinate output for grounding. |
ShowO2Options |
Configuration options for Show-o2: improved native unified multimodal model. |
ShowOOptions |
Configuration options for Show-o: single transformer for unified understanding and generation. |
SigLIP2Options |
Configuration options for SigLIP 2 (Multilingual Vision-Language Encoders with Improved Semantic Understanding). |
SigLIPOptions |
Configuration options for the SigLIP (Sigmoid Loss for Language-Image Pre-training) model. |
SigLIPSOOptions |
Configuration options for SigLIP-SO (Shape-Optimized SigLIP), a 400M vision encoder. |
SkyEyeGPTOptions |
Configuration options for SkyEyeGPT. |
SkyworkR1V2Options |
Configuration options for Skywork R1V2: hybrid RL (MPO + GRPO) for multimodal reasoning SOTA. |
SkyworkR1VOptions |
Configuration options for Skywork R1V: cross-modal transfer of reasoning LLMs to vision. |
SlowFastLLaVAOptions |
Configuration options for SlowFast-LLaVA: token-efficient slow/fast pathways for long video. |
SmartEditOptions |
Configuration options for SmartEdit: enhanced instruction understanding for complex image editing. |
SmolVLMOptions |
Options for SmolVLM (tiny efficient VLMs: 256M/500M/2.2B from HuggingFace). |
SuryaOptions |
Configuration options for Surya: multi-language OCR with layout analysis support. |
TextMonkeyOptions |
Configuration options for TextMonkey: OCR-free text understanding with shifted window attention. |
ThreeDGraphLLMOptions |
Configuration options for 3DGraphLLM. |
ThreeDLLMOptions |
Configuration options for 3D-LLM. |
ThreeDVLAOptions |
Configuration options for 3D-VLA. |
ThreeDVLMOptions |
Base configuration options for 3D vision-language models. |
TransfusionOptions |
Configuration options for Transfusion: combined autoregressive and diffusion loss in single transformer. |
UNITEROptions |
Configuration options for UNITER (Universal Image-TExt Representation). |
UReaderOptions |
Configuration options for UReader: universal OCR-free visually-situated language model. |
UnifiedVisionOptions |
Base configuration options for unified understanding + generation vision models. |
VILAOptions |
Options for VILA (NVIDIA visual language models with joint pre-training on interleaved data). |
VILAUOptions |
Options for VILA-U (NVIDIA unified understanding + generation via shared visual embedding). |
ViLBERTOptions |
Configuration options for ViLBERT (Vision-and-Language BERT) with co-attention. |
ViLTOptions |
Configuration options for ViLT (Vision-and-Language Transformer). |
ViTOptions |
Configuration options for the Vision Transformer (ViT) model. |
VideoChat2Options |
Configuration options for VideoChat2: progressive video training with diverse data. |
VideoLLaMA2Options |
Configuration options for VideoLLaMA 2: spatial-temporal convolution for video tokens. |
VideoLLaMA3Options |
Configuration options for VideoLLaMA 3: frontier multimodal for image and video. |
VideoLLaVAOptions |
Configuration options for Video-LLaVA: united visual representation for video understanding. |
VideoLanguageOptions |
Base configuration options for video-language models. |
VinVLOptions |
Configuration options for VinVL (Visual Features in Vision-Language). |
VisionEncoderOptions |
Base configuration options for standalone vision encoder models. |
VisionLanguageActionOptions |
Base configuration options for Vision-Language-Action (VLA) models. |
VisualBERTOptions |
Configuration options for VisualBERT single-stream fusion model. |