InternVL<T>

Models & Types · AiDotNet.VisionLanguage.InstructionTuned

InternVL: 6B InternViT with progressive alignment and LLaMA language backbone.

Inheritance: NeuralNetworkBase<T> → VisionLanguageModelBase<T> → InternVL<T>

Implements: IInstructionTunedVLM<T>

For Beginners

InternVL uses a massive 6 billion parameter vision encoder called InternViT-6B — one of the largest vision-only models ever built. It progressively aligns this vision encoder with a LLaMA language model through a two-stage training process: first contrastive learning (matching images to text) and then generative training (learning to produce text from images). The huge vision encoder provides extremely rich visual representations that capture both fine details and high-level semantics, making it effective for a wide range of visual understanding tasks. Default values follow the original paper settings.

How It Works

InternVL (Chen et al., 2024) scales up the vision foundation model to 6B parameters (InternViT-6B) and progressively aligns it with LLaMA through contrastive learning and generative training. The large-scale vision encoder provides rich visual representations for instruction-following tasks.

References:

  • Paper: “InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks” (Chen et al., 2024)

Example

// Create an InternVL model for visual-linguistic understanding
// using InternViT-6B with progressive LLaMA alignment
var architecture = new NeuralNetworkArchitecture<double>(
    inputType: InputType.TwoDimensional,
    taskType: NeuralNetworkTaskType.ImageClassification,
    inputHeight: 224, inputWidth: 224, inputDepth: 3, outputSize: 512);
            
// ONNX inference mode with pre-trained model
var model = new InternVL<double>(architecture, "internvl.onnx");
            
// Training mode with native layers
var trainModel = new InternVL<double>(architecture, new InternVLOptions());

Constructors

Constructor Summary
InternVL(NeuralNetworkArchitecture<T> architecture, InternVLOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer)
InternVL(NeuralNetworkArchitecture<T> architecture, string modelPath, InternVLOptions options)

Properties

Property Summary
int DecoderEmbeddingDim { get; }
int EmbeddingDimension { get; }
int MaxGenerationLength { get; }
string LanguageModelName { get; }

Methods

Method Summary
ModelMetadata<T> GetModelMetadata()
Tensor<T> Chat(Tensor<T> image, IEnumerable<ValueTuple<string, string>> conversationHistory, string userMessage)
Tensor<T> EncodeImage(Tensor<T> image)
Tensor<T> GenerateFromImage(Tensor<T> image, string prompt) Generates text using InternVL’s progressive alignment architecture.