InternVL2<T>

Models & Types · AiDotNet.VisionLanguage.InstructionTuned

InternVL2: dynamic resolution with pixel shuffle and InternLM2 backbone.

Inheritance: NeuralNetworkBase<T> → VisionLanguageModelBase<T> → InternVL2<T>

Implements: IInstructionTunedVLM<T>

For Beginners

InternVL2 improves on InternVL by adding dynamic resolution support — it can process images at different sizes by splitting them into tiles and using a “pixel shuffle” technique to efficiently downsample the visual features. This means it handles both small icons and large high-resolution images well. It switches the language backbone to InternLM2 and achieves performance competitive with commercial models like GPT-4V on many benchmarks, making it one of the strongest open-source multimodal models available. Default values follow the original paper settings.

How It Works

InternVL2 (Chen et al., 2024) introduces dynamic resolution support through pixel shuffle downsampling and dynamic image tiling. Combined with InternLM2 as the language backbone, it achieves competitive performance with commercial models across diverse benchmarks.

References:

  • Paper: “InternVL2: Better than the Best - Expanding Performance Boundaries of Open-Source Multimodal Models” (Chen et al., 2024)

Example

// Create an InternVL2 model for dynamic resolution visual understanding
// using pixel shuffle downsampling with InternLM2 backbone
var architecture = new NeuralNetworkArchitecture<double>(
    inputType: InputType.TwoDimensional,
    taskType: NeuralNetworkTaskType.ImageClassification,
    inputHeight: 224, inputWidth: 224, inputDepth: 3, outputSize: 512);
            
// ONNX inference mode with pre-trained model
var model = new InternVL2<double>(architecture, "internvl2.onnx");
            
// Training mode with native layers
var trainModel = new InternVL2<double>(architecture, new InternVL2Options());

Constructors

Constructor Summary
InternVL2(NeuralNetworkArchitecture<T> architecture, InternVL2Options options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer)
InternVL2(NeuralNetworkArchitecture<T> architecture, string modelPath, InternVL2Options options)

Properties

Property Summary
int DecoderEmbeddingDim { get; }
int EmbeddingDimension { get; }
int MaxGenerationLength { get; }
string LanguageModelName { get; }

Methods

Method Summary
ModelMetadata<T> GetModelMetadata()
Tensor<T> Chat(Tensor<T> image, IEnumerable<ValueTuple<string, string>> conversationHistory, string userMessage)
Tensor<T> EncodeImage(Tensor<T> image)
Tensor<T> GenerateFromImage(Tensor<T> image, string prompt) Generates text using InternVL2’s pixel shuffle + dynamic resolution architecture.