DeepSeekVL<T>

Models & Types · AiDotNet.VisionLanguage.InstructionTuned

DeepSeek-VL: real-world vision-language understanding with hybrid SigLIP + SAM-B encoder.

Inheritance: NeuralNetworkBase<T> → VisionLanguageModelBase<T> → DeepSeekVL<T>

Implements: IInstructionTunedVLM<T>

For Beginners

DeepSeek-VL uses two vision encoders working together instead of just one: SigLIP captures the semantic meaning of the image (what objects are present), while SAM-B captures fine-grained spatial details (exact shapes and boundaries). Their outputs are merged through an MLP projector and fed into the DeepSeek language model for text generation. This hybrid approach gives the model both high-level understanding and pixel-level precision for real-world tasks like reading documents, understanding charts, and answering detailed visual questions. Default values follow the original paper settings.

How It Works

DeepSeek-VL (Lu et al., 2024) uses a hybrid vision encoder combining SigLIP for semantic understanding and SAM-B for fine-grained spatial features. The dual encoder outputs are merged via MLP projection into the DeepSeek language model for real-world multimodal tasks.

References:

  • Paper: “DeepSeek-VL: Towards Real-World Vision-Language Understanding” (Lu et al., 2024)

Example

// Create a DeepSeek-VL model for real-world visual understanding
// using hybrid SigLIP + SAM-B dual vision encoder
var architecture = new NeuralNetworkArchitecture<double>(
    inputType: InputType.TwoDimensional,
    taskType: NeuralNetworkTaskType.ImageClassification,
    inputHeight: 224, inputWidth: 224, inputDepth: 3, outputSize: 512);
            
// ONNX inference mode with pre-trained model
var model = new DeepSeekVL<double>(architecture, "deepseekvl.onnx");
            
// Training mode with native layers
var trainModel = new DeepSeekVL<double>(architecture, new DeepSeekVLOptions());

Constructors

Constructor Summary
DeepSeekVL(NeuralNetworkArchitecture<T> architecture, DeepSeekVLOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer)
DeepSeekVL(NeuralNetworkArchitecture<T> architecture, string modelPath, DeepSeekVLOptions options)

Properties

Property Summary
int DecoderEmbeddingDim { get; }
int EmbeddingDimension { get; }
int MaxGenerationLength { get; }
string LanguageModelName { get; }

Methods

Method Summary
ModelMetadata<T> GetModelMetadata()
Tensor<T> Chat(Tensor<T> image, IEnumerable<ValueTuple<string, string>> conversationHistory, string userMessage)
Tensor<T> EncodeImage(Tensor<T> image)
Tensor<T> GenerateFromImage(Tensor<T> image, string prompt) Generates text using DeepSeek-VL’s hybrid vision encoder architecture.