JanusPro<T>

Models & Types · AiDotNet.VisionLanguage.Unified

Janus-Pro: unified multimodal understanding and generation with decoupled vision encoders (Chen et al., DeepSeek 2025, arXiv:2501.17811).

Inheritance: NeuralNetworkBase<T> → VisionLanguageModelBase<T> → JanusPro<T>

Implements: IUnifiedVisionModel<T>

For Beginners

Janus-Pro is the first model that does BOTH “understand image, answer in text” AND “describe in text, generate image” with one unified backbone — but it uses two completely different vision encoders for each direction, which the paper shows is much better than trying to share. Default values follow the published 1.5B configuration (scale up via JanusProOptions for the 7B variant).

How It Works

Janus-Pro is the scaled-up successor to Janus (Wu et al. 2024, arXiv:2410.13848). Both models share Janus’s central design insight: vision encoding for understanding (an image-to-language path that feeds SigLIP-style continuous features into the LLM) and vision encoding for generation (a VQ-VAE codebook that turns the LLM’s output token stream back into pixels) are fully decoupled. The two paths converge only at the autoregressive transformer backbone in the middle. Janus-Pro adds: a 16384-entry VQ codebook (vs Janus’s 8192), curriculum-based training, expanded synthetic data, and a 7B-parameter DeepSeek-LLM backbone.

Paper-faithful pieces implemented here:

What is NOT verified in-session:

Example

var image = Tensor<double>.CreateRandom(1, 3, 32, 32);
var arch = new NeuralNetworkArchitecture<double>(
    inputType: InputType.TwoDimensional,
    taskType: NeuralNetworkTaskType.ImageClassification,
    inputHeight: 384, inputWidth: 384, inputDepth: 3, outputSize: 4096);
var model = new JanusPro<double>(arch, new JanusProOptions());
            
// Understanding path
var hidden = model.GenerateFromImage(image, "what do you see?");
            
// Generation path
var generated = model.GenerateImage("a red apple on a wooden table");

Constructors

Constructor Summary
JanusPro(NeuralNetworkArchitecture<T> architecture, JanusProOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer)
JanusPro(NeuralNetworkArchitecture<T> architecture, string modelPath, JanusProOptions options)

Properties

Property Summary
bool SupportsGeneration { get; }
double CfgScale { get; } Classifier-free guidance scale used during autoregressive image generation.
int DecoderEmbeddingDim { get; }
int EmbeddingDimension { get; }
int GenerationTokenCount { get; } Number of generation-side VQ tokens in the output grid (24×24 for 384px output, matching paper §3.3).
int MaxGenerationLength { get; }

Methods

Method Summary
ModelMetadata<T> GetModelMetadata()
Tensor<T> EncodeImage(Tensor<T> image) Janus-Pro understanding path: image → SigLIP-style continuous features → LLM hidden state.
Tensor<T> GenerateFromImage(Tensor<T> image, string prompt) Image-to-text (understanding) forward pass.
Tensor<T> GenerateImage(string textDescription) Text-to-image (generation) forward pass.