Janus<T>

Models & Types · AiDotNet.VisionLanguage.Unified

Janus: decoupled visual encoding for understanding vs generation.

Inheritance: NeuralNetworkBase<T> → VisionLanguageModelBase<T> → Janus<T>

Implements: IUnifiedVisionModel<T>

For Beginners

Janus is a unified vision-language model from DeepSeek that uses separate visual paths for understanding and generation. Default values follow the original paper settings.

How It Works

Janus (DeepSeek, 2024) decouples visual encoding into separate pathways for understanding and generation. It uses a SigLIP encoder for visual understanding tasks and a VQ tokenizer for image generation, both feeding into a shared LLM backbone. This decoupling resolves the tension between high-level semantic features needed for understanding and low-level visual tokens needed for generation.

References:

  • Paper: “Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generation” (DeepSeek, 2024)

Example

// Create a Janus model for unified multimodal understanding and generation
// with decoupled visual encoding paths from DeepSeek
var architecture = new NeuralNetworkArchitecture<double>(
    inputType: InputType.TwoDimensional,
    taskType: NeuralNetworkTaskType.ImageClassification,
    inputHeight: 224, inputWidth: 224, inputDepth: 3, outputSize: 512);
            
// ONNX inference mode with pre-trained model
var model = new Janus<double>(architecture, "janus.onnx");
            
// Training mode with native layers
var trainModel = new Janus<double>(architecture, new JanusOptions());

Constructors

Constructor Summary
Janus(NeuralNetworkArchitecture<T> architecture, JanusOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer)
Janus(NeuralNetworkArchitecture<T> architecture, string modelPath, JanusOptions options)

Properties

Property Summary
bool SupportsGeneration { get; }
int DecoderEmbeddingDim { get; }
int EmbeddingDimension { get; }
int MaxGenerationLength { get; }

Methods

Method Summary
ModelMetadata<T> GetModelMetadata()
Tensor<T> EncodeImage(Tensor<T> image)
Tensor<T> GenerateFromImage(Tensor<T> image, string prompt) Generates text from image using Janus’s decoupled understanding encoder.
Tensor<T> GenerateImage(string textDescription) Generates an image from text using Janus’s decoupled generation encoder.