KandinskyModel<T>

Models & Types · AiDotNet.Diffusion.TextToImage

Kandinsky 2.2/3.0 model for text-to-image generation.

Inheritance: DiffusionModelBase<T> → LatentDiffusionModelBase<T> → KandinskyModel<T>

For Beginners

Kandinsky works in two phases, like a translator:

How Kandinsky works:

  1. Prior stage: Translates text embeddings (CLIP) into image embeddings
  2. Decoder stage: A latent diffusion model generates images conditioned on the image embeddings

Key characteristics:

  • Two-stage pipeline: Prior → Decoder
  • Prior: Diffusion-based mapping from text to image embedding space
  • Decoder: U-Net latent diffusion model (similar to SD architecture)
  • Text encoder: CLIP ViT-G/14 (1280-dim) + multilingual XLM-RoBERTa
  • Image encoder: CLIP ViT-G/14 (used to train the prior)
  • Native resolution: 1024x1024 (Kandinsky 3.0) or 512x512 (Kandinsky 2.2)
  • VAE: Movq (MoVQ-GAN), 4 latent channels

Advantages:

  • Strong multilingual support through XLM-RoBERTa
  • Two-stage architecture allows separate optimization of text→embedding and embedding→image
  • Good prompt adherence through CLIP-space prior

Limitations:

  • Two-stage pipeline adds latency
  • Smaller ecosystem than Stable Diffusion

How It Works

Kandinsky is a two-stage text-to-image model developed by Sber AI and AI Forever. It uses a prior model (diffusion or transformer) to map text embeddings to image embeddings, then a latent diffusion decoder to generate images from those embeddings.

Technical specifications:

  • Architecture: Prior + Latent Diffusion Decoder
  • Prior: Diffusion transformer, maps CLIP text→image embeddings
  • Decoder U-Net: ~1.2B parameters, 4-channel latent, channel multipliers [1, 2, 4, 4]
  • Text encoder: CLIP ViT-G/14 (1280-dim)
  • VAE: MoVQ-GAN, 4 latent channels, scale factor 0.18215
  • Noise schedule: Linear beta schedule, 1000 training timesteps

Reference: Razzhigaev et al., “Kandinsky: an Improved Text-to-Image Synthesis with Image Prior and Latent Diffusion”, 2023

Example

// Create with industry-standard defaults
var kandinsky = new KandinskyModel<float>();
            
// Generate a 1024x1024 image from text
var image = kandinsky.GenerateFromText(
    prompt: "An oil painting of a Russian winter landscape",
    negativePrompt: "blurry, deformed",
    width: 1024,
    height: 1024,
    numInferenceSteps: 50,
    guidanceScale: 4.0,
    seed: 42);

Constructors

Constructor Summary
KandinskyModel(NeuralNetworkArchitecture<T> architecture, DiffusionModelOptions<T> options, INoiseScheduler<T> scheduler, UNetNoisePredictor<T> priorUnet, UNetNoisePredictor<T> decoderUnet, StandardVAE<T> vae, IConditioningModule<T> conditioner, KandinskyVersion version, int? seed) Initializes a new instance of KandinskyModel with full customization support.

Properties

Property Summary
IConditioningModule<T> Conditioner { get; }
INoisePredictor<T> NoisePredictor { get; }
INoisePredictor<T> PriorModel { get; } Gets the prior model that maps text embeddings to image embeddings.
IVAEModel<T> VAE { get; }
KandinskyVersion Version { get; } Gets the model version.
int CrossAttentionDim { get; } Gets the cross-attention dimension (1280 for CLIP ViT-G/14).
int LatentChannels { get; }

Methods

Method Summary
ModelMetadata<T> GetModelMetadata()
Tensor<T> GenerateFromText(string prompt, string negativePrompt, int width, int height, int numInferenceSteps, double? guidanceScale, int? seed)
Tensor<T> ImageToImage(Tensor<T> inputImage, string prompt, string negativePrompt, double strength, int numInferenceSteps, double? guidanceScale, int? seed)

Fields

Field Summary
int DefaultHeight Default image height for Kandinsky 3.0 generation.
int DefaultWidth Default image width for Kandinsky 3.0 generation.