CogView4Model<T>

Models & Types · AiDotNet.Diffusion.TextToImage

CogView-4 model for bilingual text-to-image generation by Zhipu AI.

Inheritance: DiffusionModelBase<T> → LatentDiffusionModelBase<T> → CogView4Model<T>

For Beginners

CogView-4 is a model from Zhipu AI (Tsinghua University spinoff) that understands both Chinese and English prompts equally well.

How CogView-4 works:

  1. Text is encoded by GLM (General Language Model) with bilingual support
  2. A SiT (Scalable Interpolant Transformer) processes text and noise tokens
  3. Relay diffusion enables efficient high-resolution generation in two stages
  4. A 16-channel VAE decodes latents to full images

Key characteristics:

  • SiT architecture with scalable interpolant formulation
  • GLM text encoder for bilingual Chinese-English understanding
  • Relay diffusion: low-res generation → high-res refinement
  • 16 latent channels
  • Strong understanding of Asian cultural contexts
  • 30 inference steps typical

Advantages:

  • Best-in-class bilingual prompt understanding
  • Strong cultural context for Asian aesthetics
  • Relay diffusion enables efficient high-res generation
  • Open-weight model

Limitations:

  • Smaller English-only community
  • Less LoRA/adapter ecosystem
  • Relay diffusion adds complexity

How It Works

CogView-4 uses a Scalable Interpolant Transformer (SiT) architecture with relay diffusion for high-resolution image generation. It features bilingual Chinese-English text understanding through a GLM-based text encoder, providing strong comprehension of both languages and Asian cultural contexts.

Technical specifications:

  • Architecture: SiT with relay diffusion
  • Text encoder: GLM (bilingual, 4096-dim)
  • Hidden size: 2048, 24 layers, 32 heads
  • Latent channels: 16
  • Training: Continuous-time diffusion with interpolant formulation
  • Default: 30 steps, guidance scale 7.5
  • Resolution: 1024x1024 default

Reference: Zheng et al., “CogView-4: Bilingual Text-to-Image Generation with Relay Diffusion”, 2024

Example

// Create CogView-4
var cogView = new CogView4Model<float>();
            
// Generate with Chinese prompt
var image = cogView.GenerateFromText(
    prompt: "A traditional Chinese ink painting of mountains in mist",
    width: 1024,
    height: 1024,
    numInferenceSteps: 30,
    guidanceScale: 7.5,
    seed: 42);

Constructors

Constructor Summary
CogView4Model(NeuralNetworkArchitecture<T> architecture, DiffusionModelOptions<T> options, INoiseScheduler<T> scheduler, SiTPredictor<T> predictor, StandardVAE<T> vae, IConditioningModule<T> conditioner, int? seed) Initializes a new instance of CogView4Model.

Properties

Property Summary
IConditioningModule<T> Conditioner { get; }
INoisePredictor<T> NoisePredictor { get; }
IVAEModel<T> VAE { get; }
bool SupportsBilingual { get; } Gets whether this model supports bilingual Chinese-English prompts.
int LatentChannels { get; }

Methods

Method Summary
ModelMetadata<T> GetModelMetadata()
Tensor<T> GenerateFromText(string prompt, string negativePrompt, int width, int height, int numInferenceSteps, double? guidanceScale, int? seed)
Tensor<T> ImageToImage(Tensor<T> inputImage, string prompt, string negativePrompt, double strength, int numInferenceSteps, double? guidanceScale, int? seed)

Fields

Field Summary
int DefaultHeight Default image height for CogView-4 (1024x1024).
int DefaultWidth Default image width for CogView-4 (1024x1024).