LuminaT2XModel<T>

Models & Types · AiDotNet.Diffusion.TextToImage

Lumina-T2X unified framework for transforming text into any modality.

Inheritance: DiffusionModelBase<T> → LatentDiffusionModelBase<T> → LuminaT2XModel<T>

For Beginners

Lumina-T2X is a versatile model that can generate not just images, but potentially video, audio, and 3D content too — all from text descriptions.

How Lumina-T2X works:

  1. Text is encoded by Gemma 2B for broad language understanding
  2. A shared Flag-DiT backbone processes the conditioning
  3. Modality-specific heads produce output for the target format
  4. Flow matching enables flexible resolution and duration handling

Key characteristics:

  • Unified backbone for multi-modality generation
  • Flag-DiT architecture with flow matching
  • Gemma 2B text encoder
  • Modality-specific output heads (image, video, audio, 3D)
  • Resolution-agnostic and duration-agnostic design

Advantages:

  • Single model backbone for multiple modalities
  • Open-source and well-documented
  • Flexible resolution and aspect ratio support
  • Clean modular design for extension

How It Works

Lumina-T2X is a unified framework for text-to-any generation using Flag-DiT (Flow-Aware Generative DiT) blocks. It supports generating images, videos, 3D content, and audio from text prompts using a shared transformer backbone with modality-specific output heads. This implementation focuses on the image generation modality.

Reference: Gao et al., “Lumina-T2X: Transforming Text into Any Modality, Resolution, and Duration via Flow-based Large Diffusion Transformers”, 2024

Example

var options = new DiffusionModelOptions<float> { LatentChannels = 4, DefaultInferenceSteps = 30 };
            
var trainX = Tensor<float>.CreateRandom(2, 4, 32, 32);
var trainY = Tensor<float>.CreateRandom(2, 4, 32, 32);
            
var result = new AiModelBuilder<float, Tensor<float>, Tensor<float>>()
    .ConfigureModel(new LuminaT2XModel<float>(options: options))
    .Build(trainX, trainY);
            
var noise = Tensor<float>.CreateRandom(new[] { 1, 4, 128, 128 });
var generated = result.Predict(noise);

Constructors

Constructor Summary
LuminaT2XModel(NeuralNetworkArchitecture<T> architecture, DiffusionModelOptions<T> options, INoiseScheduler<T> scheduler, FlagDiTPredictor<T> predictor, StandardVAE<T> vae, IConditioningModule<T> conditioner, int? seed)

Properties

Property Summary
IConditioningModule<T> Conditioner { get; }
INoisePredictor<T> NoisePredictor { get; }
IVAEModel<T> VAE { get; }
bool SupportsMultiModality { get; } Gets whether this model supports multi-modality output.
int LatentChannels { get; }

Methods

Method Summary
ModelMetadata<T> GetModelMetadata()
Tensor<T> GenerateFromText(string prompt, string negativePrompt, int width, int height, int numInferenceSteps, double? guidanceScale, int? seed)
Tensor<T> ImageToImage(Tensor<T> inputImage, string prompt, string negativePrompt, double strength, int numInferenceSteps, double? guidanceScale, int? seed)

Fields

Field Summary
int DefaultHeight
int DefaultWidth