LuminaT2XModel<T>
Models & Types · AiDotNet.Diffusion.TextToImage
Lumina-T2X unified framework for transforming text into any modality.
Inheritance: DiffusionModelBase<T> → LatentDiffusionModelBase<T> → LuminaT2XModel<T>
For Beginners
Lumina-T2X is a versatile model that can generate not just images, but potentially video, audio, and 3D content too — all from text descriptions.
How Lumina-T2X works:
- Text is encoded by Gemma 2B for broad language understanding
- A shared Flag-DiT backbone processes the conditioning
- Modality-specific heads produce output for the target format
- Flow matching enables flexible resolution and duration handling
Key characteristics:
- Unified backbone for multi-modality generation
- Flag-DiT architecture with flow matching
- Gemma 2B text encoder
- Modality-specific output heads (image, video, audio, 3D)
- Resolution-agnostic and duration-agnostic design
Advantages:
- Single model backbone for multiple modalities
- Open-source and well-documented
- Flexible resolution and aspect ratio support
- Clean modular design for extension
How It Works
Lumina-T2X is a unified framework for text-to-any generation using Flag-DiT (Flow-Aware Generative DiT) blocks. It supports generating images, videos, 3D content, and audio from text prompts using a shared transformer backbone with modality-specific output heads. This implementation focuses on the image generation modality.
Reference: Gao et al., “Lumina-T2X: Transforming Text into Any Modality, Resolution, and Duration via Flow-based Large Diffusion Transformers”, 2024
Example
var options = new DiffusionModelOptions<float> { LatentChannels = 4, DefaultInferenceSteps = 30 };
var trainX = Tensor<float>.CreateRandom(2, 4, 32, 32);
var trainY = Tensor<float>.CreateRandom(2, 4, 32, 32);
var result = new AiModelBuilder<float, Tensor<float>, Tensor<float>>()
.ConfigureModel(new LuminaT2XModel<float>(options: options))
.Build(trainX, trainY);
var noise = Tensor<float>.CreateRandom(new[] { 1, 4, 128, 128 });
var generated = result.Predict(noise);
Constructors
| Constructor | Summary |
|---|---|
LuminaT2XModel(NeuralNetworkArchitecture<T> architecture, DiffusionModelOptions<T> options, INoiseScheduler<T> scheduler, FlagDiTPredictor<T> predictor, StandardVAE<T> vae, IConditioningModule<T> conditioner, int? seed) |
Properties
| Property | Summary |
|---|---|
IConditioningModule<T> Conditioner { get; } |
|
INoisePredictor<T> NoisePredictor { get; } |
|
IVAEModel<T> VAE { get; } |
|
bool SupportsMultiModality { get; } |
Gets whether this model supports multi-modality output. |
int LatentChannels { get; } |
Methods
| Method | Summary |
|---|---|
ModelMetadata<T> GetModelMetadata() |
|
Tensor<T> GenerateFromText(string prompt, string negativePrompt, int width, int height, int numInferenceSteps, double? guidanceScale, int? seed) |
|
Tensor<T> ImageToImage(Tensor<T> inputImage, string prompt, string negativePrompt, double strength, int numInferenceSteps, double? guidanceScale, int? seed) |
Fields
| Field | Summary |
|---|---|
int DefaultHeight |
|
int DefaultWidth |