DiffBIRModel<T>

Models & Types · AiDotNet.Diffusion.SuperResolution

DiffBIR model for blind image restoration with generative diffusion prior.

Inheritance: DiffusionModelBase<T> → LatentDiffusionModelBase<T> → DiffBIRModel<T>

For Beginners

DiffBIR restores degraded images (blur, noise, JPEG artifacts, old photos).

How DiffBIR works:

  1. Stage 1 (SwinIR): A regression network removes obvious degradation from the input
  2. Stage 2 (Diffusion): The SD prior adds realistic details to the cleaned image
  3. LAControlNet: Controls how much the diffusion stage can modify the Stage 1 output

Key characteristics:

  • Handles blind (unknown) degradation types automatically
  • Two-stage design separates degradation removal from detail generation
  • Controllable balance between fidelity to input and generated quality
  • Works for face restoration, general images, and old photo restoration
  • Based on Stable Diffusion 1.5 backbone for high-quality detail generation

When to use DiffBIR:

  • Restoring old or damaged photographs
  • Removing JPEG compression artifacts
  • Denoising and deblurring real-world images
  • Face restoration in group photos or surveillance footage

Limitations:

  • Two-stage pipeline is slower than single-stage methods
  • May hallucinate details not present in the original image
  • Best results at 512x512 (SD 1.5 native resolution)

How It Works

DiffBIR (Diffusion-Based Blind Image Restoration) uses a two-stage pipeline for real-world image restoration. The first stage removes degradation using a regression module (SwinIR), and the second stage refines details using a Stable Diffusion prior with a controllable module for balancing fidelity and quality.

Architecture components:

  • Stage 1: SwinIR regression module for degradation removal
  • Stage 2: SD 1.5 U-Net backbone (320 base channels, [1,2,4,4] multipliers)
  • Controllable feature wrapping module (LAControlNet) for fidelity-quality balance
  • Standard VAE (4-channel latent space, 0.18215 scale factor)
  • CLIP ViT-L/14 text encoder (768-dim cross-attention)

Technical specifications:

  • Architecture: Two-stage (SwinIR + SD 1.5 U-Net with LAControlNet)
  • Stage 1: SwinIR with 6 RSTB blocks, 180 channels, window size 8
  • Stage 2: SD 1.5 U-Net backbone, 320 base channels, [1,2,4,4] multipliers
  • Cross-attention dimension: 768 (CLIP ViT-L/14)
  • VAE: 4 latent channels, scale factor 0.18215, 8x downsampling
  • Noise schedule: Scaled linear beta [0.00085, 0.012], 1000 timesteps
  • Default guidance scale: 7.5
  • Default resolution: 512x512

Reference: Lin et al., “DiffBIR: Towards Blind Image Restoration with Generative Diffusion Prior”, ECCV 2024

Example

var degradedPhoto = Tensor<float>.CreateRandom(1, 3, 32, 32);
// Create with industry-standard defaults
var diffbir = new DiffBIRModel<float>();
            
// Restore a degraded image using image-to-image pipeline
var restored = diffbir.ImageToImage(
    inputImage: degradedPhoto,
    prompt: "high quality, sharp, detailed",
    negativePrompt: "blurry, noisy, artifacts",
    strength: 0.5,
    numInferenceSteps: 50,
    guidanceScale: 7.5);
            
// Generate from text (for unconditional restoration with prompt guidance)
var result = diffbir.GenerateFromText(
    prompt: "a sharp, detailed photograph",
    width: 512,
    height: 512,
    numInferenceSteps: 50);

Constructors

Constructor Summary
DiffBIRModel(NeuralNetworkArchitecture<T> architecture, DiffusionModelOptions<T> options, INoiseScheduler<T> scheduler, UNetNoisePredictor<T> unet, StandardVAE<T> vae, IConditioningModule<T> conditioner, int? seed) Initializes a new instance of DiffBIRModel with full customization support.

Properties

Property Summary
IConditioningModule<T> Conditioner { get; }
INoisePredictor<T> NoisePredictor { get; }
IVAEModel<T> VAE { get; }
int CrossAttentionDim { get; } Gets the cross-attention dimension (768, matching CLIP ViT-L/14).
int LatentChannels { get; }

Methods

Method Summary
ModelMetadata<T> GetModelMetadata()
Tensor<T> GenerateFromText(string prompt, string negativePrompt, int width, int height, int numInferenceSteps, double? guidanceScale, int? seed) Generates an image from a text prompt using DiffBIR defaults.
Tensor<T> ImageToImage(Tensor<T> inputImage, string prompt, string negativePrompt, double strength, int numInferenceSteps, double? guidanceScale, int? seed) Performs image restoration using the DiffBIR diffusion pipeline.

Fields

Field Summary
int DefaultHeight Default image height for DiffBIR restoration.
int DefaultWidth Default image width for DiffBIR restoration.