D3PO<T>

Models & Types · AiDotNet.Diffusion.Alignment

Direct Preference for Denoising Diffusion Policy Optimization (D3PO).

For Beginners

Regular DPO teaches a model by comparing two finished images. D3PO goes deeper — it compares images at every step of the generation process. This is like having a coach who gives feedback on every brush stroke rather than just rating the final painting, leading to more precise control over image quality.

How It Works

D3PO extends DPO to operate directly on the denoising process of diffusion models. Rather than comparing final generated images, it applies preference optimization at each denoising step, treating the denoising trajectory as a Markov Decision Process. This enables finer-grained alignment by optimizing the generation policy at the step level.

Reference: Yang et al., “Using Human Feedback to Fine-tune Diffusion Models without Any Reward Model”, CVPR 2024

Constructors

Constructor Summary
D3PO(IDiffusionModel<T> model, IDiffusionModel<T> referenceModel, double beta, double stepWeight) Initializes a new D3PO trainer.

Properties

Property Summary
double Beta { get; } Gets the temperature parameter beta.
double StepWeight { get; } Gets the per-step weighting factor.

Methods

Method Summary
T ComputeStepLoss(Vector<T> preferredNoisePred, Vector<T> disPreferredNoisePred, Vector<T> refPreferredNoisePred, Vector<T> refDisPreferredNoisePred) Computes the D3PO per-step loss at a single denoising timestep.