D3PO<T>
Models & Types · AiDotNet.Diffusion.Alignment
Direct Preference for Denoising Diffusion Policy Optimization (D3PO).
For Beginners
Regular DPO teaches a model by comparing two finished images. D3PO goes deeper — it compares images at every step of the generation process. This is like having a coach who gives feedback on every brush stroke rather than just rating the final painting, leading to more precise control over image quality.
How It Works
D3PO extends DPO to operate directly on the denoising process of diffusion models. Rather than comparing final generated images, it applies preference optimization at each denoising step, treating the denoising trajectory as a Markov Decision Process. This enables finer-grained alignment by optimizing the generation policy at the step level.
Reference: Yang et al., “Using Human Feedback to Fine-tune Diffusion Models without Any Reward Model”, CVPR 2024
Constructors
| Constructor | Summary |
|---|---|
D3PO(IDiffusionModel<T> model, IDiffusionModel<T> referenceModel, double beta, double stepWeight) |
Initializes a new D3PO trainer. |
Properties
| Property | Summary |
|---|---|
double Beta { get; } |
Gets the temperature parameter beta. |
double StepWeight { get; } |
Gets the per-step weighting factor. |
Methods
| Method | Summary |
|---|---|
T ComputeStepLoss(Vector<T> preferredNoisePred, Vector<T> disPreferredNoisePred, Vector<T> refPreferredNoisePred, Vector<T> refDisPreferredNoisePred) |
Computes the D3PO per-step loss at a single denoising timestep. |