AsyncOnlineDPO<T>
Models & Types · AiDotNet.Diffusion.Alignment
Asynchronous Online DPO for diffusion model alignment with on-policy generation.
For Beginners
Regular DPO learns from a fixed dataset of preferred/dispreferred image pairs collected beforehand. Async Online DPO is smarter — it continuously generates new images with the current model, gets feedback on them, and learns from that fresh feedback. It’s like a student who keeps practicing and getting real-time feedback rather than studying old examples.
How It Works
Async Online DPO extends standard DPO by generating new preference pairs during training rather than relying on a static dataset. It asynchronously generates image pairs from the current policy, obtains preference labels (from a reward model or human feedback), and uses these fresh pairs for DPO updates. This on-policy approach leads to better alignment than offline DPO trained on stale data.
Reference: Calandriello et al., “Human Alignment of Large Language Models through Online Preference Optimization”, 2024; adapted for diffusion models
Constructors
| Constructor | Summary |
|---|---|
AsyncOnlineDPO(IDiffusionModel<T> model, IDiffusionModel<T> referenceModel, double beta, double onlineMixingRatio, int bufferSize) |
Initializes a new Async Online DPO trainer. |
Properties
| Property | Summary |
|---|---|
double Beta { get; } |
Gets the temperature parameter. |
double OnlineMixingRatio { get; } |
Gets the ratio of online vs offline samples used in each batch. |
int TotalUpdates { get; } |
Gets the total number of DPO updates performed. |
Methods
| Method | Summary |
|---|---|
T ComputeOnlineDPOLoss(T preferredModelLogProb, T disPreferredModelLogProb, T preferredRefLogProb, T disPreferredRefLogProb, bool isOnlineSample) |
Computes the online DPO loss combining online and offline preference pairs. |
bool ShouldRefreshReference(int refreshInterval) |
Determines if the reference model should be refreshed based on update count. |
void RecordUpdate() |
Increments the update counter after a training step. |