DiffusionRLHF<T>

Models & Types · AiDotNet.Diffusion.Alignment

Reinforcement Learning from Human Feedback (RLHF) adapted for diffusion models.

For Beginners

RLHF is a two-step process: first, train a “reward model” that learns what humans prefer (like a judge). Then use that judge to give feedback to the diffusion model during training. The diffusion model learns to generate images the judge would rate highly. KL regularization prevents the model from “cheating” by exploiting the judge’s blind spots.

How It Works

Diffusion-RLHF uses a trained reward model to provide feedback signals that guide the diffusion model toward generating outputs aligned with human preferences. The reward model is trained on human preference data, then used to fine-tune the diffusion model via policy gradient methods with KL regularization against a reference model.

Reference: Black et al., “Training Diffusion Models with Reinforcement Learning”, 2024

Constructors

Constructor Summary
DiffusionRLHF(IDiffusionModel<T> model, IDiffusionModel<T> referenceModel, double klWeight, double rewardScale, double clipRange) Initializes a new Diffusion-RLHF trainer.

Properties

Property Summary
IDiffusionModel<T> Model { get; } Gets the model being aligned.
IDiffusionModel<T> ReferenceModel { get; } Gets the frozen reference model.
double ClipRange { get; } Gets the PPO clipping range.
double KLWeight { get; } Gets the KL divergence weight.
double RewardScale { get; } Gets the reward scaling factor.

Methods

Method Summary
T ComputePPOLoss(T advantage, T ratio) Computes the PPO-clipped policy gradient loss for diffusion denoising steps.
T ComputeRLHFObjective(T reward, T modelLogProb, T refLogProb) Computes the RLHF objective: reward minus KL penalty.