FederatedRLHF<T>
Models & Types ยท AiDotNet.FederatedLearning.Alignment
Configuration and orchestration for Federated RLHF (Reinforcement Learning from Human Feedback).
Inheritance: FederatedLearningComponentBase<T> โ FederatedRLHF<T>
For Beginners
RLHF is how modern LLMs learn to be helpful, harmless, and honest. Normally, human feedback data is centralized. Federated RLHF keeps the feedback private at each organization: each client trains a local reward model on their preference data, and the server aggregates these reward models. The policy (LLM) is then fine-tuned using the aggregated reward signal via PPO or similar RL algorithms.
How It Works
Pipeline:
Reference: Federated RLHF for Privacy-Preserving LLM Alignment (2024).
Constructors
| Constructor | Summary |
|---|---|
FederatedRLHF(FederatedRLHFOptions options) |
Creates a new Federated RLHF orchestrator. |
Properties
| Property | Summary |
|---|---|
FederatedRLHFOptions Options { get; } |
Gets the RLHF configuration options. |
Methods
| Method | Summary |
|---|---|
Dictionary<string, T[]> AggregateRewardModels(Dictionary<int, Dictionary<string, T[]>> clientRewardModels, Dictionary<int, double> clientWeights) |
Aggregates reward model parameters from multiple clients. |
double ComputeKLPenalty(double[] policyLogProbs, double[] referenceLogProbs) |
Computes the KL penalty for PPO-style policy updates. |
double ComputePPOLoss(double[] logProbsNew, double[] logProbsOld, double[] advantages) |
Computes the PPO clipped surrogate loss for a batch of tokens. |
double[] ComputeGAE(double[] rewards, double[] values, double gamma, double lambda) |
Computes GAE (Generalized Advantage Estimation) for PPO training. |
double[] ComputeRewardsWithKLPenalty(double[] rewardModelScores, double[] policyLogProbs, double[] referenceLogProbs) |
Computes per-token rewards by combining reward model scores with KL penalty. |