FederatedRLHF<T>

Models & Types ยท AiDotNet.FederatedLearning.Alignment

Configuration and orchestration for Federated RLHF (Reinforcement Learning from Human Feedback).

Inheritance: FederatedLearningComponentBase<T> โ†’ FederatedRLHF<T>

For Beginners

RLHF is how modern LLMs learn to be helpful, harmless, and honest. Normally, human feedback data is centralized. Federated RLHF keeps the feedback private at each organization: each client trains a local reward model on their preference data, and the server aggregates these reward models. The policy (LLM) is then fine-tuned using the aggregated reward signal via PPO or similar RL algorithms.

How It Works

Pipeline:

Reference: Federated RLHF for Privacy-Preserving LLM Alignment (2024).

Constructors

Constructor Summary
FederatedRLHF(FederatedRLHFOptions options) Creates a new Federated RLHF orchestrator.

Properties

Property Summary
FederatedRLHFOptions Options { get; } Gets the RLHF configuration options.

Methods

Method Summary
Dictionary<string, T[]> AggregateRewardModels(Dictionary<int, Dictionary<string, T[]>> clientRewardModels, Dictionary<int, double> clientWeights) Aggregates reward model parameters from multiple clients.
double ComputeKLPenalty(double[] policyLogProbs, double[] referenceLogProbs) Computes the KL penalty for PPO-style policy updates.
double ComputePPOLoss(double[] logProbsNew, double[] logProbsOld, double[] advantages) Computes the PPO clipped surrogate loss for a batch of tokens.
double[] ComputeGAE(double[] rewards, double[] values, double gamma, double lambda) Computes GAE (Generalized Advantage Estimation) for PPO training.
double[] ComputeRewardsWithKLPenalty(double[] rewardModelScores, double[] policyLogProbs, double[] referenceLogProbs) Computes per-token rewards by combining reward model scores with KL penalty.