AdversarialPreferenceAlignment<T>

Models & Types ยท AiDotNet.AdversarialRobustness.Defenses

Implements adversarial preference alignment that combines RLHF with adversarial robustness, ensuring the model maintains alignment properties even under adversarial perturbation.

Implements: IAlignmentMethod<T>

For Beginners

RLHF teaches an AI to give good responses. But an attacker can craft tricky inputs that bypass this training. This module trains the AI to give good responses EVEN when the input has been tampered with โ€” like teaching a student to follow school rules even when other students try to distract or trick them.

How It Works

Standard RLHF alignment can be broken by adversarial inputs that shift model behavior away from aligned responses. This defense augments the RLHF training loop with adversarial examples, teaching the reward model and policy to maintain preference alignment even when inputs are adversarially perturbed. The result is a model that remains helpful, harmless, and honest even under attack.

References:

  • Adversarial RLHF: Adversarially robust alignment (2024)
  • Safety-Tuned LLaMAs: Lessons from improving safety of LLMs (2024)
  • Robustness of RLHF alignment under adversarial prompts (NAACL 2025)

Constructors

Constructor Summary
AdversarialPreferenceAlignment(AlignmentMethodOptions<T> options, double adversarialRatio, double perturbationBudget, int adversarialSteps) Initializes a new adversarial preference alignment module.

Methods

Method Summary
AlignmentMethodOptions<T> GetOptions()
AlignmentMetrics<T> EvaluateAlignment(IPredictiveModel<T, Vector<T>, Vector<T>> model, AlignmentEvaluationData<T> evaluationData)
IPredictiveModel<T, Vector<T>, Vector<T>> AlignModel(IPredictiveModel<T, Vector<T>, Vector<T>> baseModel, AlignmentFeedbackData<T> feedbackData)
IPredictiveModel<T, Vector<T>, Vector<T>> ApplyConstitutionalPrinciples(IPredictiveModel<T, Vector<T>, Vector<T>> model, string[] principles)
RedTeamingResults PerformRedTeaming(Func<string, string> model, SafetyPipeline<T> pipeline)
byte[] Serialize()
void Deserialize(byte[] data)
void LoadModel(string filePath)
void Reset()
void SaveModel(string filePath)