FineTuningMetrics<T>

Models & Types · AiDotNet.Models.Options

Metrics for evaluating fine-tuning quality.

For Beginners

These metrics tell you how well the fine-tuning worked. Lower loss is generally better, higher win rates mean the model learned preferences well, and safety metrics ensure the model behaves appropriately.

How It Works

This class contains metrics relevant to various fine-tuning methods:

Constructors

Constructor Summary
FineTuningMetrics()

Properties

Property Summary
DateTime TrainingEndTime { get; set; } Gets or sets when the training completed.
DateTime TrainingStartTime { get; set; } Gets or sets when the training started.
Dictionary<string, double> CustomMetrics { get; set; } Gets or sets additional custom metrics.
List<double> LossHistory { get; set; } Gets or sets the loss history over training.
double AverageOutputLength { get; set; } Gets or sets the average output length.
double AverageReward { get; set; } Gets or sets the average reward achieved.
double BleuScore { get; set; } Gets or sets the BLEU score for generation quality.
double ChosenLogProb { get; set; } Gets or sets the chosen response log probability.
double DistillationLoss { get; set; } Gets or sets the distillation loss.
double FalseRefusalRate { get; set; } Gets or sets the false refusal rate.
double GroupAdvantage { get; set; } Gets or sets the average group advantage.
double GroupRewardVariance { get; set; } Gets or sets the within-group reward variance.
double HarmlessnessScore { get; set; } Gets or sets the harmlessness score.
double HelpfulnessScore { get; set; } Gets or sets the helpfulness score.
double HonestyScore { get; set; } Gets or sets the honesty score.
double KLDivergence { get; set; } Gets or sets the KL divergence from the reference model.
double LogProbMargin { get; set; } Gets or sets the margin between chosen and rejected log probabilities.
double PeakMemoryGB { get; set; } Gets or sets the peak memory usage in GB.
double Perplexity { get; set; } Gets or sets the perplexity on validation data.
double PolicyEntropy { get; set; } Gets or sets the entropy of the policy.
double PolicyLoss { get; set; } Gets or sets the policy loss.
double PreferenceAccuracy { get; set; } Gets or sets the preference accuracy on held-out data.
double RefusalRate { get; set; } Gets or sets the refusal rate for harmful prompts.
double RejectedLogProb { get; set; } Gets or sets the rejected response log probability.
double RewardStd { get; set; } Gets or sets the reward standard deviation.
double RougeLScore { get; set; } Gets or sets the ROUGE-L score.
double TeacherAgreementRate { get; set; } Gets or sets the agreement rate with teacher model.
double ThroughputSamplesPerSecond { get; set; } Gets or sets the throughput in samples per second.
double TrainingLoss { get; set; } Gets or sets the final training loss.
double TrainingTimeSeconds { get; set; } Gets or sets the total training time in seconds.
double ValidationLoss { get; set; } Gets or sets the validation loss.
double ValueLoss { get; set; } Gets or sets the value function loss (for actor-critic methods).
double WinRate { get; set; } Gets or sets the win rate against the reference model.
int TrainingSteps { get; set; } Gets or sets the number of training steps completed.
long TrainableParameters { get; set; } Gets or sets the number of trainable parameters.
string MethodName { get; set; } Gets or sets the fine-tuning method used.

Methods

Method Summary
string GetSummary() Gets a summary of the metrics suitable for logging.