LlmFeedbackOptions

Options & Configuration · AiDotNet.Configuration

Configures how a language model’s opinion of a program is blended into that program’s fitness.

For Beginners

This controls a second opinion on each candidate program. Your tests say whether it works; the model says whether it is any good as code. The two are mixed, with the tests carrying most of the weight, so a program that fails its tests cannot be talked into a high score. List the qualities you care about in Criteria — the defaults are correctness, efficiency, and readability — and leave the rest alone until you have a reason to change it.

How It Works

A judge scores qualities an automated test cannot: whether an approach is readable, whether it generalises, whether it is doing something clever or merely memorising the examples. Those scores are never allowed to become the fitness on their own. CombinedBlend is the share of the final quality that stays with the measured evaluator, so the default of 0.7 means a judge can move a candidate by at most three tenths of the scale no matter how enthusiastic it is.

Three defaults differ deliberately from the reference implementation. Upstream hard-codes the 0.7/0.3 blend in the evaluator body, applies its weights when averaging the individual criteria but not when blending, and runs the judge even on candidates whose own evaluation failed, so a program that crashed can still be praised into a respectable score. Here the blend is a configured number, Weight is applied consistently, and RunOnFailedEvaluations defaults to false.

Nothing here contacts a model by itself. The options take effect only once a chat client is supplied to the judge evaluator, and an answer that cannot be parsed leaves the measured quality untouched rather than failing the candidate, so an unreliable judge degrades the search’s extra signal rather than its correctness.

Constructors

Constructor Summary
LlmFeedbackOptions()

Properties

Property Summary
IList<string> Criteria { get; set; } Gets or sets the qualities the judge is asked to score, one per prompt line.
bool CarryCritiqueForward { get; set; } Gets or sets whether the judge’s written criticism is carried forward to the next proposal.
bool Enabled { get; set; } Gets or sets whether the judge runs at all.
bool JudgeWithEveryEnsembleMember { get; set; } Gets or sets whether every ensemble member judges and their scores are averaged by weight.
bool RecordObjectives { get; set; } Gets or sets whether judge scores are appended to the result’s objectives as well as its descriptors.
bool RequestJsonResponseFormat { get; set; } Gets or sets whether the request asks the provider to constrain the answer to JSON.
bool RunOnFailedEvaluations { get; set; } Gets or sets whether the judge also runs on candidates whose own evaluation did not complete.
double CombinedBlend { get; set; } Gets or sets the share of the final quality that stays with the measured evaluator.
double Weight { get; set; } Gets or sets the multiplier applied to every judge score before they are averaged.
double? Temperature { get; set; } Gets or sets the sampling temperature for judge requests, or null for the client’s default.
int MaxCritiqueChars { get; set; } Gets or sets the largest critique carried forward, in characters.
int MaxJudgeRetries { get; set; } Gets or sets how many times an unparseable judge answer is requested again.
int? MaxOutputTokens { get; set; } Gets or sets the output token cap for judge requests, or null for the client’s default.
string CritiqueField { get; set; } Gets or sets the JSON field the judge’s written criticism is read from.
string MetricPrefix { get; set; } Gets or sets the prefix given to each judge metric recorded on the result.
string ResponseSchema { get; set; } Gets or sets the JSON shape the judge’s answer must take, or null to derive one from the criteria.

Methods

Method Summary
LlmFeedbackOptions Clone() Creates an independent copy so a running evaluator is unaffected by later mutation.
void Validate() Validates the criteria, the blend, and the request bounds.

Fields

Field Summary
string AverageMetricSuffix The metric name holding the mean of the individual judge scores.
string CritiqueArtifactKey The artifact key the carried-forward critique is attached under.
string DefaultCritiqueField The JSON field the judge’s written criticism is read from unless another is configured.
string DefaultMetricPrefix The prefix given to every judge-produced metric name.