LlmJudgeProgramFitnessEvaluator<T>

Models & Types · AiDotNet.Evolution.Programs

Blends a language model’s judgement of a program into the score a measured evaluator produced.

Implements: IProgramFitnessEvaluator

For Beginners

Tests tell you whether a program works, but not whether it is well written or likely to keep working on inputs you did not think of. This asks a language model those questions and folds its answer into the score, with the tests keeping most of the weight so a broken program cannot be flattered into winning. If the model answers with something unreadable, the score simply stays as your tests measured it.

How It Works

The evaluator runs the inner IProgramFitnessEvaluator first, then asks the model to score the same program against Criteria and mixes the two. The measured result keeps CombinedBlend of the final quality, so a judge cannot overturn a failing test suite. Every individual judge score is recorded as its own descriptor under MetricPrefix, together with their mean, so a run can be analysed afterwards to see whether the judge and the tests agreed.

Four things make this strictly better than the reference OpenEvolve feedback path. The blend is a configured number rather than a hard-coded 0.7/0.3 in the evaluator body. The configured weight is applied consistently, where upstream weights the criteria average and then blends the unweighted one, so its weight silently does nothing. A candidate whose own evaluation failed is not judged at all by default, where upstream judges error results and lets a crashed program earn a respectable combined score. And the request asks the provider to constrain its answer to JSON, so on a provider that supports constrained decoding an unparseable judge answer is impossible rather than merely unlikely; where it is still unparseable, the measured quality passes through untouched and a bounded diagnostic records why.

Determinism is preserved: the judge’s sampling seed is derived from the evaluation’s own random stream, so a replayed run asks the judge identically. Security follows the untrusted-content rule — the program reaches the prompt only through the bounding and redaction the prompt builder applies, and a provider exception contributes only its type name to a diagnostic, never its message.

Constructors

Constructor Summary
LlmJudgeProgramFitnessEvaluator(IChatClient<T> chatClient, IProgramFitnessEvaluator inner, ProgramPromptBuilder promptBuilder, LlmFeedbackOptions options, string id) Initializes a judging evaluator around a measured one.

Properties

Property Summary
IProgramFitnessEvaluator Inner { get; } Gets the measured evaluator whose score this judge adjusts.
long JudgeCalls { get; } Gets how many judging requests were sent since this evaluator was constructed.
long JudgeFailures { get; } Gets how many candidates ended with no usable judge answer and kept their measured score.
string Id { get; }
string VersionHash { get; }

Methods

Method Summary
LlmFeedbackOptions GetOptions() Gets a copy of the judging settings this evaluator uses.
ValueTask<EvolutionTaskResult> EvaluateAsync(ProgramGenome candidate, EvolutionEvaluationContext context, CancellationToken cancellationToken)