LlmProgramNoveltyJudge<T>

Models & Types · AiDotNet.Evolution.Programs.Novelty

Asks a language model whether a candidate is a meaningful change from the incumbent it most resembles.

Implements: IProgramNoveltyJudge

For Beginners

This is the expensive last resort of a novelty check. Two programs look nearly identical to the cheap checks, so a language model is shown both and asked whether the change is real. If it cannot answer, this says so rather than guessing, and your policy decides what to do.

How It Works

The prompt asks for the same distinction the reference implementation asks for — algorithmic, structural, functional, implementation and hyperparameter differences count; renames, formatting and comments do not — but the answer handling differs in three ways that matter.

First, the parse is correct. Upstream instructs the model to answer NOT_NOVEL and then searches the answer for the string "NOT NOVEL" with a space, which never matches; the underscored form contains NOVEL at offset four, so upstream reads its own canonical negative answer as a positive one and its judge can only ever reject when the model spontaneously uses a spacing the prompt never asked for. Here every spelling — underscore, space, or hyphen — is recognised, and the earliest verdict token in the answer wins so that a model which reasons first and concludes last is still read correctly.

Second, an unusable answer is reported as Unavailable instead of being silently converted to “novel”, so the calling policy decides whether an unreachable judge admits or discards. Third, both programs pass through PromptTextRedactor and a byte budget before they reach the request, so an evolved program that printed a credential into itself cannot carry it into a provider call, and a pathological candidate cannot inflate the request without bound.

Constructors

Constructor Summary
LlmProgramNoveltyJudge(IChatClient<T> chatClient, int maxProgramBytes, double? temperature, int? maxOutputTokens, string id) Initializes a language-model novelty judge.

Properties

Property Summary
double? Temperature { get; } Gets the sampling temperature used for judging requests, or null for the client’s default.
int MaxProgramBytes { get; } Gets the per-program byte budget applied before a request is sent.
int? MaxOutputTokens { get; } Gets the output token cap used for judging requests, or null for the client’s default.
long Judgements { get; } Gets how many judging requests were sent since this judge was constructed.
long UnavailableAnswers { get; } Gets how many requests produced no usable verdict.
string Id { get; }

Methods

Method Summary
ProgramNoveltyVerdict ParseVerdict(string answer) Reads a verdict out of a free-text model answer.
ValueTask<ProgramNoveltyVerdict> JudgeAsync(ProgramGenome candidate, ProgramGenome incumbent, CancellationToken cancellationToken)

Fields

Field Summary
int DefaultMaxProgramBytes The default byte budget applied to each program before it reaches the request.