ScriptProgramFitnessEvaluator

Models & Types · AiDotNet.Evolution.Programs

Scores a candidate by running a caller-supplied evaluator script inside the same sandbox, under the same limits, and reading the JSON metrics it prints.

Implements: IProgramFitnessEvaluator

For Beginners

Sometimes the question “is this program good?” needs judgement that a table of expected outputs cannot express. This class lets you write that judgement as a small script: it receives the candidate’s code on standard input and prints a score as JSON. Your script runs in the same locked-down sandbox as the programs being evolved, so a mistake in your scoring script cannot hang or harm the run either — it just gets reported as a failed evaluation.

How It Works

The contract is deliberately small. The evaluator script runs as the sandboxed program; the candidate’s source arrives on its standard input; and it prints exactly one JSON object to standard output: {"quality": 0.83, "descriptors": {"length": 0.4}, "objectives": [0.83, 0.1], "artifacts": {"note": "…"}}. Only quality is required and it must be a finite number. Descriptors and objectives must be finite too; artifacts are free text that becomes bounded, redacted diagnostics so a chatty evaluator cannot bloat a checkpoint. Anything else in the object is ignored, so an evaluator may print extra fields for its own use.

Two things here are stricter than the reference implementation. First, the evaluator script is sandboxed exactly like the candidates: it goes through the same IProgramExecutionEngine, so an evaluator with an infinite loop is killed by the same wall-clock limit instead of hanging a worker, and one that prints without stopping is truncated by the same output cap. OpenEvolve writes its evaluator to a file and executes it unsandboxed in its worker processes, which means a bad evaluator is a worse problem than a bad candidate. Second, the metrics are schema-checked: a non-finite quality, a descriptor that is not a number, or output that is not a JSON object produces a Failed result with a diagnostic naming the problem, never an exception and never a silently-zero score that would look like a legitimately bad program.

Because the sandbox deliberately gives the script a pinned path and a scrubbed environment, an evaluator script should judge the candidate’s text — its structure, its style, a domain rule. Scoring a candidate by running it belongs in SandboxedProgramFitnessEvaluator, which executes the candidate itself.

Constructors

Constructor Summary
ScriptProgramFitnessEvaluator(IProgramExecutionEngine engine, string evaluatorScript, ScriptProgramEvaluationOptions options, string id, ProgramMetricAggregator metricAggregator) Initializes an evaluator that runs evaluatorScript against every candidate.

Properties

Property Summary
ProgramLanguage EvaluatorScriptLanguage { get; } Gets the language the evaluator script is executed as.
string Id { get; }
string VersionHash { get; }

Methods

Method Summary
ScriptProgramEvaluationOptions GetOptions() Gets an independent copy of the options this evaluator was constructed with.
ValueTask<EvolutionTaskResult> EvaluateAsync(ProgramGenome candidate, EvolutionEvaluationContext context, CancellationToken cancellationToken)