ScriptProgramFitnessEvaluator
Models & Types · AiDotNet.Evolution.Programs
Scores a candidate by running a caller-supplied evaluator script inside the same sandbox, under the same limits, and reading the JSON metrics it prints.
Implements: IProgramFitnessEvaluator
For Beginners
Sometimes the question “is this program good?” needs judgement that a table of expected outputs cannot express. This class lets you write that judgement as a small script: it receives the candidate’s code on standard input and prints a score as JSON. Your script runs in the same locked-down sandbox as the programs being evolved, so a mistake in your scoring script cannot hang or harm the run either — it just gets reported as a failed evaluation.
How It Works
The contract is deliberately small. The evaluator script runs as the sandboxed program; the candidate’s source
arrives on its standard input; and it prints exactly one JSON object to standard output:
{"quality": 0.83, "descriptors": {"length": 0.4}, "objectives": [0.83, 0.1], "artifacts": {"note": "…"}}.
Only quality is required and it must be a finite number. Descriptors and objectives must be finite too;
artifacts are free text that becomes bounded, redacted diagnostics so a chatty evaluator cannot bloat a
checkpoint. Anything else in the object is ignored, so an evaluator may print extra fields for its own use.
Two things here are stricter than the reference implementation. First, the evaluator script is sandboxed exactly
like the candidates: it goes through the same IProgramExecutionEngine, so an evaluator with an
infinite loop is killed by the same wall-clock limit instead of hanging a worker, and one that prints without
stopping is truncated by the same output cap. OpenEvolve writes its evaluator to a file and executes it
unsandboxed in its worker processes, which means a bad evaluator is a worse problem than a bad candidate.
Second, the metrics are schema-checked: a non-finite quality, a descriptor that is not a number, or output that
is not a JSON object produces a Failed result with a diagnostic naming
the problem, never an exception and never a silently-zero score that would look like a legitimately bad program.
Because the sandbox deliberately gives the script a pinned path and a scrubbed environment, an evaluator script
should judge the candidate’s text — its structure, its style, a domain rule. Scoring a candidate by
running it belongs in SandboxedProgramFitnessEvaluator, which executes the candidate itself.
Constructors
| Constructor | Summary |
|---|---|
ScriptProgramFitnessEvaluator(IProgramExecutionEngine engine, string evaluatorScript, ScriptProgramEvaluationOptions options, string id, ProgramMetricAggregator metricAggregator) |
Initializes an evaluator that runs evaluatorScript against every candidate. |
Properties
| Property | Summary |
|---|---|
ProgramLanguage EvaluatorScriptLanguage { get; } |
Gets the language the evaluator script is executed as. |
string Id { get; } |
|
string VersionHash { get; } |
Methods
| Method | Summary |
|---|---|
ScriptProgramEvaluationOptions GetOptions() |
Gets an independent copy of the options this evaluator was constructed with. |
ValueTask<EvolutionTaskResult> EvaluateAsync(ProgramGenome candidate, EvolutionEvaluationContext context, CancellationToken cancellationToken) |