SandboxedProgramFitnessEvaluator
Models & Types · AiDotNet.Evolution.Programs
Scores a candidate program by running it on input/output examples through the asynchronous sandbox contract, and records why each failing case failed.
Implements: IProgramFitnessEvaluator
For Beginners
This runs a generated program once per example, compares what it printed with what it should have printed, and reports the fraction it got right. Because you supply the runner, you decide how dangerous code is contained — a separate process or a container, never this one. When a program fails, the result tells you which kind of failure it was: too slow, would not compile, crashed, or simply wrong. That distinction is what lets you tell “the model is writing slow code” apart from “the model is writing broken code”.
How It Works
This evaluator is the sibling of InputOutputProgramFitnessEvaluator and computes the same quality
— the fraction of examples whose captured standard output matches the expected output — but it drives
ExecuteAsync instead of the synchronous overload, and that difference
shows up in the diagnostics. A candidate that hung, one that failed to compile, and one that ran and printed the
wrong answer are three genuinely different situations, and the synchronous contract collapses all of them into
“false plus a message”. Here each failing example is labelled with its own code, and an example whose output was
cut off at the sandbox’s output cap says so rather than silently counting as a mismatch of unknown origin.
Running asynchronously also matters for throughput: a population being scored against a process sandbox spends nearly all of its time waiting for child processes, so keeping those waits off the thread pool lets the sandbox’s own concurrency limit be the thing that bounds parallelism.
Everything reaching a diagnostic is sanitized, truncated, and marked redacted, and at most eight example-level
diagnostics are retained, so a program that prints a megabyte to standard error cannot bloat a checkpoint or leak
raw payloads into a log. VersionHash folds in the examples and the comparison mode, so changing the
test set correctly invalidates older checkpoints.
Constructors
| Constructor | Summary |
|---|---|
SandboxedProgramFitnessEvaluator(IProgramExecutionEngine engine, IEnumerable<ProgramInputOutputExample> examples, ProgramOutputComparison comparison, string id) |
Initializes a sandboxed input/output evaluator. |
Properties
| Property | Summary |
|---|---|
IReadOnlyList<ProgramInputOutputExample> Examples { get; } |
Gets the input/output cases this evaluator scores against. |
ProgramOutputComparison Comparison { get; } |
Gets how captured output is compared with expected output. |
string Id { get; } |
|
string VersionHash { get; } |
Methods
| Method | Summary |
|---|---|
ValueTask<EvolutionTaskResult> EvaluateAsync(ProgramGenome candidate, EvolutionEvaluationContext context, CancellationToken cancellationToken) |