SandboxedProgramFitnessEvaluator

Models & Types · AiDotNet.Evolution.Programs

Scores a candidate program by running it on input/output examples through the asynchronous sandbox contract, and records why each failing case failed.

Implements: IProgramFitnessEvaluator

For Beginners

This runs a generated program once per example, compares what it printed with what it should have printed, and reports the fraction it got right. Because you supply the runner, you decide how dangerous code is contained — a separate process or a container, never this one. When a program fails, the result tells you which kind of failure it was: too slow, would not compile, crashed, or simply wrong. That distinction is what lets you tell “the model is writing slow code” apart from “the model is writing broken code”.

How It Works

This evaluator is the sibling of InputOutputProgramFitnessEvaluator and computes the same quality — the fraction of examples whose captured standard output matches the expected output — but it drives ExecuteAsync instead of the synchronous overload, and that difference shows up in the diagnostics. A candidate that hung, one that failed to compile, and one that ran and printed the wrong answer are three genuinely different situations, and the synchronous contract collapses all of them into “false plus a message”. Here each failing example is labelled with its own code, and an example whose output was cut off at the sandbox’s output cap says so rather than silently counting as a mismatch of unknown origin.

Running asynchronously also matters for throughput: a population being scored against a process sandbox spends nearly all of its time waiting for child processes, so keeping those waits off the thread pool lets the sandbox’s own concurrency limit be the thing that bounds parallelism.

Everything reaching a diagnostic is sanitized, truncated, and marked redacted, and at most eight example-level diagnostics are retained, so a program that prints a megabyte to standard error cannot bloat a checkpoint or leak raw payloads into a log. VersionHash folds in the examples and the comparison mode, so changing the test set correctly invalidates older checkpoints.

Constructors

Constructor Summary
SandboxedProgramFitnessEvaluator(IProgramExecutionEngine engine, IEnumerable<ProgramInputOutputExample> examples, ProgramOutputComparison comparison, string id) Initializes a sandboxed input/output evaluator.

Properties

Property Summary
IReadOnlyList<ProgramInputOutputExample> Examples { get; } Gets the input/output cases this evaluator scores against.
ProgramOutputComparison Comparison { get; } Gets how captured output is compared with expected output.
string Id { get; }
string VersionHash { get; }

Methods

Method Summary
ValueTask<EvolutionTaskResult> EvaluateAsync(ProgramGenome candidate, EvolutionEvaluationContext context, CancellationToken cancellationToken)