HumanEvalDataLoader<T>

Models & Types · AiDotNet.Data.Text.Benchmarks

Loads the HumanEval Python code-generation benchmark (Chen et al. 2021).

Inheritance: DataLoaderBase<T> → InputOutputDataLoaderBase<T, Tensor<T>, Tensor<T>> → HumanEvalDataLoader<T>

How It Works

Expects {DataPath}/HumanEval.jsonl. Auto-download fetches the canonical OpenAI release. Each record has prompt, canonical_solution, test, entry_point; this loader uses prompt as features and canonical_solution as the target. Scoring against the unit tests requires running the model output through a Python sandbox and is out of scope for the loader.

Constructors

Constructor Summary
HumanEvalDataLoader(HumanEvalDataLoaderOptions options) Creates a new HumanEval data loader.

Properties

Property Summary
int FeatureCount { get; }
int OutputDimension { get; }
int TotalCount { get; }
string Description { get; }
string Name { get; }

Methods

Method Summary
ValueTuple<IInputOutputDataLoader<T, Tensor<T>, Tensor<T>>, IInputOutputDataLoader<T, Tensor<T>, Tensor<T>>, IInputOutputDataLoader<T, Tensor<T>, Tensor<T>>> Split(double trainRatio, double validationRatio, int? seed)