HumanEvalBenchmark<T>
Models & Types · AiDotNet.Reasoning.Benchmarks
HumanEval benchmark for evaluating Python code generation capabilities.
Implements: IBenchmark<T>
For Beginners
HumanEval is a benchmark of 164 Python programming problems. Each problem asks the model to write a function that passes a set of test cases.
Example problem:
Write a function that returns True if a number is prime, False otherwise.
def is_prime(n: int) -> bool:
# Your code here
Why it’s important:
- Tests code generation abilities
- Requires understanding algorithms
- Tests correctness via unit tests
- Standard benchmark for code models
Performance levels:
- GPT-3.5: ~48%
- GPT-4: ~67%
- ChatGPT o1: ~92%
- AlphaCode: ~53%
- CodeGen: ~29%
Research:
“Evaluating Large Language Models Trained on Code” (Chen et al., 2021)
https://arxiv.org/abs/2107.03374
Constructors
| Constructor | Summary |
|---|---|
HumanEvalBenchmark(HumanEvalBenchmarkOptions options) |
Properties
| Property | Summary |
|---|---|
int TotalProblems { get; } |
|
string BenchmarkName { get; } |
|
string Description { get; } |
Methods
| Method | Summary |
|---|---|
Task<BenchmarkResult<T>> EvaluateAsync(Func<string, Task<string>> evaluateFunction, Func<string, string[], CancellationToken, Task<bool>> executionEvaluator, int? sampleSize, CancellationToken cancellationToken) |
|
Task<BenchmarkResult<T>> EvaluateAsync(Func<string, Task<string>> evaluateFunction, int? sampleSize, CancellationToken cancellationToken) |
|
Task<List<BenchmarkProblem>> LoadProblemsAsync(int? count) |