HumanEvalBenchmark<T>

Models & Types · AiDotNet.Reasoning.Benchmarks

HumanEval benchmark for evaluating Python code generation capabilities.

Implements: IBenchmark<T>

For Beginners

HumanEval is a benchmark of 164 Python programming problems. Each problem asks the model to write a function that passes a set of test cases.

Example problem:

Write a function that returns True if a number is prime, False otherwise.
def is_prime(n: int) -> bool:
# Your code here

Why it’s important:

  • Tests code generation abilities
  • Requires understanding algorithms
  • Tests correctness via unit tests
  • Standard benchmark for code models

Performance levels:

  • GPT-3.5: ~48%
  • GPT-4: ~67%
  • ChatGPT o1: ~92%
  • AlphaCode: ~53%
  • CodeGen: ~29%

Research: “Evaluating Large Language Models Trained on Code” (Chen et al., 2021) https://arxiv.org/abs/2107.03374

Constructors

Constructor Summary
HumanEvalBenchmark(HumanEvalBenchmarkOptions options)

Properties

Property Summary
int TotalProblems { get; }
string BenchmarkName { get; }
string Description { get; }

Methods

Method Summary
Task<BenchmarkResult<T>> EvaluateAsync(Func<string, Task<string>> evaluateFunction, Func<string, string[], CancellationToken, Task<bool>> executionEvaluator, int? sampleSize, CancellationToken cancellationToken)
Task<BenchmarkResult<T>> EvaluateAsync(Func<string, Task<string>> evaluateFunction, int? sampleSize, CancellationToken cancellationToken)
Task<List<BenchmarkProblem>> LoadProblemsAsync(int? count)