GSM8KBenchmark<T>

Models & Types · AiDotNet.Reasoning.Benchmarks

Grade School Math 8K (GSM8K) benchmark for evaluating mathematical reasoning.

Implements: IBenchmark<T>

For Beginners

GSM8K is a dataset of 8,500 grade school math word problems. These are the kinds of problems you’d see in elementary school math:

Example problems:

  • “Janet has 15 apples. She gives 40% to her friend. How many does she have left?”
  • “A train travels 60 mph for 2.5 hours. How far does it go?”
  • “If 3 pizzas cost $45, how much does 1 pizza cost?”

Why it’s important:

  • Tests basic mathematical reasoning
  • Requires understanding word problems
  • Needs step-by-step calculation
  • Benchmark for many reasoning models

Performance levels:

  • Human performance: ~90-95%
  • GPT-3.5: ~57%
  • GPT-4: ~92%
  • ChatGPT o1: ~95%
  • DeepSeek-R1: ~97%

Research: “Training Verifiers to Solve Math Word Problems” (Cobbe et al., 2021) https://arxiv.org/abs/2110.14168

Constructors

Constructor Summary
GSM8KBenchmark() Initializes a new instance of the GSM8KBenchmark class.

Properties

Property Summary
int TotalProblems { get; }
string BenchmarkName { get; }
string Description { get; }

Methods

Method Summary
Task<BenchmarkResult<T>> EvaluateAsync(Func<string, Task<string>> evaluateFunction, int? sampleSize, CancellationToken cancellationToken)
Task<List<BenchmarkProblem>> LoadProblemsAsync(int? count)