MATHBenchmark<T>

Models & Types · AiDotNet.Reasoning.Benchmarks

MATH benchmark for evaluating advanced mathematical reasoning.

Implements: IBenchmark<T>

For Beginners

The MATH dataset contains 12,500 challenging competition mathematics problems from high school math competitions (AMC, AIME, etc.). These are significantly harder than GSM8K.

Example problems:

  • “Find the sum of all positive integers n such that sqrt(n^2 + 85) is an integer.”
  • “A square is inscribed in a circle. What is the ratio of the area of the circle to the square?”
  • “Solve the system of equations: x + y + z = 6, xy + xz + yz = 11, xyz = 6”

Why it’s important:

  • Tests advanced mathematical reasoning
  • Requires complex multi-step solutions
  • Includes algebra, geometry, number theory, calculus
  • Benchmark for reasoning capability at competition level

Performance levels:

  • Human (expert): 90-95%
  • GPT-3.5: ~7%
  • GPT-4: ~42%
  • ChatGPT o1: ~85%
  • DeepSeek-R1: ~79.8%
  • Minerva (540B): ~50%

Research: “Measuring Mathematical Problem Solving With the MATH Dataset” (Hendrycks et al., 2021) https://arxiv.org/abs/2103.03874

Constructors

Constructor Summary
MATHBenchmark()

Properties

Property Summary
int TotalProblems { get; }
string BenchmarkName { get; }
string Description { get; }

Methods

Method Summary
Task<BenchmarkResult<T>> EvaluateAsync(Func<string, Task<string>> evaluateFunction, int? sampleSize, CancellationToken cancellationToken)
Task<List<BenchmarkProblem>> LoadProblemsAsync(int? count)