IBenchmark<T>

Interfaces · AiDotNet.Interfaces

Defines the contract for reasoning benchmarks that evaluate model performance.

Known implementations: ARCAGIBenchmark<T>, BoolQBenchmark<T>, CodeXGlueBenchmark<T>, CommonsenseQABenchmark<T>, DROPBenchmark<T>, GSM8KBenchmark<T>, HellaSwagBenchmark<T>, HumanEvalBenchmark<T>, LogiQABenchmark<T>, MATHBenchmark<T>, MBPPBenchmark<T>, MMLUBenchmark<T>, PIQABenchmark<T>, TruthfulQABenchmark<T>, WinoGrandeBenchmark<T>

For Beginners

A benchmark is like a standardized test for AI reasoning systems. Just like students take SAT or ACT tests to measure their abilities, AI systems are evaluated on benchmarks to measure their reasoning capabilities.

Common benchmarks:

  • GSM8K: Grade school math problems (8,000 questions)
  • MATH: Competition-level mathematics
  • HumanEval: Code generation tasks
  • MMLU: Multiple choice questions across many subjects
  • ARC-AGI: Abstract reasoning puzzles

Why benchmarks matter:

  • Objective measurement of performance
  • Compare different approaches
  • Track improvements over time
  • Identify strengths and weaknesses

Example:

var benchmark = new GSM8KBenchmark<double>();
// your model answers one question at a time
Func<string, Task<string>> reasoner = question => Task.FromResult("42");
var results = await benchmark.EvaluateAsync(reasoner, sampleSize: 100);
// Result is available in the returned value // "Accuracy: 87.5%"

Properties

Property Summary
int TotalProblems { get; } Gets the total number of problems in this benchmark.
string BenchmarkName { get; } Gets the name of this benchmark.
string Description { get; } Gets a description of what this benchmark measures.

Methods

Method Summary
Task<BenchmarkResult<T>> EvaluateAsync(Func<string, Task<string>> evaluateFunction, int? sampleSize, CancellationToken cancellationToken) Evaluates a reasoning strategy on this benchmark.
Task<List<BenchmarkProblem>> LoadProblemsAsync(int? count) Loads benchmark problems (for inspection or custom evaluation).