MBPPBenchmark<T>

Models & Types · AiDotNet.Reasoning.Benchmarks

MBPP (Mostly Basic Python Problems) benchmark for evaluating Python code generation.

Implements: IBenchmark<T>

For Beginners

MBPP is a collection of basic Python programming problems, designed to test fundamental programming skills and algorithmic thinking.

What is MBPP? MBPP contains 974 short Python programming problems at an entry-level difficulty. Each problem includes:

  • A natural language description
  • Code solution
  • 3 test cases

MBPP vs HumanEval:

  • MBPP: 974 problems, more basic, multiple test cases provided
  • HumanEval: 164 problems, more challenging, function signature given
  • Overlap: Both test code generation, but MBPP is more comprehensive

Example problems:

Problem 1: Sum of numbers

Task: Write a function to find the sum of all numbers in a list.
Test Cases:

- sum_list([1, 2, 3]) == 6
- sum_list([10, 20]) == 30
- sum_list([]) == 0

Problem 2: Check palindrome

Task: Write a function to check if a string is a palindrome.
Test Cases:

- is_palindrome("racecar") == True
- is_palindrome("hello") == False
- is_palindrome("") == True

Problem 3: Remove duplicates

Task: Write a function to remove duplicate elements from a list.
Test Cases:

- remove_duplicates([1, 2, 2, 3]) == [1, 2, 3]
- remove_duplicates([]) == []
- remove_duplicates([1, 1, 1]) == [1]

Categories:

  • List operations (sorting, filtering, searching)
  • String manipulation
  • Mathematical operations
  • Basic algorithms (searching, sorting)
  • Data structure operations (lists, dictionaries)
  • Boolean logic

Difficulty levels:

  • Basic: Simple operations (50% of problems)
  • Intermediate: Multiple steps (40% of problems)
  • Advanced: Complex logic (10% of problems)

Performance levels:

  • GPT-3 (Codex): ~59%
  • GPT-3.5: ~70%
  • GPT-4: ~82%
  • Claude 3 Opus: ~78%
  • Claude 3.5 Sonnet: ~85%
  • ChatGPT o1: ~90%
  • AlphaCode: ~75%
  • CodeGen: ~65%

Why it’s useful:

  • Tests basic programming competency
  • More comprehensive than HumanEval (974 vs 164)
  • Includes test cases (can verify correctness)
  • Entry-level difficulty (good for beginners)
  • Real-world relevance (common programming tasks)

Research:

  • “Program Synthesis with Large Language Models” (Austin et al., 2021)
  • https://arxiv.org/abs/2108.07732
  • Dataset: 974 problems with solutions and test cases
  • Used by Google Research for code generation evaluation

Integration with CodeExecutionVerifier: MBPP works particularly well with CodeExecutionVerifier since each problem includes test cases that can be executed to verify correctness.

Constructors

Constructor Summary
MBPPBenchmark()

Properties

Property Summary
int TotalProblems { get; }
string BenchmarkName { get; }
string Description { get; }

Methods

Method Summary
Task<BenchmarkResult<T>> EvaluateAsync(Func<string, Task<string>> evaluateFunction, int? sampleSize, CancellationToken cancellationToken)
Task<List<BenchmarkProblem>> LoadProblemsAsync(int? count)