MMLUBenchmark<T>
Models & Types · AiDotNet.Reasoning.Benchmarks
MMLU (Massive Multitask Language Understanding) benchmark for evaluating world knowledge.
Implements: IBenchmark<T>
For Beginners
MMLU is like a comprehensive standardized test for AI, covering 57 subjects from elementary to professional level.
What is MMLU? MMLU tests knowledge across diverse academic and professional domains:
- STEM: Mathematics, Physics, Chemistry, Biology, Computer Science
- Humanities: History, Philosophy, Law
- Social Sciences: Psychology, Economics, Sociology
- Other: Medicine, Business, Professional Knowledge
Format: Multiple choice questions (A, B, C, D) spanning different difficulty levels:
- Elementary
- High School
- College
- Professional
Example questions:
Elementary Math: Q: What is 7 × 8? A) 54 B) 56 C) 64 D) 48 Answer: B
College Physics: Q: What is the ground state energy of a hydrogen atom? A) -13.6 eV B) -27.2 eV C) -6.8 eV D) 0 eV Answer: A
Professional Medicine: Q: A 45-year-old presents with sudden chest pain. What is the most appropriate first test? A) CT scan B) ECG C) Blood test D) X-ray Answer: B
Why it’s important:
- Comprehensive knowledge evaluation
- Tests reasoning + memorization
- Standard benchmark for LLMs
- Measures real-world applicability
Performance levels:
- Random guessing: 25%
- Average human expert: ~90% (in their domain)
- GPT-3.5: ~70%
- GPT-4: ~86%
- Claude 3 Opus: ~87%
- Claude 3.5 Sonnet: ~89%
- ChatGPT o1: ~91%
- Gemini Pro 1.5: ~90%
57 Subject categories:
STEM (18 subjects):
- Abstract Algebra, Astronomy, College Biology, College Chemistry
- College Computer Science, College Mathematics, College Physics
- Conceptual Physics, Electrical Engineering, Elementary Mathematics
- High School Biology, High School Chemistry, High School Computer Science
- High School Mathematics, High School Physics, High School Statistics
- Machine Learning
Humanities (13 subjects):
- Formal Logic, High School European History, High School US History
- High School World History, International Law, Jurisprudence
- Logical Fallacies, Moral Disputes, Moral Scenarios
- Philosophy, Prehistory, Professional Law, World Religions
Social Sciences (12 subjects):
- Econometrics, High School Geography, High School Government and Politics
- High School Macroeconomics, High School Microeconomics
- High School Psychology, Human Sexuality, Professional Psychology
- Public Relations, Security Studies, Sociology, US Foreign Policy
Other (14 subjects):
- Anatomy, Business Ethics, Clinical Knowledge, College Medicine
- Global Facts, Human Aging, Management, Marketing
- Medical Genetics, Miscellaneous, Nutrition, Professional Accounting
- Professional Medicine, Virology
Research:
- “Measuring Massive Multitask Language Understanding” (Hendrycks et al., 2021)
https://arxiv.org/abs/2009.03300- Dataset: 15,908 questions across 57 tasks
Constructors
| Constructor | Summary |
|---|---|
MMLUBenchmark() |
Properties
| Property | Summary |
|---|---|
int TotalProblems { get; } |
|
string BenchmarkName { get; } |
|
string Description { get; } |
Methods
| Method | Summary |
|---|---|
Task<BenchmarkResult<T>> EvaluateAsync(Func<string, Task<string>> evaluateFunction, int? sampleSize, CancellationToken cancellationToken) |
|
Task<List<BenchmarkProblem>> LoadProblemsAsync(int? count) |