MmluDataLoaderOptions
Options & Configuration · AiDotNet.Data.Text.Benchmarks
Configuration options for the MMLU (Massive Multitask Language Understanding) loader (Hendrycks et al. 2021).
How It Works
MMLU — 57 subjects × ~270 multi-choice questions each, spanning STEM, humanities, social sciences, professional, and other categories. The canonical broad-knowledge LLM eval benchmark since GPT-3. 4-way multiple choice. Splits: dev (5/subject, used for few-shot prompting), val (~85/subject), test (~14k total).
Constructors
| Constructor | Summary |
|---|---|
MmluDataLoaderOptions() |
Properties
| Property | Summary |
|---|---|
DatasetSplit Split { get; set; } |
Dataset split. |
bool AutoDownload { get; set; } |
Automatically download if not present. |
int MaxQuestionLength { get; set; } |
Maximum encoded question + choice length in tokens. |
int VocabularySize { get; set; } |
Maximum vocabulary size for the BPE-style tokenizer. |
int? MaxSamples { get; set; } |
Optional maximum number of samples to load (for fast iteration / smoke testing). |
string DataPath { get; set; } |
Root data path. |
string SubjectFilter { get; set; } |
Optional subject filter (case-insensitive substring). |
Methods
| Method | Summary |
|---|---|
void Validate() |
Validates that all option values are within acceptable ranges. |