MmluDataLoaderOptions

Options & Configuration · AiDotNet.Data.Text.Benchmarks

Configuration options for the MMLU (Massive Multitask Language Understanding) loader (Hendrycks et al. 2021).

How It Works

MMLU — 57 subjects × ~270 multi-choice questions each, spanning STEM, humanities, social sciences, professional, and other categories. The canonical broad-knowledge LLM eval benchmark since GPT-3. 4-way multiple choice. Splits: dev (5/subject, used for few-shot prompting), val (~85/subject), test (~14k total).

Constructors

Constructor Summary
MmluDataLoaderOptions()

Properties

Property Summary
DatasetSplit Split { get; set; } Dataset split.
bool AutoDownload { get; set; } Automatically download if not present.
int MaxQuestionLength { get; set; } Maximum encoded question + choice length in tokens.
int VocabularySize { get; set; } Maximum vocabulary size for the BPE-style tokenizer.
int? MaxSamples { get; set; } Optional maximum number of samples to load (for fast iteration / smoke testing).
string DataPath { get; set; } Root data path.
string SubjectFilter { get; set; } Optional subject filter (case-insensitive substring).

Methods

Method Summary
void Validate() Validates that all option values are within acceptable ranges.