BenchmarkingOptions

Options & Configuration · AiDotNet.Configuration

Configuration options for running benchmarks through the AiDotNet facade.

For Beginners

Benchmarking is how you measure model quality using standardized tests. You choose a suite (like GSM8K for math reasoning), and AiDotNet runs it and returns a report.

How It Works

This options class is designed for use with AiModelBuilder.ConfigureBenchmarking(...) and AiModelResult.EvaluateBenchmarksAsync(...). Users specify which benchmark suites to run, and AiDotNet orchestrates the execution behind the scenes.

Constructors

Constructor Summary
BenchmarkingOptions()

Properties

Property Summary
BenchmarkFailurePolicy FailurePolicy { get; set; } Gets or sets how failures should be handled.
BenchmarkReportDetailLevel DetailLevel { get; set; } Gets or sets how much detail should be included in reports.
BenchmarkSuite[] Suites { get; set; } Gets or sets the benchmark suites to run.
CausalLmBenchmarkOptions LanguageModel { get; set; } Gets or sets causal language-model benchmark configuration (LAMBADA and Perplexity).
FederatedTabularBenchmarkOptions Tabular { get; set; } Gets or sets federated tabular benchmark configuration (synthetic non-IID suite).
FederatedTextBenchmarkOptions Text { get; set; } Gets or sets federated text benchmark configuration (Sent140/Shakespeare suites).
FederatedVisionBenchmarkOptions Vision { get; set; } Gets or sets federated vision benchmark configuration (FEMNIST/CIFAR suites).
LeafFederatedBenchmarkOptions Leaf { get; set; } Gets or sets LEAF federated benchmark configuration (required when running LEAF).
bool AttachReportToResult { get; set; } Gets or sets whether the generated report should be attached to the model result.
bool CiMode { get; set; } Gets or sets whether benchmarking should run in CI-friendly mode.
int? SampleSize { get; set; } Gets or sets an optional sample size for suites that support sampling.
int? Seed { get; set; } Gets or sets an optional deterministic seed used for CI-friendly sampling.