CodeXGlueBenchmark<T>

Models & Types · AiDotNet.Reasoning.Benchmarks

CodeXGLUE benchmark harness (dataset-loader + metric computation).

Implements: IBenchmark<T>

How It Works

CodeXGLUE is a suite of code understanding and code generation tasks. This harness is intentionally “dataset agnostic”: callers provide a JSONL file path and field mapping, and provide the model invocation function.

This harness does not attempt to ship, download, or cache CodeXGLUE datasets; it only provides the evaluation glue.

Constructors

Constructor Summary
CodeXGlueBenchmark(CodeXGlueBenchmarkOptions options)

Properties

Property Summary
int TotalProblems { get; }
string BenchmarkName { get; }
string Description { get; }

Methods

Method Summary
Task<BenchmarkResult<T>> EvaluateAsync(Func<string, Task<string>> evaluateFunction, int? sampleSize, CancellationToken cancellationToken)
Task<List<BenchmarkProblem>> LoadProblemsAsync(int? count)