CodeXGlueBenchmark<T>
Models & Types · AiDotNet.Reasoning.Benchmarks
CodeXGLUE benchmark harness (dataset-loader + metric computation).
Implements: IBenchmark<T>
How It Works
CodeXGLUE is a suite of code understanding and code generation tasks. This harness is intentionally “dataset agnostic”: callers provide a JSONL file path and field mapping, and provide the model invocation function.
This harness does not attempt to ship, download, or cache CodeXGLUE datasets; it only provides the evaluation glue.
Constructors
| Constructor | Summary |
|---|---|
CodeXGlueBenchmark(CodeXGlueBenchmarkOptions options) |
Properties
| Property | Summary |
|---|---|
int TotalProblems { get; } |
|
string BenchmarkName { get; } |
|
string Description { get; } |
Methods
| Method | Summary |
|---|---|
Task<BenchmarkResult<T>> EvaluateAsync(Func<string, Task<string>> evaluateFunction, int? sampleSize, CancellationToken cancellationToken) |
|
Task<List<BenchmarkProblem>> LoadProblemsAsync(int? count) |