PaperOptimizerAttribute

Attributes · AiDotNet.Attributes

Declares the training recipe a model’s research paper specifies — the optimizer, its hyperparameters, the learning-rate schedule and gradient clipping — so that a model given no optimizer trains the way its paper says rather than at an arbitrary library default.

Inheritance: Attribute → PaperOptimizerAttribute

For Beginners

A paper does not just give a learning rate — it gives a whole recipe: which optimizer, what learning rate, how that rate changes over training, and how gradients are clipped. Those parts only work together. ResNet’s paper uses SGD at 0.1; feeding 0.1 to Adam instead would diverge immediately. So the recipe is declared as a unit, and the library builds the optimizer the paper actually used.

How It Works

Sits alongside ResearchPaperAttribute in the declarative block a model already carries, so the citation and the recipe it produced are read together and can be checked against each other:

The whole recipe, not just the rate. An earlier revision recorded only scalars and applied them to whatever optimizer the model happened to construct, skipping the declaration when the kinds disagreed. That was the wrong shape: it left a model with the wrong optimizer and the wrong schedule while appearing to be paper-faithful. Resolution now CONSTRUCTS the declared optimizer.

Only declare what the paper states. Every hyperparameter is an optional named property, and anything left unset falls back to the library default. An unset property reads as “the paper does not say”; a set one is a claim about the literature. Source exists to make that discipline checkable — a value that cannot be pointed at a section should not be declared, because an invented number looks authoritative and nobody re-checks it.

Precedence. A caller’s ConfigureOptimizer always wins. This supplies the default for the case where nothing was configured at all.

Constructors

Constructor Summary
PaperOptimizerAttribute(OptimizerKind optimizer) Declares the training recipe this model’s paper specifies.

Properties

Property Summary
CyclicMode CyclicPolicy { get; set; } Which cyclic policy the paper uses, for Cyclic.
DecayMode PostWarmupDecay { get; set; } What the rate does AFTER warmup finishes, when Schedule is LinearWarmup.
LearningRateSchedulerType Schedule { get; set; } The learning-rate schedule the paper uses.
OptimizerKind Optimizer { get; } The optimizer the paper specifies, and the one that will be constructed.
RecipeProvenance Provenance { get; set; } How this recipe’s values were arrived at.
SchedulerStepMode ScheduleStepMode { get; set; } Interval, in steps or epochs, between decay events.
TrainingPhase InheritsFrom { get; set; } A phase whose unstated values this one copies, for papers that say a later stage uses the same settings as an earlier one.
TrainingPhase Phase { get; set; } Which stage of training this recipe describes.
bool DeclaresAnyHyperparameter { get; } True when this declaration states anything beyond the optimizer’s identity.
bool UseNesterov { get; set; } Whether the paper uses Nesterov momentum.
double Beta1 { get; set; } First moment decay (Adam-family beta1).
double Beta2 { get; set; } Second moment decay (Adam-family beta2).
double DecayRate { get; set; } Multiplicative decay factor, for exponential and step schedules.
double EmaDecay { get; set; } Decay of an exponential moving average kept over the weights.
double Epsilon { get; set; } Numerical-stability epsilon.
double HoldFraction { get; set; } How much of the run is spent holding the peak rate before decay begins, for TriStage.
double LayerwiseLearningRateDecay { get; set; } Per-layer multiplier applied to the base rate, deepest layer first.
double LearningRate { get; set; } The paper’s learning rate.
double MaxGradientNorm { get; set; } The paper’s gradient-norm clip.
double MinLearningRate { get; set; } Floor the schedule decays towards.
double Momentum { get; set; } Momentum coefficient, for SGD-momentum, RMSProp and friends.
double Rho { get; set; } RMSProp’s decay / smoothing constant (often written rho or alpha).
double WarmupFraction { get; set; } Warmup expressed as a fraction of the whole run, for papers that state it that way.
double WeightDecay { get; set; } The paper’s weight decay.
double[] MilestoneFractions { get; set; } Decay points stated as fractions of the whole run, for papers that give them that way.
double[] SearchedValues { get; set; } The learning rates a paper searched over, when Provenance is Searched.
int EarlyStoppingPatience { get; set; } Epochs without improvement before training stops.
int GradientAccumulationSteps { get; set; } Micro-batches accumulated before each optimizer step.
int ReferenceBatchSize { get; set; } The batch size the paper’s learning rate was chosen for.
int StepSize { get; set; }
int WarmupSteps { get; set; } Warmup steps before the main schedule begins.
int[] Milestones { get; set; } The exact steps a paper decays at, for schedules stated as a list rather than an interval.
string Component { get; set; } Which part of a composite model this recipe applies to, for example "discriminator".
string Source { get; set; } Where in the paper this recipe comes from, for example `“Sec.
string Variant { get; set; } Which model size or configuration variant this recipe applies to, matched against PaperOptimizerVariant.