JitCompilationConfig

Options & Configuration · AiDotNet.Configuration

Configuration for JIT (Just-In-Time) compilation of model forward/backward passes.

For Beginners

JIT compilation is like building a shortcut for your model. The first time you call Predict the library watches which math operations happen in what order, compiles that pattern into a flat fast pipeline, and from then on just replays the pipeline. Your model doesn’t change — only the plumbing around it gets faster.

How It Works

JIT compilation traces the model’s computation graph on the first call and replays the compiled plan on subsequent calls, eliminating virtual dispatch, per-op allocation, and bounds-checking overhead. Typical speedup is 1.5-3x on CPU and up to 10x on GPU for small batches where dispatch dominates.

Under the hood this binds to the TensorCodecOptions in the Tensors package. The builder writes the options into the thread-local codec config before the built model sees any work; Predict and the tape-based training path then route through their compiled variants via CompiledModelCache.

When to enable:

  • Production inference where Predict is called many times at the same shape.
  • Training loops where the forward+backward is invoked each iteration.
  • Diffusion/autoregressive generation where a sub-network runs tens of times per call.

When to disable / when to keep ThrowOnFailure off:

  • Debugging a model whose forward path has non-Engine tensor accesses

(direct span writes, scalar control flow) — those bake at trace time and will replay stale data. The default fallback to eager execution hides this.

  • Tiny models where the compile cost isn’t amortized (rare — overhead is

one traced forward).

Example YAML (binds through YamlConfigApplier):

Example code:

Constructors

Constructor Summary
JitCompilationConfig()

Properties

Property Summary
JitCompilationConfig Aggressive { get; } Aggressive config — all fusion and constant-folding passes on.
JitCompilationConfig Default { get; } Default config — compilation enabled, failures fall back silently to eager.
JitCompilationConfig Disabled { get; } Explicitly disabled — compiled paths short-circuit to eager.
bool EnableAlgebraicBackward { get; set; } Phase C: Symbolically simplify the backward graph at compile time (CSE, double-transpose elimination, associative regrouping).
bool EnableAttentionFusion { get; set; } Phase 4.2: Fuse attention Q@K^T → Softmax → V patterns.
bool EnableBlasBatch { get; set; } Phase 7.1: Group independent MatMuls into batched calls.
bool EnableConstantFolding { get; set; } Phase 4.5: Precompute static subgraphs at compile time.
bool EnableConvBnFusion { get; set; } Phase 4.1: Fold BatchNorm into Conv2D weights at compile time.
bool EnableDataflowFusion { get; set; } Phase B: Fuse consecutive linear layers into a single multi-layer kernel that keeps data in registers / L1 across layer boundaries.
bool EnableForwardCSE { get; set; } Phase 6.2: Deduplicate identical computations across layers.
bool EnableMixedPrecisionCompilation { get; set; } Phase 7.3: Mixed precision (fp16 forward, fp32 backward).
bool EnablePointwiseFusion { get; set; } Phase 4.3: Merge consecutive pointwise ops into fewer dispatch steps.
bool EnableSpectralDecomposition { get; set; } Phase A: SVD-factorize frozen weight matrices for faster inference.
bool Enabled { get; set; } Master switch.
bool PersistCompiledProgram { get; set; } Whether a saved model embeds its compiled program (default true): the network’s forward as a portable AiDotNet.Tensors program plus a compiled plan for the device it was saved on.
bool ThrowOnFailure { get; set; } When true, a compilation failure in the compiled predictor the model facade uses for neural-network predictions (CompiledNeuralPredictor) propagates as an exception instead of silently falling back to the eager Predict path.
float SpectralErrorTolerance { get; set; } Maximum approximation error per element for spectral decomposition (used as energyThreshold = 1.0 - tolerance for SVD rank selection).
int DataflowFusionMaxHidden { get; set; } Maximum hidden dimension for dataflow fusion L1 residency.

Methods

Method Summary
void ApplyToTensorCodec() Projects this config onto the Tensors-package TensorCodecOptions and installs it as the current thread-local config.
void Validate() Validates the config and throws if settings are inconsistent.