ForwardScratchGate

Helpers & Utilities · AiDotNet.Helpers

Process-wide gate for the #1672 destination-buffer (*Into) forward path.

How It Works

The DiT / SiT inference forward allocates a fresh tensor for every op — SDPA scores (~4 GB/forward), the AdaLN/gate broadcasts (~1.9 GB), the MLP/QKV projections — which stalls the cores on memory bandwidth (issue #1672). When this gate is ON, the affected call sites route through the new AiDotNet.Tensors``*Into engine overloads that write into a per-predictor reusable scratch buffer instead of allocating.

OFF by default. Enable with environment variable AIDOTNET_FWD_SCRATCH=1. The *Into path is the same math into a reused buffer, so output is bit-identical; the flag exists only to give a clean isolated A/B measurement and to keep the scratch reuse strictly opt-in (it is safe only on the no-tape inference forward — the scratch is a dedicated per-op buffer that no other live tensor aliases, distinct from the #1668 shared arena). When OFF, the original allocating path runs unchanged.

Properties

Property Summary
bool AdaLn { get; } AdaLN/gate broadcast *Into sub-gate.
bool Enabled { get; } True when the #1672 scratch path is enabled at all.
bool FusedLinear { get; } FusedLinear *Into sub-gate.
bool FusedQkv { get; } FusedQKV sub-gate.
bool Sdpa { get; } SDPA *Into sub-gate.

Fields

Field Summary
bool? Override Optional in-process override for the gate, used by the correctness A/B probe to flip the path on the SAME model instance (weights are random per process, so a cross-process checksum can’t prove bit-identicality).