ForwardScratchGate
Helpers & Utilities · AiDotNet.Helpers
Process-wide gate for the #1672 destination-buffer (*Into) forward path.
How It Works
The DiT / SiT inference forward allocates a fresh tensor for every op — SDPA scores
(~4 GB/forward), the AdaLN/gate broadcasts (~1.9 GB), the MLP/QKV projections — which
stalls the cores on memory bandwidth (issue #1672). When this gate is ON, the
affected call sites route through the new AiDotNet.Tensors``*Into engine
overloads that write into a per-predictor reusable scratch buffer instead of allocating.
OFF by default. Enable with environment variable AIDOTNET_FWD_SCRATCH=1.
The *Into path is the same math into a reused buffer, so output is bit-identical;
the flag exists only to give a clean isolated A/B measurement and to keep the scratch
reuse strictly opt-in (it is safe only on the no-tape inference forward — the scratch is a
dedicated per-op buffer that no other live tensor aliases, distinct from the #1668 shared
arena). When OFF, the original allocating path runs unchanged.
Properties
| Property | Summary |
|---|---|
bool AdaLn { get; } |
AdaLN/gate broadcast *Into sub-gate. |
bool Enabled { get; } |
True when the #1672 scratch path is enabled at all. |
bool FusedLinear { get; } |
FusedLinear *Into sub-gate. |
bool FusedQkv { get; } |
FusedQKV sub-gate. |
bool Sdpa { get; } |
SDPA *Into sub-gate. |
Fields
| Field | Summary |
|---|---|
bool? Override |
Optional in-process override for the gate, used by the correctness A/B probe to flip the path on the SAME model instance (weights are random per process, so a cross-process checksum can’t prove bit-identicality). |