InferenceArenaSettings

Options & Configuration · AiDotNet.NeuralNetworks

Global opt-in switch for the inference forward-caching allocator (#1661 / Tensors #661). When enabled, Predict runs the forward inside a recycled TensorArena so intermediate-tensor buffers are reused per call (~98% intermediate allocation reduction, bit-identical output — proven in Tensors #661).

How It Works

Default ON — per the facade pattern, this is an industry-standard zero-config default (PyTorch’s caching allocator is always on for inference): callers get the allocation win without opting in. Every NeuralNetworkBase{T} model routes through the PredictCore funnel, and the forward is bit-identical with the arena on or off. Set AIDOTNET_INFERENCE_ARENA=0 to disable (escape hatch), or set Enabled in code (e.g. for A/B alloc tests).

Process-wide by design: TensorArena.Current is [ThreadStatic], so concurrent Predict on a single model instance already requires external serialization — the same contract as the existing eval-mode flip in Predict.

Properties

Property Summary
bool DiffusionDenoiseEnabled { get; set; } Whether the multi-step diffusion denoise loop (DiffusionModelBase.Generate) opens a per-step TensorArena (recycling each step’s noise-predictor + scheduler intermediates instead of GC-churning them).
bool Enabled { get; set; } Whether Predict opens a per-call TensorArena around the forward.
bool RecycleLayerScratch { get; set; } Whether the eager inference forward Resets the per-call TensorArena BETWEEN layers so a single forward reuses one block’s scratch for every layer — making the forward’s peak memory ~O(one block) instead of O(depth · per-block churn) (#182…