WeightStreamingConfig
Options & Configuration · AiDotNet.Deployment.Configuration
Configuration for weight streaming (paging large model weights to disk when they don’t fit in RAM). Issue #1222 / weight-streaming v1.
For Beginners
Modern foundation models (GPT-class LLMs, vision-language models like PaLM-E) have so many parameters that the raw weights don’t fit in your machine’s RAM — a 562B-parameter model is ~2.25 TB at fp32. Weight streaming pages those weights to a fast local disk and only loads the slice the model needs RIGHT NOW into RAM, so the model can run on a laptop instead of a multi-GPU workstation. The trade-off is throughput: cold disk reads add latency. AiDotNet’s defaults handle this for you (auto-enabled when the model crosses ~10B parameters); use this config to override.
How It Works
The default behavior is “smart on”: every neural network you
build through IAiModelBuilder<T,TInput,TOutput>
runs eagerly (zero overhead) until its parameter count crosses the
threshold, at which point streaming auto-engages. To force streaming
on / off regardless of size, set Enabled explicitly. To
override the auto-detect threshold, set
ThresholdParameters.
Constructors
| Constructor | Summary |
|---|---|
WeightStreamingConfig() |
Properties
| Property | Summary |
|---|---|
bool? Enabled { get; set; } |
Gets or sets whether weight streaming is enabled. |
long? ThresholdParameters { get; set; } |
Gets or sets the parameter-count threshold above which auto-detect engages streaming. |