ModelShard<T>
Models & Types · AiDotNet.Diffusion.Memory
Enables model sharding across multiple devices for large model inference.
For Beginners
Large AI models like Stable Diffusion XL may not fit in a single GPU.
Model sharding solves this by:
- Splitting the model layers across devices
- Running each device’s layers in sequence
- Passing intermediate results between devices
Example: UNet with 24 layers on 4 GPUs:
- GPU 0: Layers 1-6
- GPU 1: Layers 7-12
- GPU 2: Layers 13-18
- GPU 3: Layers 19-24
Data flows: Input -> GPU0 -> GPU1 -> GPU2 -> GPU3 -> Output
Usage:
var layers = new List<ILayer<float>>
{
new DenseLayer<float>(outputSize: 256),
new DenseLayer<float>(outputSize: 128)
};
var input = Tensor<float>.CreateRandom(1, 512);
var shard = new ModelShard<float>(layers, numDevices: 4);
var output = shard.Forward(input);
How It Works
Model sharding (also known as model parallelism or pipeline parallelism) splits a large model across multiple devices (GPUs/CPUs) when it’s too large to fit on a single device.
Constructors
| Constructor | Summary |
|---|---|
ModelShard(IEnumerable<ILayer<T>> layers, int numDevices, ShardingConfig config) |
Initializes model sharding across specified number of devices. |
Properties
| Property | Summary |
|---|---|
ShardingConfig Config { get; } |
Sharding configuration. |
Methods
| Method | Summary |
|---|---|
IReadOnlyDictionary<int, long> GetDeviceMemoryUsage() |
Gets memory usage per device. |
IReadOnlyList<ILayer<T>> GetDeviceLayers(int device) |
Gets layers assigned to a specific device. |
Tensor<T> Forward(Tensor<T> input) |
Performs forward pass through all sharded layers. |
Tensor<T> Forward(Tensor<T> input, Tensor<T> context) |
Performs forward pass with context (for conditional generation). |
int GetLayerDevice(ILayer<T> layer) |
Gets the device assignment for a layer. |
string ToString() |
Gets a summary of the sharding distribution. |
void UpdateParameters(T learningRate) |
Updates parameters on all devices. |