ModelShard<T>

Models & Types · AiDotNet.Diffusion.Memory

Enables model sharding across multiple devices for large model inference.

For Beginners

Large AI models like Stable Diffusion XL may not fit in a single GPU.

Model sharding solves this by:

  • Splitting the model layers across devices
  • Running each device’s layers in sequence
  • Passing intermediate results between devices

Example: UNet with 24 layers on 4 GPUs:

  • GPU 0: Layers 1-6
  • GPU 1: Layers 7-12
  • GPU 2: Layers 13-18
  • GPU 3: Layers 19-24

Data flows: Input -> GPU0 -> GPU1 -> GPU2 -> GPU3 -> Output

Usage:

var layers = new List<ILayer<float>>
{
new DenseLayer<float>(outputSize: 256),
new DenseLayer<float>(outputSize: 128)
};
var input = Tensor<float>.CreateRandom(1, 512);
var shard = new ModelShard<float>(layers, numDevices: 4);
var output = shard.Forward(input);

How It Works

Model sharding (also known as model parallelism or pipeline parallelism) splits a large model across multiple devices (GPUs/CPUs) when it’s too large to fit on a single device.

Constructors

Constructor Summary
ModelShard(IEnumerable<ILayer<T>> layers, int numDevices, ShardingConfig config) Initializes model sharding across specified number of devices.

Properties

Property Summary
ShardingConfig Config { get; } Sharding configuration.

Methods

Method Summary
IReadOnlyDictionary<int, long> GetDeviceMemoryUsage() Gets memory usage per device.
IReadOnlyList<ILayer<T>> GetDeviceLayers(int device) Gets layers assigned to a specific device.
Tensor<T> Forward(Tensor<T> input) Performs forward pass through all sharded layers.
Tensor<T> Forward(Tensor<T> input, Tensor<T> context) Performs forward pass with context (for conditional generation).
int GetLayerDevice(ILayer<T> layer) Gets the device assignment for a layer.
string ToString() Gets a summary of the sharding distribution.
void UpdateParameters(T learningRate) Updates parameters on all devices.