IShardedModel<T, TInput, TOutput>

Interfaces · AiDotNet.DistributedTraining

Defines the contract for models that support distributed training with parameter sharding.

Implements: IFullModel<T, TInput, TOutput>

Known implementations: ShardedModelBase<T, TInput, TOutput>

For Beginners

A sharded model is like having a team working on a large puzzle together. Instead of one person holding all the puzzle pieces (parameters), each person holds only a portion. When someone needs to see the full picture, everyone shares their pieces (AllGather). When the team learns something new, everyone combines their learnings (AllReduce).

How It Works

This allows training models that are too large to fit on a single GPU or machine.

Properties

Property Summary
IFullModel<T, TInput, TOutput> WrappedModel { get; } Gets the underlying wrapped model.
IShardingConfiguration<T> ShardingConfiguration { get; } Gets the configuration for this sharded model.
Vector<T> LocalParameterShard { get; } Gets the portion of parameters owned by this process.
int Rank { get; } Gets the rank of this process in the distributed group.
int WorldSize { get; } Gets the total number of processes in the distributed group.

Methods

Method Summary
Vector<T> GatherFullParameters() Gets the full set of parameters by gathering from all processes.
void SynchronizeGradients() Synchronizes gradients across all processes using AllReduce.