ShardedModelBase<T, TInput, TOutput>
Base Classes · AiDotNet.DistributedTraining
Provides base implementation for distributed models with parameter sharding.
Implements: IGradientComputable<T, TInput, TOutput>, IModelShape, IShardedModel<T, TInput, TOutput>
Known derived types: DDPModel<T, TInput, TOutput>, FSDPModel<T, TInput, TOutput>, HybridShardedModel<T, TInput, TOutput>, PipelineParallelModel<T, TInput, TOutput>, TensorParallelModel<T, TInput, TOutput>, ZeRO1Model<T, TInput, TOutput>, ZeRO2Model<T, TInput, TOutput>
For Beginners
This is the foundation that all distributed models build upon.
Think of this as a template for splitting a big model across multiple computers or GPUs. It handles common tasks like:
- Dividing model parameters into chunks (sharding)
- Collecting all chunks when needed (gathering)
- Sharing learning updates across all processes (gradient sync)
- Saving and loading distributed models
Specific types of distributed models (like fully sharded or hybrid sharded) inherit from this and add their own strategies. This prevents code duplication and ensures all distributed models work consistently.
How It Works
This abstract class implements common functionality for all sharded models, including parameter management, sharding logic, gradient synchronization, and integration with the model serialization system. Derived classes can customize the sharding strategy, communication pattern, or add optimization-specific features.
Properties
| Property | Summary |
|---|---|
IFullModel<T, TInput, TOutput> WrappedModel { get; } |
|
ILossFunction<T> DefaultLossFunction { get; } |
|
IShardingConfiguration<T> ShardingConfiguration { get; } |
|
Vector<T> LocalParameterShard { get; } |
|
bool SupportsParameterInitialization { get; } |
|
int Rank { get; } |
|
int WorldSize { get; } |
|
long ParameterCount { get; } |
Methods
| Method | Summary |
|---|---|
Dictionary<string, T> GetFeatureImportance() |
|
DynamicShapeInfo GetDynamicShapeInfo() |
|
IEnumerable<int> GetActiveFeatureIndices() |
|
IFullModel<T, TInput, TOutput> Clone() |
|
IFullModel<T, TInput, TOutput> DeepCopy() |
|
IFullModel<T, TInput, TOutput> WithParameters(Vector<T> parameters) |
|
ModelMetadata<T> GetModelMetadata() |
|
TOutput Predict(TInput input) |
|
Vector<T> ComputeGradients(TInput input, TOutput target, ILossFunction<T> lossFunction) |
|
Vector<T> GatherFullParameters() |
|
Vector<T> GetParameters() |
|
Vector<T> SanitizeParameters(Vector<T> parameters) |
|
bool IsFeatureUsed(int featureIndex) |
|
byte[] Serialize() |
|
int[] GetInputShape() |
|
int[] GetOutputShape() |
|
void ApplyGradients(Vector<T> gradients, T learningRate) |
|
void Deserialize(byte[] data) |
|
void Dispose() |
|
void LoadModel(string filePath) |
|
void LoadState(Stream stream) |
Loads the model’s state from a stream. |
void SaveModel(string filePath) |
|
void SaveState(Stream stream) |
Saves the model’s current state to a stream. |
void SetActiveFeatureIndices(IEnumerable<int> featureIndices) |
|
void SetParameters(Vector<T> parameters) |
|
void SynchronizeGradients() |
|
void Train(TInput input, TOutput expectedOutput) |