ShardedModelBase<T, TInput, TOutput>

Base Classes · AiDotNet.DistributedTraining

Provides base implementation for distributed models with parameter sharding.

Implements: IGradientComputable<T, TInput, TOutput>, IModelShape, IShardedModel<T, TInput, TOutput>

Known derived types: DDPModel<T, TInput, TOutput>, FSDPModel<T, TInput, TOutput>, HybridShardedModel<T, TInput, TOutput>, PipelineParallelModel<T, TInput, TOutput>, TensorParallelModel<T, TInput, TOutput>, ZeRO1Model<T, TInput, TOutput>, ZeRO2Model<T, TInput, TOutput>

For Beginners

This is the foundation that all distributed models build upon.

Think of this as a template for splitting a big model across multiple computers or GPUs. It handles common tasks like:

  • Dividing model parameters into chunks (sharding)
  • Collecting all chunks when needed (gathering)
  • Sharing learning updates across all processes (gradient sync)
  • Saving and loading distributed models

Specific types of distributed models (like fully sharded or hybrid sharded) inherit from this and add their own strategies. This prevents code duplication and ensures all distributed models work consistently.

How It Works

This abstract class implements common functionality for all sharded models, including parameter management, sharding logic, gradient synchronization, and integration with the model serialization system. Derived classes can customize the sharding strategy, communication pattern, or add optimization-specific features.

Properties

Property Summary
IFullModel<T, TInput, TOutput> WrappedModel { get; }
ILossFunction<T> DefaultLossFunction { get; }
IShardingConfiguration<T> ShardingConfiguration { get; }
Vector<T> LocalParameterShard { get; }
bool SupportsParameterInitialization { get; }
int Rank { get; }
int WorldSize { get; }
long ParameterCount { get; }

Methods

Method Summary
Dictionary<string, T> GetFeatureImportance()
DynamicShapeInfo GetDynamicShapeInfo()
IEnumerable<int> GetActiveFeatureIndices()
IFullModel<T, TInput, TOutput> Clone()
IFullModel<T, TInput, TOutput> DeepCopy()
IFullModel<T, TInput, TOutput> WithParameters(Vector<T> parameters)
ModelMetadata<T> GetModelMetadata()
TOutput Predict(TInput input)
Vector<T> ComputeGradients(TInput input, TOutput target, ILossFunction<T> lossFunction)
Vector<T> GatherFullParameters()
Vector<T> GetParameters()
Vector<T> SanitizeParameters(Vector<T> parameters)
bool IsFeatureUsed(int featureIndex)
byte[] Serialize()
int[] GetInputShape()
int[] GetOutputShape()
void ApplyGradients(Vector<T> gradients, T learningRate)
void Deserialize(byte[] data)
void Dispose()
void LoadModel(string filePath)
void LoadState(Stream stream) Loads the model’s state from a stream.
void SaveModel(string filePath)
void SaveState(Stream stream) Saves the model’s current state to a stream.
void SetActiveFeatureIndices(IEnumerable<int> featureIndices)
void SetParameters(Vector<T> parameters)
void SynchronizeGradients()
void Train(TInput input, TOutput expectedOutput)