DyLoRAAdapter<T>
Models & Types · AiDotNet.LoRA.Adapters
DyLoRA (Dynamic LoRA) adapter that trains with multiple ranks simultaneously.
Inheritance: LayerBase<T> → LoRAAdapterBase<T> → DyLoRAAdapter<T>
For Beginners
DyLoRA is like LoRA with a superpower - flexibility!
Standard LoRA problem:
- You choose rank=8 and train
- Later realize rank=4 would work fine (save memory/speed)
- Or need rank=16 for better quality
- Must retrain from scratch with the new rank
DyLoRA solution:
- Train once with multiple ranks (e.g., [2, 4, 8, 16])
- Deploy with ANY of those ranks without retraining
- Switch between ranks at runtime based on device capabilities
How it works:
- Train with MaxRank (e.g., 16) but randomly use smaller ranks during training
- Nested dropout ensures each rank works independently
- After training, pick deployment rank based on needs (2=fastest, 16=best quality)
Use cases:
- Deploy same model to mobile (rank=2) and server (rank=16)
- Dynamic quality scaling based on battery level
- A/B testing different rank/quality trade-offs
- Training once, deploying everywhere
Example: Train with ActiveRanks=[2,4,8], deploy with:
- Rank=2 for mobile devices (98% parameter reduction, good quality)
- Rank=4 for tablets (95% parameter reduction, better quality)
- Rank=8 for desktops (90% parameter reduction, best quality)
How It Works
DyLoRA extends the standard LoRA approach by training multiple rank configurations simultaneously using a nested dropout technique. This allows a single trained adapter to be deployed at different rank levels without retraining, providing flexibility for different hardware constraints or performance requirements.
The key innovation is nested dropout: during training, for each forward pass, a random rank r is selected from the active ranks, and only the first r components of matrices A and B are used. This ensures that smaller ranks can function independently and don’t rely on higher-rank components.
Constructors
| Constructor | Summary |
|---|---|
DyLoRAAdapter(ILayer<T> baseLayer, int maxRank, int[] activeRanks, double alpha, bool freezeBaseLayer) |
Initializes a new DyLoRA adapter with the specified parameters. |
Properties
| Property | Summary |
|---|---|
bool IsTraining { get; set; } |
Gets or sets whether the adapter is in training mode. |
int CurrentDeploymentRank { get; set; } |
Gets or sets the current deployment rank used during inference. |
int MaxRank { get; } |
Gets the maximum rank of the DyLoRA adapter. |
int[] ActiveRanks { get; } |
Gets the array of active ranks used during training. |
Methods
| Method | Summary |
|---|---|
ILayer<T> MergeToOriginalLayer() |
Merges the DyLoRA adaptation into the base layer using the current deployment rank. |
void Eval() |
Sets the adapter to evaluation mode (uses fixed deployment rank). |
void SetDeploymentRank(int rank) |
Sets the deployment rank for inference. |
void Train() |
Sets the adapter to training mode (enables nested dropout). |
void TrainWithNestedDropout(Tensor<T>[] inputs, Tensor<T>[] targets, int epochs, T learningRate, Func<Tensor<T>, Tensor<T>, T> lossFunction) |
Trains the adapter with nested dropout across all active ranks, differentiating a caller-supplied tensor-valued loss. |
void TrainWithNestedDropout(Tensor<T>[] inputs, Tensor<T>[] targets, int epochs, T learningRate, Func<Tensor<T>, Tensor<T>, Tensor<T>> lossObjective) |
Trains the adapter with nested dropout across all active ranks, differentiating a caller-supplied tensor-valued loss. |
void UpdateParameters(T learningRate) |
Updates parameters for the base layer and the LoRA layer using cached gradients. |