ReLoRAAdapter<T>
Models & Types · AiDotNet.LoRA.Adapters
Restart LoRA (ReLoRA) adapter that periodically merges and restarts LoRA training for continual learning.
Inheritance: LayerBase<T> → LoRAAdapterBase<T> → ReLoRAAdapter<T>
For Beginners
ReLoRA is like having multiple rounds of LoRA training.
Imagine you’re fine-tuning a model on data that keeps changing:
- Round 1: Train LoRA on dataset A for 1000 steps
- Merge: Add the learned changes into the base model
- Restart: Reset LoRA matrices and train on dataset B for 1000 steps
- Merge: Add these new changes to the (already updated) base model
- Repeat…
Benefits:
- Continual learning: Can keep learning from new data indefinitely
- No catastrophic forgetting: Old knowledge is preserved in the base layer
- Parameter efficient: LoRA matrices stay small even after many restarts
- Flexible: Can adapt to distribution shifts and new tasks
How it works:
- Train normally with LoRA for N steps (restart interval)
- At step N: Merge LoRA weights → AccumulatedWeight += LoRA
- Reset LoRA matrices to zero (fresh start)
- Continue training for another N steps
- Repeat indefinitely
Use cases:
- Training on streaming data (news articles, user behavior, etc.)
- Adapting to distribution shifts over time
- Long-running training sessions that need checkpoints
- Multi-task learning with periodic task switches
Reference: “ReLoRA: High-Rank Training Through Low-Rank Updates” (2023)
https://arxiv.org/abs/2307.05695
How It Works
ReLoRA addresses the challenge of continual learning and long-running training by periodically:
- Merging the LoRA weights into the base layer (accumulating the adaptation)
- Resetting the LoRA matrices to restart training fresh
- Continuing training with a clean slate while preserving previous learning
This approach:
- Prevents catastrophic forgetting by accumulating adaptations into the base layer
- Allows continuous adaptation to new data without losing old knowledge
- Maintains parameter efficiency by resetting LoRA to small matrices
- Enables training on continuously evolving data streams
Example
using AiDotNet.LoRA;
using AiDotNet.LoRA.Adapters;
var adapter = new ReLoRAAdapter<double>(null, rank: 8, alpha: 8, freezeBaseLayer: true);
var config = new DefaultLoRAConfiguration<double>(rank: 8, alpha: 8, loraAdapter: adapter);
Console.WriteLine($"Configured ReLoRAAdapter (rank {config.Rank}).");
Constructors
| Constructor | Summary |
|---|---|
ReLoRAAdapter(ILayer<T> baseLayer, int rank, double alpha, int restartInterval, bool freezeBaseLayer, bool useWarmup, int warmupSteps) |
Initializes a new ReLoRA adapter with restart-based continual learning. |
Properties
| Property | Summary |
|---|---|
int CurrentStep { get; } |
Gets the current step within the current restart cycle. |
int RestartCount { get; } |
Gets the total number of restarts that have occurred. |
int RestartInterval { get; } |
Gets the number of training steps between restarts. |
Methods
| Method | Summary |
|---|---|
ILayer<T> MergeToOriginalLayer() |
Merges all accumulated adaptations into the base layer and returns the merged layer. |
Matrix<T> GetAccumulatedWeight() |
Gets a copy of the accumulated weight matrix. |
bool ShouldRestart() |
Checks if a restart should be performed based on the current step count. |
void ForceRestart() |
Manually triggers a restart (useful for checkpointing or manual control). |
void ResetState() |
Resets the internal state of all layers. |
void ResetStepCounter() |
Resets the step counter without performing a restart (useful for aligning with external training loops). |
void RestartLoRA() |
Performs the restart operation: merges current LoRA weights and reinitializes. |
void UpdateParameters(T learningRate) |
Updates parameters with optional warmup after restarts. |