ChainLoRAAdapter<T>
Models & Types · AiDotNet.LoRA.Adapters
Chain-of-LoRA adapter that implements sequential composition of multiple LoRA adapters.
Inheritance: LayerBase<T> → LoRAAdapterBase<T> → ChainLoRAAdapter<T>
For Beginners
Imagine you’re learning a complex skill in stages:
- First, you learn the basics (Adapter 1)
- Then you practice and the basics become automatic (Merge)
- Next, you learn intermediate techniques on top of the basics (Adapter 2)
- Again, you practice until they’re automatic (Merge)
- Finally, you learn advanced skills building on everything before (Adapter 3)
Chain-of-LoRA works the same way: each adapter learns something new, then it’s consolidated into the model, and the next adapter can focus on the next refinement. This stepwise approach often achieves better results than trying to learn everything at once.
How It Works
Chain-of-LoRA (COLA) is an advanced LoRA technique that enables sequential composition of multiple LoRA adaptations through an iterative optimization framework. Unlike standard LoRA which applies a single low-rank adaptation, COLA builds a chain of adaptations where each adapter is trained, merged into the model, and then a new adapter is initialized for further refinement.
This approach bridges the performance gap between standard LoRA and full fine-tuning by employing residual learning principles. Each iteration in the chain adds incremental improvements to the model’s task-specific performance without incurring additional computational costs or memory overhead during inference.
**Key Concepts:**Sequential Adaptation: Chain-of-LoRA applies adaptations in sequence (Task A → Task B → Task C), where each stage builds upon the previous one. This is inspired by the Frank-Wolfe optimization algorithm, which makes greedy updates along the direction of maximum improvement.
Merge and Re-initialize: After training each LoRA adapter, the learned weights are merged back into the base layer, and a new LoRA adapter is initialized. This “tying a knot” process allows the model to consolidate learned knowledge before adding new adaptations.
Knowledge Preservation: By freezing the base layer and only training the LoRA components, the chain preserves previously learned knowledge while allowing new task-specific adaptations. Each adapter in the chain captures a specific aspect of the task or a refinement step.
Incremental Fine-tuning Pipeline: COLA enables continual learning scenarios where tasks are presented sequentially, and the model must adapt to new tasks while maintaining performance on previous ones.
Benefits of Chain-of-LoRA:
- Better Performance: Achieves up to 6.47% relative accuracy gain over standard LoRA
- No Extra Overhead: After merging, inference cost is identical to the base model
- Modular Adaptation: Each adapter can be trained, tested, and validated independently
- Catastrophic Forgetting Mitigation: Sequential merging helps preserve prior knowledge
- Task Chaining: Naturally supports multi-task learning and transfer learning scenarios
- Flexible Deployment: Can deploy the full chain or selected adapters as needed
Research Reference:
Based on “Chain of LoRA: Efficient Fine-tuning of Language Models via Residual Learning” (arXiv:2401.04151, January 2024). The paper demonstrates that sequential low-rank adaptations can significantly improve task performance compared to single-stage LoRA, especially on complex reasoning and multi-step tasks.
Usage Example:
Example
using AiDotNet.LoRA;
using AiDotNet.LoRA.Adapters;
var adapter = new ChainLoRAAdapter<double>(null, rank: 8, alpha: 8, freezeBaseLayer: true);
var config = new DefaultLoRAConfiguration<double>(rank: 8, alpha: 8, loraAdapter: adapter);
Console.WriteLine($"Configured ChainLoRAAdapter (rank {config.Rank}).");
Constructors
| Constructor | Summary |
|---|---|
ChainLoRAAdapter(ILayer<T> baseLayer, int rank, int chainLength, double alpha, bool freezeBaseLayer) |
Initializes a new Chain-of-LoRA adapter with the specified configuration. |
Properties
| Property | Summary |
|---|---|
IReadOnlyList<LoRALayer<T>> AdapterChain { get; } |
Gets the list of LoRA adapters in the chain. |
IReadOnlyList<bool> FrozenStatus { get; } |
Gets the frozen status of each adapter in the chain. |
int ActiveAdapterIndex { get; } |
Gets the index of the currently active adapter (0-based). |
int ChainLength { get; } |
Gets the total number of adapters in the chain. |
Methods
| Method | Summary |
|---|---|
ILayer<T> MergeToOriginalLayer() |
Merges all adapters in the chain into the original base layer. |
IReadOnlyList<ILayer<T>> GetSubLayers() |
|
int GetFrozenCount() |
Gets the number of adapters that have been frozen. |
int GetTrainableAdapterCount() |
Gets the number of adapters that are still trainable (not frozen). |
void FreezeActiveAdapter() |
Freezes the currently active adapter to prevent further training. |
void ResetState() |
Resets the internal state of the base layer and all adapters in the chain. |
void SetActiveAdapterIndex(int index) |
Sets which adapter in the chain is currently active for training. |
void UnfreezeAdapter(int index) |
Unfreezes a previously frozen adapter, making it trainable again. |
void UpdateParameters(T learningRate) |
Updates parameters using the specified learning rate. |