LoRAFAAdapter<T>
Models & Types · AiDotNet.LoRA.Adapters
LoRA-FA (LoRA with Frozen A matrix) adapter for parameter-efficient fine-tuning.
Inheritance: LayerBase<T> → LoRAAdapterBase<T> → LoRAFAAdapter<T>
For Beginners
LoRA-FA makes LoRA even more efficient!
Standard LoRA uses two small matrices (A and B) that both get trained:
- Matrix A: Compresses input (trained)
- Matrix B: Expands to output (trained)
LoRA-FA optimizes this further:
- Matrix A: Compresses input (frozen - never changes after initialization)
- Matrix B: Expands to output (trained - the only thing that learns)
Why freeze matrix A?
- Research shows matrix A can be randomly initialized and frozen without much performance loss
- This cuts trainable parameters in half (only matrix B is trained)
- Training is faster and uses less memory
- Perfect when you need maximum efficiency
Example parameter counts for a 1000×1000 layer with rank=8:
- Standard LoRA: 8,000 (A) + 8,000 (B) = 16,000 trainable parameters
- LoRA-FA: 0 (A frozen) + 8,000 (B) = 8,000 trainable parameters (50% reduction!)
When to use LoRA-FA:
- Memory is very limited
- Training speed is critical
- You can tolerate a small performance trade-off
- You’re working with very large models
How It Works
LoRA-FA is a variant of standard LoRA that freezes matrix A after random initialization and only trains matrix B. This provides approximately 50% parameter reduction compared to standard LoRA with minimal performance loss in most scenarios.
Example
using AiDotNet.LoRA;
using AiDotNet.LoRA.Adapters;
var adapter = new LoRAFAAdapter<double>(null, rank: 8, alpha: 8, freezeBaseLayer: true);
var config = new DefaultLoRAConfiguration<double>(rank: 8, alpha: 8, loraAdapter: adapter);
Console.WriteLine($"Configured LoRAFAAdapter (rank {config.Rank}).");
Constructors
| Constructor | Summary |
|---|---|
LoRAFAAdapter(ILayer<T> baseLayer, int rank, double alpha, bool freezeBaseLayer) |
Initializes a new LoRA-FA adapter wrapping an existing layer. |
Properties
| Property | Summary |
|---|---|
bool IsMatrixAFrozen { get; } |
Gets whether matrix A is frozen during training (always true for LoRA-FA). |
Methods
| Method | Summary |
|---|---|
ILayer<T> MergeToOriginalLayer() |
Merges the LoRA-FA adaptation into the base layer and returns the merged layer. |
void UpdateParameters(T learningRate) |
Updates parameters, but only for matrix B (matrix A remains frozen). |