StandardLoRAAdapter<T>
Models & Types · AiDotNet.LoRA.Adapters
Standard LoRA implementation (original LoRA algorithm).
Inheritance: LayerBase<T> → LoRAAdapterBase<T> → StandardLoRAAdapter<T>
For Beginners
This adapter lets you add LoRA to any layer type. Think of it like adding a “correction layer” that learns what adjustments are needed:
- The base layer keeps its original weights (optionally frozen)
- The LoRA layer learns a small correction
- The final output is: original_output + lora_correction
This is incredibly useful for fine-tuning pre-trained models:
- Load a pre-trained model with any layer type
- Wrap those layers with StandardLoRAAdapter
- Freeze the base layers
- Train only the small LoRA corrections
- Achieve similar results with 100x fewer trainable parameters!
Example: If you have a dense layer with 1000x1000 weights, wrapping it with rank=8 LoRA (frozen) reduces trainable parameters from 1,000,000 to just 16,000!
How It Works
The StandardLoRAAdapter wraps any layer and adds a LoRA layer in parallel. During forward pass, both the base layer and LoRA layer process the input, and their outputs are summed. The base layer’s parameters can be frozen while only the LoRA parameters are trained.
Example
using AiDotNet.LoRA;
using AiDotNet.LoRA.Adapters;
var adapter = new StandardLoRAAdapter<double>(null, rank: 8, alpha: 8, freezeBaseLayer: true);
var config = new DefaultLoRAConfiguration<double>(rank: 8, alpha: 8, loraAdapter: adapter);
Console.WriteLine($"Configured StandardLoRAAdapter (rank {config.Rank}).");
Constructors
| Constructor | Summary |
|---|---|
StandardLoRAAdapter(ILayer<T> baseLayer, int rank, double alpha, bool freezeBaseLayer) |
Initializes a new Standard LoRA adapter wrapping an existing layer. |
Methods
| Method | Summary |
|---|---|
ILayer<T> MergeToOriginalLayer() |
Merges the LoRA adaptation into the base layer and returns the merged layer. |