NOLAAdapter<T>
Models & Types · AiDotNet.LoRA.Adapters
Implements NOLA (Compressing LoRA using Linear Combination of Random Basis) adapter for extreme parameter efficiency.
Inheritance: LayerBase<T> → LoRAAdapterBase<T> → NOLAAdapter<T>
For Beginners
NOLA is an extreme compression technique for LoRA that makes fine-tuning even more efficient. Instead of storing and training two low-rank matrices (A and B), NOLA:
- Generates random “template” matrices on-the-fly (same random numbers every time due to fixed seed)
- Only trains small coefficients that control how much of each template to use
- Achieves 2-3x fewer parameters than LoRA while maintaining performance
Think of it like this:
- Traditional LoRA: You have 100 adjustable knobs (parameters)
- NOLA: You have 5 master controls that blend pre-defined settings
Key innovations:
- Memory efficiency: Random basis matrices are discarded after use and regenerated when needed
- Parameter efficiency: Only coefficients are trained, not full matrices
- Performance: Achieves similar or better results than LoRA with far fewer parameters
Example compression (1000x1000 layer, rank=8):
- LoRA: 16,000 parameters (1000×8 + 8×1000)
- NOLA with 100 basis: 200 parameters (100 coefficients for A + 100 for B) - 80x reduction!
On LLaMA-2 70B, NOLA achieves 20x compression over LoRA with no accuracy loss.
How It Works
NOLA overcomes the rank-one lower bound in traditional LoRA by re-parameterizing the low-rank matrices using linear combinations of randomly generated basis matrices. Instead of optimizing the full low-rank matrices A and B, NOLA:
- Generates fixed random basis matrices using a deterministic seed
- Optimizes only scalar coefficients that linearly combine these basis matrices
- Regenerates basis matrices during forward/backward passes to minimize memory usage
This decouples the number of trainable parameters from both the choice of rank and the network architecture, achieving compression ratios of 20x over standard LoRA without accuracy degradation.
Reference: NOLA: Compressing LoRA using Linear Combination of Random Basis
(Koohpayegani et al., ICLR 2024) - https://arxiv.org/abs/2310.02556
Constructors
| Constructor | Summary |
|---|---|
NOLAAdapter(ILayer<T> baseLayer, int rank, int numBasis, double alpha, int seed, bool freezeBaseLayer) |
Initializes a new NOLA adapter with the specified parameters. |
Properties
| Property | Summary |
|---|---|
double CompressionRatio { get; } |
Gets the compression ratio compared to standard LoRA. |
int NumBasis { get; } |
Gets the number of basis matrices used for compression. |
Methods
| Method | Summary |
|---|---|
Dictionary<string, string> GetParameterRoles() |
Returns parameter roles for per-role optimizer configuration (e.g., weight decay exemption for biases). |
ILayer<T> MergeToOriginalLayer() |
Merges the NOLA adaptation into the base layer and returns the merged layer. |
IReadOnlyList<Tensor<T>> GetTrainableParameters() |
Returns all trainable parameter tensors marked with [TrainableParameter]. |
Vector<T> GetCoefficientsA() |
Gets the current coefficient values for matrix A (for inspection). |
Vector<T> GetCoefficientsB() |
Gets the current coefficient values for matrix B (for inspection). |
void ResetState() |
Resets the internal state of the adapter. |
void SetTrainableParameters(IReadOnlyList<Tensor<T>> parameters) |
Replaces trainable parameter tensors (e.g., with ParameterBuffer views). |
void UpdateParameters(T learningRate) |
Updates parameters using the specified learning rate. |
void ZeroGrad() |
Clears all gradient fields discovered by convention ({paramName}Gradient). |