FloraAdapter<T>

Models & Types · AiDotNet.LoRA.Adapters

Implements Flora (Low-Rank Adapters Are Secretly Gradient Compressors) adapter for memory-efficient fine-tuning.

Inheritance: LayerBase<T> → LoRAAdapterBase<T> → FloraAdapter<T>

How It Works

Flora reinterprets LoRA as a gradient compression mechanism and achieves high-rank updates through periodic resampling of projection matrices while maintaining sublinear space complexity for optimizer states.

Research Paper: “Flora: Low-Rank Adapters Are Secretly Gradient Compressors” by Yongchang Hao et al., ICML 2024. arXiv:2402.03293

Key Innovation: Unlike standard LoRA which restricts weight updates to a fixed low-rank subspace, Flora periodically resamples the projection matrices (A and B), allowing the effective rank of cumulative updates to grow over time. This achieves performance comparable to full-rank fine-tuning while maintaining the memory efficiency of LoRA.

Example

using AiDotNet.LoRA;
using AiDotNet.LoRA.Adapters;

var adapter = new FloraAdapter<double>(null, rank: 8, alpha: 8, freezeBaseLayer: true);
var config = new DefaultLoRAConfiguration<double>(rank: 8, alpha: 8, loraAdapter: adapter);
Console.WriteLine($"Configured FloraAdapter (rank {config.Rank}).");

Constructors

Constructor Summary
FloraAdapter(ILayer<T> baseLayer, int rank, double alpha, int resamplingInterval, double momentumDecay, double secondMomentDecay, bool useAdaptiveLearningRate, bool freezeBaseLayer, int seed)

Properties

Property Summary
int CurrentStep { get; }
int ResamplingInterval { get; }

Methods

Method Summary
ILayer<T> MergeToOriginalLayer()
void ResetState()
void UpdateParameters(T learningRate)