HRAAdapter<T>

Models & Types · AiDotNet.LoRA.Adapters

HRA (Hybrid Rank Adaptation) adapter that combines low-rank and full-rank updates for optimal parameter efficiency.

Inheritance: LayerBase<T> → LoRAAdapterBase<T> → HRAAdapter<T>

For Beginners

HRA is like having two tools instead of one:

Standard LoRA problem:

  • Uses only low-rank updates (compressed, efficient)
  • Some parameters need precise full-rank updates
  • Full fine-tuning is too expensive
  • Need something in between

HRA solution:

  • Most parameters use low-rank updates (efficient, covers 95% of needs)
  • Critical parameters get full-rank updates (precise, covers remaining 5%)
  • Automatically learns which parameters are critical
  • Best quality with minimal parameter overhead

Analogy: Think of home renovation:

  • Low-rank updates: Paint the walls (cheap, covers large area, good enough)
  • Full-rank updates: Replace key structural beams (expensive, small area, critical)
  • HRA: Do both where appropriate for best results

How it works:

  1. Start with LoRA-style low-rank matrices (B * A)
  2. Add sparse full-rank updates for most important parameters
  3. Track importance scores during training
  4. Allocate parameter budget optimally between low-rank and sparse full-rank

Benefits:

  • Better quality than pure LoRA (full-rank updates where needed)
  • More efficient than full fine-tuning (most updates are low-rank)
  • Adaptive: learns which parameters need full-rank updates
  • Flexible: adjustable sparsity budget for full-rank component

Use cases:

  • Tasks where LoRA quality is not quite sufficient
  • Fine-tuning with specific architectural bottlenecks
  • When you have slightly more parameter budget than LoRA but much less than full fine-tuning
  • Domains where certain parameters are known to be critical

Example parameter comparison for a 1000x1000 layer:

  • Full fine-tuning: 1,000,000 parameters
  • Standard LoRA (rank=8): 16,000 parameters (98.4% reduction)
  • HRA (rank=8, 1% sparsity): 26,000 parameters (97.4% reduction, better quality)

Reference: Based on “Hybrid Rank Adaptation” research combining low-rank and sparse full-rank approaches

How It Works

HRA addresses a key limitation of standard LoRA: while low-rank updates are efficient, some parameters benefit from full-rank updates. HRA uses a hybrid approach:

  • Dense low-rank updates for most parameters (efficient, like LoRA)
  • Sparse full-rank updates for critical parameters (precise, targeted)
  • Importance-based allocation between the two components

The forward computation is: output = base_layer(input) + low_rank(input) + sparse_full_rank(input) where the hybrid allocation provides the best of both worlds.

Example

using AiDotNet.LoRA;
using AiDotNet.LoRA.Adapters;

var adapter = new HRAAdapter<double>(null, rank: 8, alpha: 8, freezeBaseLayer: true);
var config = new DefaultLoRAConfiguration<double>(rank: 8, alpha: 8, loraAdapter: adapter);
Console.WriteLine($"Configured HRAAdapter (rank {config.Rank}).");

Constructors

Constructor Summary
HRAAdapter(ILayer<T> baseLayer, int rank, double sparsityRatio, double alpha, bool freezeBaseLayer, int importanceUpdateInterval, double importanceEMA, bool useDynamicAllocation) Initializes a new HRA adapter with hybrid low-rank and sparse full-rank updates.

Properties

Property Summary
double SparsityRatio { get; } Gets the current sparsity ratio.
int ActiveSparseParams { get; } Gets the number of active sparse full-rank parameters.
int MaxSparseParams { get; } Gets the maximum allowed sparse parameters.

Methods

Method Summary
Dictionary<ValueTuple<int, int>, T> GetSparseUpdates() Gets the positions and values of current sparse full-rank updates.
Dictionary<string, string> GetParameterRoles() Returns parameter roles for per-role optimizer configuration (e.g., weight decay exemption for biases).
ILayer<T> MergeToOriginalLayer() Merges the HRA adaptation into the base layer and returns the merged layer.
IReadOnlyList<Tensor<T>> GetTrainableParameters() Returns all trainable parameter tensors marked with [TrainableParameter].
IReadOnlyList<ValueTuple<string, Tensor<T>>> GetRegisteredBuffers()
Matrix<T> GetParameterImportance() Gets a copy of the current parameter importance matrix.
void SetTrainableParameters(IReadOnlyList<Tensor<T>> parameters) Replaces trainable parameter tensors (e.g., with ParameterBuffer views).
void UpdateParameters(T learningRate) Updates parameters using the specified learning rate.
void ZeroGrad() Clears all gradient fields discovered by convention ({paramName}Gradient).