RoSAAdapter<T>
Models & Types · AiDotNet.LoRA.Adapters
RoSA (Robust Adaptation) adapter for parameter-efficient fine-tuning with improved robustness to distribution shifts.
Inheritance: LayerBase<T> → LoRAAdapterBase<T> → RoSAAdapter<T>
For Beginners
RoSA is like LoRA with a safety net for unusual cases.
Think of it this way:
- Low-rank LoRA is like learning general rules (“most images of cats have pointed ears”)
- Sparse component is like remembering specific exceptions (“this one cat breed has round ears”)
- Together they make a robust model that handles both common and rare cases
Why RoSA is more robust:
- Low-rank component: Efficient for common patterns across domains
- Sparse component: Handles outliers and domain-specific quirks
- Result: Better performance when test data differs from training data
When to use RoSA over standard LoRA:
- When you expect distribution shifts (train on news, test on social media)
- When your data has outliers or rare patterns that matter
- When you need robustness more than absolute parameter efficiency
- When adapting to multiple related but distinct domains
Trade-offs vs standard LoRA:
- More robust to distribution shifts
- Better handles rare patterns
- More flexible adaptation
- Slightly more parameters (sparse component adds ~5-15%)
- Slightly more computation (extra sparse matrix multiply)
- Requires tuning sparsity ratio
How It Works
RoSA (Robust Adaptation) extends standard LoRA by combining two complementary components:
- Low-rank component (standard LoRA): Captures common, structured patterns in adaptations
- Sparse component: Captures specific, rare, or outlier patterns that low-rank cannot represent
Mathematical Formulation: Given input x and pre-trained weights W, RoSA computes:
- Low-rank component: L = (alpha/rank) * B * A * x
- Sparse component: S = W_sparse * x (where W_sparse is highly sparse)
- Final output: y = W*x + L + S
The sparse component is maintained through magnitude-based pruning, keeping only the most significant weights and zeroing out the rest. This creates a sparse matrix that captures specific patterns while remaining parameter-efficient.
Research Context: RoSA was introduced in January 2024 as a robust alternative to standard LoRA. The key insight is that low-rank approximations work well for common patterns but struggle with distribution shifts and rare patterns. By adding a sparse component, RoSA can capture outliers and domain-specific patterns without significantly increasing parameter count.
In experiments on domain adaptation tasks, RoSA showed:
- Better generalization to new domains (+5-10% over standard LoRA)
- More robust to distribution shifts
- Ability to capture both global patterns (low-rank) and local exceptions (sparse)
- Only modest increase in parameters (typically 5-15% more than pure LoRA)
Reference: “RoSA: Robust Adaptation through Sparse Regularization” January 2024
Example
using AiDotNet.LoRA;
using AiDotNet.LoRA.Adapters;
var adapter = new RoSAAdapter<double>(null, rank: 8, alpha: 8, freezeBaseLayer: true);
var config = new DefaultLoRAConfiguration<double>(rank: 8, alpha: 8, loraAdapter: adapter);
Console.WriteLine($"Configured RoSAAdapter (rank {config.Rank}).");
Constructors
| Constructor | Summary |
|---|---|
RoSAAdapter(ILayer<T> baseLayer, int rank, double alpha, double sparsityRatio, double sparseThreshold, bool freezeBaseLayer) |
Initializes a new RoSA adapter wrapping an existing layer. |
Properties
| Property | Summary |
|---|---|
double SparseThreshold { get; set; } |
Threshold for magnitude-based pruning of sparse weights. |
double SparsityRatio { get; set; } |
Target sparsity ratio (fraction of zeros in sparse component). |
Methods
| Method | Summary |
|---|---|
Dictionary<string, string> GetParameterRoles() |
Returns parameter roles for per-role optimizer configuration (e.g., weight decay exemption for biases). |
ILayer<T> MergeToOriginalLayer() |
Merges the RoSA adaptation into the base layer and returns the merged layer. |
IReadOnlyList<Tensor<T>> GetTrainableParameters() |
Returns all trainable parameter tensors marked with [TrainableParameter]. |
double GetSparsity() |
Gets the current sparsity of the sparse component. |
void PruneSparseWeights() |
Prunes sparse weights based on magnitude to maintain target sparsity. |
void ResetState() |
Resets the internal state of the adapter. |
void SetTrainableParameters(IReadOnlyList<Tensor<T>> parameters) |
Replaces trainable parameter tensors (e.g., with ParameterBuffer views). |
void UpdateParameters(T learningRate) |
Updates parameters using the specified learning rate. |
void ZeroGrad() |
Clears all gradient fields discovered by convention ({paramName}Gradient). |