QALoRAAdapter<T>
Models & Types · AiDotNet.LoRA.Adapters
Quantization-Aware LoRA (QA-LoRA) adapter that combines parameter-efficient fine-tuning with group-wise quantization awareness.
Inheritance: LayerBase<T> → LoRAAdapterBase<T> → QALoRAAdapter<T>
For Beginners
QA-LoRA solves a critical problem when deploying models to resource-constrained devices.
The Problem:
- Modern neural networks use high-precision numbers (32-bit floats)
- Mobile/edge devices need lower precision (4-bit or 8-bit integers) for speed and memory
- Converting after training (post-training quantization) often loses accuracy
QA-LoRA’s Solution:
- Simulates low-precision during training (quantization-aware training)
- Learns to compensate for quantization errors
- Uses LoRA for parameter efficiency (only trains the adaptation, not full model)
- Applies group-wise quantization (groups of weights share scaling factors)
Key Concepts:
- Quantization: Converting high-precision numbers to low-precision
Example: 32-bit float 0.7234 → 4-bit integer 11 (range 0-15)
- Group-wise Quantization: Instead of one scale for all weights, weights are divided into groups,
each with its own scale. This preserves more information. Example: 64 weights → 4 groups of 16 weights each, each group has its own scale
- Quantization-Aware Training: During training, simulate quantization in forward pass:
- Convert weights to low-precision (quantize)
- Immediately convert back to high-precision (dequantize)
- Use these “quantized” values for computation
- Gradients learn to compensate for the quantization noise
- Straight-Through Estimator (STE): During backward pass, treat quantization as identity
- Forward: y = quantize(x)
- Backward: ∂y/∂x ≈ 1 (gradient flows through unchanged)
- This allows gradients to update the full-precision weights
Parameters:
- QuantizationBits: How many bits to use (4-bit, 8-bit, etc.)
- GroupSize: How many weights per quantization group (e.g., 64, 128)
- Smaller GroupSize = more scales = better accuracy but more overhead
- Larger GroupSize = fewer scales = more efficient but less accurate
Example Workflow:
- Training: Forward pass uses simulated 4-bit quantization
- Gradients: Backward pass learns to work around quantization errors
- Deployment: Actually quantize the merged weights to 4-bit for inference
- Result: Much better accuracy than quantizing after training
Research Context:
- QLoRA (May 2023): Introduced efficient 4-bit quantization for LoRA
- QA-LoRA: Extends this with quantization-aware training for better results
- Typical improvement: 1-3% accuracy gain over post-training quantization
Use Cases:
- Deploying large language models on mobile devices
- Edge AI applications with strict memory constraints
- Reducing model size while maintaining accuracy
- Fine-tuning for deployment on specific hardware (TPUs, specialized accelerators)
How It Works
QA-LoRA extends standard LoRA by being aware of quantization during training. This allows the adapter to learn compensations for quantization errors, resulting in better final accuracy compared to post-training quantization approaches. The key innovation is simulating quantization during the forward pass so that gradients account for quantization effects.
Constructors
| Constructor | Summary |
|---|---|
QALoRAAdapter(ILayer<T> baseLayer, int rank, int quantizationBits, int groupSize, double alpha, bool freezeBaseLayer) |
Initializes a new QA-LoRA adapter with quantization awareness. |
Properties
| Property | Summary |
|---|---|
bool QuantizationEnabled { get; set; } |
Gets or sets whether quantization simulation is enabled during forward/backward passes. |
int GroupSize { get; set; } |
Gets or sets the group size for group-wise quantization. |
int QuantizationBits { get; set; } |
Gets or sets the number of bits used for quantization. |
Methods
| Method | Summary |
|---|---|
ILayer<T> MergeToOriginalLayer() |
Merges the LoRA adaptation into the base layer and returns a quantized merged layer. |
ValueTuple<double, double, int> GetQuantizationStats() |
Gets statistics about quantization for the current LoRA parameters. |