LowRankFactorizationCompression<T>
Models & Types · AiDotNet.ModelCompression
Implements Low-Rank Factorization compression using SVD-like decomposition.
Inheritance: ModelCompressionBase<T> → LowRankFactorizationCompression<T>
For Beginners
Low-Rank Factorization is like summarizing a book.
The concept:
- A weight matrix might be 1000×1000 = 1,000,000 parameters
- But the actual “information content” might be much smaller
- We can approximate it as: W ≈ A × B where A is 1000×50 and B is 50×1000
- Now we only store: 50,000 + 50,000 = 100,000 parameters (10x compression!)
How it works:
- Treat the weight vector as a matrix (reshape it)
- Perform approximate factorization (similar to SVD)
- Keep only the top-k singular values/vectors
- Store the factored matrices instead of the original
Benefits:
- Compression ratio is controlled by the rank k
- Works especially well for fully-connected layers
- Maintains smoothness in the weight space
Trade-offs:
- Need to choose the rank k (compression vs accuracy trade-off)
- Works best when weights have inherent low-rank structure
How It Works
Low-Rank Factorization approximates weight matrices by decomposing them into products of smaller matrices. This is based on the observation that many neural network weight matrices are approximately low-rank, meaning they can be represented with fewer parameters.
Constructors
| Constructor | Summary |
|---|---|
LowRankFactorizationCompression(int targetRank, double energyThreshold, int maxIterations, double tolerance) |
Initializes a new instance of the LowRankFactorizationCompression class. |
Methods
| Method | Summary |
|---|---|
ValueTuple<Vector<T>, ICompressionMetadata<T>> Compress(Vector<T> weights) |
Compresses weights using low-rank factorization. |
Vector<T> Decompress(Vector<T> compressedWeights, ICompressionMetadata<T> metadata) |
Decompresses by reconstructing from U, S, V factors. |
long GetCompressedSize(Vector<T> compressedWeights, ICompressionMetadata<T> metadata) |
Gets the compressed size. |