TimestepDependentLora<T>
Models & Types · AiDotNet.Diffusion.StyleTransfer
Timestep-dependent low-rank adaptation: the mechanism of T-LoRA, in which the effective rank of the adapter update SHRINKS as the diffusion timestep rises.
Implements: IParameterChunkSource<T>, IParameterSource<T>
For Beginners
Teaching a picture generator a new object from a SINGLE photo usually makes it memorize that photo and lose its variety. It turns out the memorizing happens mostly during the noisy early stage of generation. So this hands the model fewer knobs to turn during that stage and all of them later, when it is only refining details.
How It Works
Soboleva, Alanov, Kuznetsov and Sobolev, “T-LoRA: Single Image Diffusion Model Customization Without Overfitting” (arXiv:2507.05964). The “T” is Timestep-Dependent, and the finding it rests on is empirical: “higher diffusion timesteps are more prone to overfitting than lower ones, necessitating a timestep-sensitive fine-tuning strategy.”
So the adapter is given less freedom exactly where overfitting happens. At high timesteps — heavy noise, where a single training image most easily dictates the output — only the leading directions of the update survive. At low timesteps, where the model is refining detail, the full rank is available.
The paper’s rank schedule (Section 3.2):
Note the FLOOR at r_min, not at one. The paper never strips the adapter down to a single
direction: even the most overfitting-prone timesteps keep half the rank. An earlier revision of
this class used ceil(r * (1 - t/T)) clamped to a minimum of one, which both used the wrong
interpolation and over-constrained the high-timestep end far past what the paper does.
Ortho-LoRA initialization (Section 3.1). The second contribution, “a weight parametrization
technique that ensures independence between adapter components”. Independence is what makes the
schedule meaningful — masking the tail of a set of CORRELATED directions removes no capacity,
because the survivors still span what was masked. The paper obtains it from the SVD of a random
matrix R ~ N(0, 1/r), taking the trailing (least-dominant) triplet:
Why there is a third trainable matrix. Standard LoRA gets “adapter is the identity at initialization” for free by setting B = 0. This paper deliberately does NOT: it wants a non-zero singular-value matrix S that training can move, so it recovers the identity property by SUBTRACTING the frozen initial product instead —
which is exactly zero while B = B_init, S = S_init, A = A_init and diverges from zero only
as training moves them. Trainable: A, B and S. Frozen: W and the three _init factors.
Constructors
| Constructor | Summary |
|---|---|
TimestepDependentLora(int rank, int inputDim, int outputDim, int totalTimesteps, Random random) |
Initializes the adapter using the paper’s Ortho-LoRA scheme. |
Properties
| Property | Summary |
|---|---|
Matrix<T> DownProjection { get; } |
Down-projection A, shape [rank, inputDim]. |
Matrix<T> UpProjection { get; } |
Up-projection B, shape [outputDim, rank]. |
Vector<T> SingularValues { get; } |
The diagonal singular-value factor S, length Rank. |
int MinRank { get; } |
Gets the floor of the rank schedule, r_min — 50% of Rank per the paper. |
int Rank { get; } |
Gets the adapter’s full rank R, available at timestep zero. |
int TotalTimesteps { get; } |
Gets the diffusion horizon T used to normalize the timestep. |
long ParameterCount { get; } |
The complete state length, including the frozen initialization triplet. |
Methods
| Method | Summary |
|---|---|
IEnumerable<ParameterChunk<T>> GetParameterStateChunks() |
Yields A, B, S followed by their frozen initial values, in checkpoint order. |
Tensor<T> EffectiveDeltaTransposed(int timestep) |
The adapter’s effective weight delta at a timestep, TRANSPOSED to [inputDim, outputDim] so it can be right-multiplied against a [tokens, inputDim] activation matrix. |
Vector<T> Apply(Vector<T> input, int timestep) |
Applies the adapter to a single vector: (B S M_t A - B_init S_init M_t A_init) x. |
Vector<T> GetParameters() |
Reads the same complete state described by GetParameterStateChunks. |
int EffectiveRank(int timestep) |
The effective rank at timestep timestep: full at t = 0, falling linearly to MinRank at the horizon. |
void CopyStateFrom(TimestepDependentLora<T> source) |
Copies the COMPLETE state of source — the trainable A, B and S and the frozen A_init, B_init and S_init — into this adapter. |
void InvalidateCache() |
Drops the cached delta. |
void SetParameters(Vector<T> parameters) |
Restores complete state without replacing the live optimizer tensors. |