SequencePackingCollateFunction<T>
Models & Types · AiDotNet.Data.Text
Packs multiple variable-length sequences into fixed-length blocks for efficient LLM training.
Implements: ICollateFunction<Tensor<T>, ValueTuple<Tensor<T>, Tensor<T>>>
How It Works
Sequence packing concatenates multiple short sequences into a single block, reducing padding waste. Each block has a corresponding attention mask that prevents cross-sequence attention.
This is the technique used by GPT-3 and LLaMA training to maximize GPU utilization. Instead of padding every sequence to max_length, shorter sequences are concatenated until the block is full, with separator tokens between them.
Constructors
| Constructor | Summary |
|---|---|
SequencePackingCollateFunction(int blockSize, int padTokenId) |
Initializes a new sequence packing collate function. |
Methods
| Method | Summary |
|---|---|
ValueTuple<Tensor<T>, Tensor<T>> Collate(IReadOnlyList<Tensor<T>> samples) |
Packs multiple token sequences into fixed-length blocks. |