SequencePackingCollateFunction<T>

Models & Types · AiDotNet.Data.Text

Packs multiple variable-length sequences into fixed-length blocks for efficient LLM training.

Implements: ICollateFunction<Tensor<T>, ValueTuple<Tensor<T>, Tensor<T>>>

How It Works

Sequence packing concatenates multiple short sequences into a single block, reducing padding waste. Each block has a corresponding attention mask that prevents cross-sequence attention.

This is the technique used by GPT-3 and LLaMA training to maximize GPU utilization. Instead of padding every sequence to max_length, shorter sequences are concatenated until the block is full, with separator tokens between them.

Constructors

Constructor Summary
SequencePackingCollateFunction(int blockSize, int padTokenId) Initializes a new sequence packing collate function.

Methods

Method Summary
ValueTuple<Tensor<T>, Tensor<T>> Collate(IReadOnlyList<Tensor<T>> samples) Packs multiple token sequences into fixed-length blocks.