TensorParallelInference
Helpers & Utilities · AiDotNet.DistributedTraining
Runs a tensor-parallel forward pass for inference (serving), executing all tensor-parallel ranks so their collective operations (column-parallel gather, row-parallel all-reduce) rendezvous and combine into the same fully-reduced output every rank sees.
For Beginners
A very large model may not fit (or run fast enough) on a single GPU. Tensor parallelism splits each layer’s weights across several GPUs so they share the work on every token. This helper runs all those splits together and stitches the answer back into exactly what the whole model would have produced.
How It Works
Tensor parallelism shards each linear layer’s weights across ranks (Megatron style): a
ColumnParallelLinear owns a slice of the output columns, and a following
RowParallelLinear owns the matching input rows and all-reduces the partial sums.
This runner drives every rank’s forward and returns the combined result — the model output is identical to
running the un-sharded model, but the per-rank compute and weight memory are divided across the ranks.
ForwardInProcess simulates the ranks in ONE process (one task per rank over an
InMemoryCommunicationBackend), which is how single-node multi-device / test scenarios run
and how the sharded-equals-unsharded equivalence is verified. A real multi-node deployment instead runs one
process per rank against a networked backend (MPI/NCCL/Gloo) and calls the same layer forwards.
Methods
| Method | Summary |
|---|---|
Tensor<T> ForwardInProcess(int worldSize, Tensor<T> input, Func<ICommunicationBackend<T>, IReadOnlyList<ILayer<T>>> buildRank) |
Runs a tensor-parallel forward across worldSize ranks in-process (one task per rank over a shared InMemoryCommunicationBackend) and returns the combined output. |