DbrxTensorSource

Models & Types · AiDotNet.ModelLoading.Pretrained

Adapts a DBRX checkpoint’s tensors to the Hugging Face parameter names the pretrained builders expect. DBRX names blocks transformer.blocks.{i}.*, fuses q/k/v into a single attn.Wqkv, and stacks all experts’ gate/up/down into single tensors (ffn.experts.mlp.w1/v1/w2); this source renames them, slices each expert out, and transposes the down projection so the standard loaders can consume them.

Implements: INamedTensorSource

How It Works

The stacked expert tensors are [num_experts · ffn, hidden] row-major, so expert e occupies the element range [e·ffn·hidden, (e+1)·ffn·hidden). w1/v1 are [out=ffn, in=hidden] (exposed as-is for the usual [out,in]→[in,out] load transpose); w2 is stored [ffn, hidden] (i.e. already [in,out]), so it is transposed to [hidden, ffn] here to match the down-projection convention. The fused Wqkv is exposed as self_attn.qkv_proj.weight for a wrapping FusedProjectionSource to split.

Constructors

Constructor Summary
DbrxTensorSource(INamedTensorSource inner, HuggingFaceConfig config) Wraps a raw DBRX tensor source, deriving the Hugging Face name mapping from config.

Properties

Property Summary
IReadOnlyCollection<string> TensorNames { get; }

Methods

Method Summary
double[] ReadAsDouble(string name)