FeatureTokenizerLayer<T>

Layers · AiDotNet.NeuralNetworks.Layers

Feature tokenizer for tabular transformers: embeds each scalar input feature into its OWN learnable embedding vector, producing a [features, embedding] token sequence that a transformer encoder can attend over.

Inheritance: LayerBase<T> → FeatureTokenizerLayer<T>

Implements: IShapeContract

How It Works

Implements the numerical feature tokenizer of FT-Transformer (Gorishniy et al. 2021, “Revisiting Deep Learning Models for Tabular Data”), the same per-column embedding idea that underlies TabTransformer (Huang et al. 2020): token[f] = x[f] · W[f] + b[f], where each feature f has its own embedding row W[f] (shape [embedding]) and bias b[f].

This is critical for tabular models: a shared projection (a single Dense layer) maps the whole feature vector to ONE vector, so self-attention runs over a length-1 sequence (no attention) — and even a per-token Dense produces collinear tokens (all x_f · W) whose only difference is a scalar that LayerNorm then removes, collapsing distinct inputs to identical outputs. Per-feature embedding directions break that degeneracy and encode feature identity, so no separate positional embedding is needed.

The feature count is resolved lazily from the first forward input (like DenseLayer), so the layer adapts to the actual fed input width even when a model’s declared input size differs. Output is always a batched [batch, features, embedding] tensor (batch=1 for an unbatched [features] input) so the downstream encoder and head treat the feature axis unambiguously. Forward is expressed with broadcast Engine ops on the registered weight/bias tensors so the tape computes their gradients automatically.

Constructors

Constructor Summary
FeatureTokenizerLayer(int embeddingDim) Initializes a tokenizer whose feature count is resolved lazily on the first forward pass.
FeatureTokenizerLayer(int numFeatures, int embeddingDim) Initializes a tokenizer with an explicit feature count.

Properties

Property Summary
bool SupportsTraining { get; }

Methods

Method Summary
Dictionary<string, string> GetParameterRoles() Returns parameter roles for per-role optimizer configuration (e.g., weight decay exemption for biases).
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank)
IReadOnlyList<Tensor<T>> GetTrainableParameters() Returns all trainable parameter tensors marked with [TrainableParameter].
void ResetState()
void SetTrainableParameters(IReadOnlyList<Tensor<T>> parameters) Replaces trainable parameter tensors (e.g., with ParameterBuffer views).
void ZeroGrad() Clears all gradient fields discovered by convention ({paramName}Gradient).