TabRBase<T>
Base Classes · AiDotNet.NeuralNetworks.Tabular
Base implementation of TabR, a retrieval-augmented model for tabular data.
Implements: IParameterSource<T>
Known derived types: TabRClassifier<T>, TabRRegression<T>
For Beginners
TabR is like having a photographic memory for training data.
Architecture overview:
- Feature Encoder: MLP that converts raw features to embeddings
- Retrieval Index: Stores embeddings of all training samples
- K-NN Search: Finds the K most similar training samples
- Context Encoder: Uses attention to aggregate neighbor information
- Prediction Head: Makes final prediction using combined information
Why this approach works:
- Tabular data often has “local” structure (similar inputs → similar outputs)
- Neural networks alone may struggle with rare patterns
- Retrieval provides explicit “memory” of similar past examples
- Attention learns which neighbors are most relevant
Think of it as:
- Traditional k-NN: Just average neighbors (simple but limited)
- TabR: Learn to encode, then learn to attend to neighbors (powerful)
How It Works
TabR combines neural networks with instance-based learning. It encodes features, retrieves similar training samples, and uses attention to aggregate neighbor information for making predictions.
The architecture is the reference yandex-research/tabular-dl-tabr``bin/tabr.py model. The encoder
is x~ = Linear(d_in, d_main)(x) followed by EncoderNumBlocks residual blocks
x~ = x~ + Dropout(dropout1)(Linear(d_block, d_main)(Dropout(dropout0)(Activation(Linear(d_main, d_block)(Norm(x~)))))),
the first without its norm. The retrieval module follows Gorishniy et al. (2024), section 3: the target and
every candidate are encoded, k = W_K(x~) (of Norm(x~) under mixer normalization); the m
candidates with the smallest ||k - k_i||^2 are retrieved; their weights are
softmax(-||k - k_i||^2), dropped by context_dropout; each neighbour contributes the value
W_Y(y_i) + T(k - k_i), with T = LinearWithoutBias(Dropout(Activation(Linear(.)))) and
W_Y the label embedding; and the predictor runs PredictorNumBlocks
pre-normalized residual blocks over x~ + sum_i w_i V_i, then the head Activation(Norm(.)) whose
final linear map a subclass adds. Candidates are re-encoded on every pass, so the encoder and W_K also
learn through the retrieved neighbours. The neighbour labels enter only when the index was built with labels
and IncludeNeighborTargets is on.
Properties
| Property | Summary |
|---|---|
bool IsIndexBuilt { get; } |
Gets whether the retrieval index has been built. |
int EmbeddingDimension { get; } |
Gets the embedding dimension. |
int NumIndexedSamples { get; } |
Gets the number of samples in the retrieval index. |
int NumNeighbors { get; } |
Gets the number of nearest neighbors to retrieve. |
long ParameterCount { get; } |
Methods
| Method | Summary |
|---|---|
Matrix<int> GetRetrievedNeighborIndices() |
Gets the retrieved neighbor indices from the last forward pass. |
Tensor<T> GetAttentionWeights() |
Gets the attention weights from the last forward pass. |
Vector<T> GetParameters() |
Gets all trainable parameters as a single vector. |
void BuildIndex(Tensor<T> features) |
Builds the retrieval index from training data. |
void ResetState() |
Resets internal state. |
void SetParameters(Vector<T> parameters) |
Restores every parameter, in the order GetParameters emitted them. |
void UpdateParameters(T learningRate) |
Updates all parameters using the calculated gradients. |