StockformerAttention<T>

Models & Types · AiDotNet.Finance.Trading.Factors

Scaled dot-product attention as Stockformer defines it, over caller-owned projection layers.

For Beginners

Attention lets each position decide how much to draw from every other position, and those decisions are learned. Averaging everything equally — which is what this replaced — throws that choice away.

How It Works

Ma, Xue, Lu and Chen, arXiv:2401.06139. The paper’s general form, used by all three of its attention sites:

This replaces a fixed averaging operator. An earlier revision approximated the temporal stage with a lower-triangular causal mean, which produces plausible activations and gives the model almost NO capacity to learn which timesteps matter — the only trainable weights were in the projection afterwards. The visible symptom was that longer training made the objective WORSE (200-step loss above 50-step), because there was nothing for the extra steps to fit. Real attention is not an optimisation here, it is the mechanism.

Q, K and V are separate parameters so the same type serves self-attention (all three the same tensor, Eq. 7 and 10) and cross-attention (query from one band, key/value from the other, Eq. 11).

Constructors

Constructor Summary
StockformerAttention(int features, ILayer<T> query, ILayer<T> key, ILayer<T> value, bool causal) Creates an attention block.

Methods

Method Summary
Tensor<T> Apply(Tensor<T> q, Tensor<T> k, Tensor<T> v) Computes Att(q, k, v) over the LEADING axis of each input.