OperationType
Enums · AiDotNet.Enums
Represents different operation types in computation graphs for JIT compilation and automatic differentiation.
For Beginners
Operation types identify mathematical operations performed on tensors in neural networks.
When building a computation graph, each operation (like adding two tensors or applying an activation function) needs to be identified so that:
- The JIT compiler can optimize the code
- The automatic differentiation system can compute gradients correctly
- The system can analyze and transform the computation graph
This enum provides type-safe identification of operations, preventing typos and enabling better tooling support.
Fields
| Field | Summary |
|---|---|
OperationType Abs |
Element-wise absolute value - |x| for each element. |
OperationType Activation |
Generic activation function application. |
OperationType AdaptivePooling |
Adaptive pooling. |
OperationType Add |
Element-wise addition of two tensors. |
OperationType AffineGrid |
Affine grid generation for spatial transformers. |
OperationType And |
Logical AND. |
OperationType AnomalyScore |
Anomaly score computation. |
OperationType Attention |
Generic attention mechanism operation. |
OperationType AveragePooling |
Average pooling operation. |
OperationType AvgPool2D |
2D average pooling. |
OperationType BatchNorm |
Batch normalization. |
OperationType BatchNormalization |
Batch normalization. |
OperationType BentIdentity |
Bent Identity - (sqrt(x² + 1) - 1) / 2 + x, smooth alternative to ReLU. |
OperationType Broadcast |
Broadcast operation - expands tensor dimensions to match target shape. |
OperationType CELU |
Continuously Differentiable ELU - max(0, x) + min(0, α * (exp(x/α) - 1)). |
OperationType CRFForward |
CRF forward algorithm for sequence labeling. |
OperationType Cast |
Type cast operation. |
OperationType CliffordInnerProduct |
Inner (contraction) product of multivectors. |
OperationType Clip |
Clip values to range. |
OperationType ComplexMatMul |
Complex matrix multiplication for quantum operations. |
OperationType ComplexMultiply |
Element-wise complex multiplication. |
OperationType Concat |
Concatenate multiple tensors along an axis. |
OperationType Constant |
Constant node - represents a constant value that doesn’t require gradients. |
OperationType Conv2D |
2D convolution operation. |
OperationType ConvTranspose2D |
2D transposed convolution (deconvolution). |
OperationType Convolution |
General convolution operation. |
OperationType Convolution2D |
2D convolution operation. |
OperationType Convolution3D |
3D convolution operation. |
OperationType Crop |
Crop tensor by removing border elements. |
OperationType CrossAttention |
Cross-attention operation. |
OperationType Custom |
Custom user-defined operation for extensibility. |
OperationType Deconvolution |
Deconvolution (transposed convolution) operation. |
OperationType DeformableConv2D |
2D deformable convolution with learnable offsets and optional modulation. |
OperationType Dense |
Dense (fully connected) layer. |
OperationType DepthwiseConv2D |
2D depthwise convolution. |
OperationType DepthwiseConvolution |
Depthwise convolution operation. |
OperationType DilatedConv2D |
2D dilated (atrous) convolution. |
OperationType DilatedConvolution |
Dilated convolution operation. |
OperationType Divide |
Element-wise division of two tensors. |
OperationType DropPath |
DropPath regularization. |
OperationType Dropout |
Dropout regularization operation - randomly zeros elements during training. |
OperationType ELU |
Exponential Linear Unit - ELU(x) = x if x > 0, alpha * (exp(x) - 1) otherwise. |
OperationType Embedding |
Embedding lookup operation. |
OperationType Equal |
Element-wise equality. |
OperationType Exp |
Element-wise exponential function - e^x for each element. |
OperationType Expand |
Expand tensor dimensions. |
OperationType FakeQuantization |
Fake quantization operation with Straight-Through Estimator (STE) for differentiable quantization. |
OperationType Flatten |
Flatten tensor to 1D. |
OperationType FullyConnected |
Fully connected layer. |
OperationType FusedAddReLU |
Fused addition + ReLU. |
OperationType FusedConvBatchNorm |
Fused convolution + batch normalization. |
OperationType FusedConvBatchNormReLU |
Fused Conv + BatchNorm + ReLU. |
OperationType FusedLayerNormAttention |
Fused LayerNorm + Attention. |
OperationType FusedLinearReLU |
Fused linear layer with ReLU (MatMul + Add + ReLU). |
OperationType FusedMatMulAdd |
Fused matrix multiplication + addition (MatMul + Add). |
OperationType FusedMatMulBias |
Fused MatMul + Bias. |
OperationType FusedMatMulBiasGELU |
Fused MatMul + Bias + GELU. |
OperationType FusedMatMulBiasReLU |
Fused MatMul + Bias + ReLU. |
OperationType FusedMultiHeadAttention |
Fused MultiHead Attention. |
OperationType GELU |
Gaussian Error Linear Unit - x * Φ(x) where Φ is standard normal CDF. |
OperationType GRU |
GRU recurrent layer. |
OperationType GRUCell |
GRU cell operation for recurrent networks. |
OperationType Gather |
Gather operation - selects elements from a tensor using indices. |
OperationType Gaussian |
Gaussian activation - exp(-x²), bell-shaped response curve. |
OperationType Gemm |
General Matrix Multiplication. |
OperationType GeometricProduct |
Geometric product of multivectors. |
OperationType GlobalAveragePooling |
Global average pooling. |
OperationType GlobalMaxPooling |
Global max pooling. |
OperationType GradeProject |
Grade projection of multivectors. |
OperationType GraphConv |
Graph convolutional operation for GNNs. |
OperationType Greater |
Element-wise greater than. |
OperationType GreaterOrEqual |
Element-wise greater or equal. |
OperationType GridSample |
Grid sampling for spatial transformers. |
OperationType GroupNormalization |
Group normalization. |
OperationType GumbelSoftmax |
Gumbel-Softmax for differentiable discrete sampling (used in stochastic layers). |
OperationType HardSigmoid |
Hard Sigmoid - piecewise linear approximation of sigmoid: clip((x + 1) / 2, 0, 1). |
OperationType HardTanh |
Hard Tanh - piecewise linear approximation of tanh: clip(x, -1, 1). |
OperationType HierarchicalSoftmax |
Hierarchical Softmax - tree-based efficient softmax for large vocabularies. |
OperationType HyperboloidDistance |
Hyperboloid distance metric. |
OperationType ISRU |
Inverse Square Root Unit - x / sqrt(1 + alpha * x²). |
OperationType Input |
Input node - represents a variable or parameter in the computation graph. |
OperationType InstanceNormalization |
Instance normalization. |
OperationType LSTM |
LSTM recurrent layer. |
OperationType LSTMCell |
LSTM cell operation for recurrent networks. |
OperationType LayerNorm |
Layer normalization. |
OperationType LayerNormalization |
Layer normalization. |
OperationType LeakyReLU |
Leaky Rectified Linear Unit - max(alpha * x, x) where alpha is typically 0.01. |
OperationType LeakyStateUpdate |
Leaky state update for reservoir/echo state networks. |
OperationType Less |
Element-wise less than. |
OperationType LessOrEqual |
Element-wise less or equal. |
OperationType LiSHT |
Linearly Scaled Hyperbolic Tangent - x * tanh(x). |
OperationType LocallyConnectedConv2D |
2D locally connected convolution. |
OperationType Log |
Element-wise natural logarithm. |
OperationType LogSoftmax |
Log-Softmax - log(softmax(x)), numerically stable for cross-entropy loss. |
OperationType LogSoftmin |
Log-Softmin - log(softmin(x)) = log(softmax(-x)). |
OperationType MatMul |
Matrix multiplication (not element-wise). |
OperationType MaxPool2D |
2D max pooling. |
OperationType MaxPool3D |
3D max pooling. |
OperationType MaxPooling |
Max pooling operation. |
OperationType Maxout |
Maxout activation - maximum over multiple linear pieces. |
OperationType Mean |
Mean operation (reduces all dimensions). |
OperationType Mish |
Mish activation - x * tanh(softplus(x)). |
OperationType MobiusAdd |
Mobius addition in Poincare ball model. |
OperationType MultiHeadAttention |
Multi-head attention operation. |
OperationType Multiply |
Element-wise multiplication (Hadamard product) of two tensors. |
OperationType MultivectorAdd |
Multivector addition. |
OperationType MultivectorReverse |
Multivector reverse operation. |
OperationType Negate |
Element-wise negation - multiplies each element by -1. |
OperationType Norm |
L2 norm computation along an axis - sqrt(sum(x²)). |
OperationType Not |
Logical NOT. |
OperationType OctonionAdd |
Octonion addition. |
OperationType OctonionConjugate |
Octonion conjugation. |
OperationType OctonionMatMul |
Octonion matrix multiplication for neural networks. |
OperationType OctonionMultiply |
Octonion multiplication (non-associative). |
OperationType Or |
Logical OR. |
OperationType Output |
Output node in computation graph. |
OperationType PReLU |
Parametric Rectified Linear Unit - max(0, x) + alpha * min(0, x) where alpha is learned. |
OperationType Pad |
Pad tensor with values. |
OperationType Permute |
Permute tensor dimensions (general transpose). |
OperationType PixelShuffle |
Pixel shuffle operation for upsampling. |
OperationType PoincareDistance |
Poincare ball distance metric. |
OperationType PoincareExpMap |
Poincare exponential map. |
OperationType PoincareLogMap |
Poincare logarithmic map. |
OperationType PositionalEncoding |
Positional encoding for transformers. |
OperationType Power |
Element-wise power operation - raises each element to a specified exponent. |
OperationType RBFKernel |
RBF (Radial Basis Function) kernel operation. |
OperationType RNN |
Basic RNN layer. |
OperationType RReLU |
Randomized Leaky ReLU - LeakyReLU with random alpha during training. |
OperationType ReLU |
Rectified Linear Unit - max(0, x). |
OperationType ReduceLogVariance |
Log-variance reduction along specified axes. |
OperationType ReduceMax |
Maximum value reduction along specified axes. |
OperationType ReduceMean |
Mean reduction along specified axes. |
OperationType ReduceMin |
Minimum value reduction. |
OperationType ReduceSum |
Sum reduction along specified axes. |
OperationType Reshape |
Reshape tensor to new dimensions. |
OperationType SELU |
Scaled Exponential Linear Unit - self-normalizing activation with fixed lambda and alpha. |
OperationType SQRBF |
Square Radial Basis Function - smooth bell-shaped activation. |
OperationType ScaledDotProductAttention |
Scaled dot-product attention. |
OperationType ScaledTanh |
Scaled Tanh - parameterized tanh with adjustable steepness β. |
OperationType Scatter |
Scatter values to indices. |
OperationType Se3Exp |
SE(3) exponential map. |
OperationType Se3Log |
SE(3) logarithmic map. |
OperationType SelfAttention |
Self-attention operation. |
OperationType Sigmoid |
Sigmoid activation - 1 / (1 + e^(-x)). |
OperationType Sign |
Sign function with surrogate gradient for training - returns -1, 0, or 1. |
OperationType Slice |
Slice tensor along an axis - extract a portion with optional stride. |
OperationType So3Exp |
SO(3) exponential map. |
OperationType So3Log |
SO(3) logarithmic map. |
OperationType SoftKNN |
Soft K-Nearest Neighbors operation for differentiable instance-based learning. |
OperationType SoftLocallyWeighted |
Soft locally-weighted regression operation for differentiable instance-based learning. |
OperationType SoftPlus |
SoftPlus activation - ln(1 + e^x), smooth approximation of ReLU. |
OperationType SoftSign |
SoftSign activation - x / (1 + |x|), alternative to tanh with polynomial tails. |
OperationType SoftSplit |
Soft split operation for differentiable decision trees. |
OperationType Softmax |
Softmax activation - converts logits to probability distribution. |
OperationType Softmin |
Softmin - softmax(-x), assigns higher probability to lower values. |
OperationType SpMM |
Sparse matrix-matrix multiplication. |
OperationType SpMV |
Sparse matrix-vector multiplication. |
OperationType SparseGather |
Sparse gather operation. |
OperationType SparseScatter |
Sparse scatter operation. |
OperationType Sparsemax |
Sparsemax - projects onto probability simplex, can produce sparse outputs. |
OperationType SphericalSoftmax |
Spherical Softmax - L2 normalization followed by softmax. |
OperationType Split |
Split tensor along an axis into multiple tensors. |
OperationType Sqrt |
Element-wise square root. |
OperationType Square |
Element-wise square - x² for each element. |
OperationType Squash |
Squashing activation for capsule networks - s(v) = ||v||² / (1 + ||v||²) * (v / ||v||). |
OperationType Squeeze |
Remove dimensions of size 1. |
OperationType Stack |
Stack tensors along new axis. |
OperationType StraightThroughThreshold |
Straight-through threshold for HTM-style sparse activations. |
OperationType Subtract |
Element-wise subtraction of two tensors. |
OperationType SurrogateSpike |
Surrogate spike function for spiking neural networks with gradient estimation. |
OperationType Swish |
Swish/SiLU activation - x * sigmoid(x). |
OperationType Tanh |
Hyperbolic tangent activation. |
OperationType TaylorSoftmax |
Taylor Softmax - softmax using Taylor series approximation of exp. |
OperationType ThresholdedReLU |
Thresholded Rectified Linear Unit - x if x > threshold, 0 otherwise. |
OperationType TopKSoftmax |
Top-K softmax for mixture-of-experts routing. |
OperationType Transpose |
Matrix transpose - swaps rows and columns. |
OperationType Unknown |
Unknown operation type. |
OperationType Unsqueeze |
Add dimension of size 1. |
OperationType Upsample |
Upsample tensor by repeating elements. |
OperationType Upsample3D |
3D upsampling operation for volumetric data. |
OperationType WedgeProduct |
Wedge (outer) product of multivectors. |
OperationType Xor |
Logical XOR. |
int value__ |