OperationType

Enums · AiDotNet.Enums

Represents different operation types in computation graphs for JIT compilation and automatic differentiation.

For Beginners

Operation types identify mathematical operations performed on tensors in neural networks.

When building a computation graph, each operation (like adding two tensors or applying an activation function) needs to be identified so that:

  1. The JIT compiler can optimize the code
  2. The automatic differentiation system can compute gradients correctly
  3. The system can analyze and transform the computation graph

This enum provides type-safe identification of operations, preventing typos and enabling better tooling support.

Fields

Field Summary
OperationType Abs Element-wise absolute value - |x| for each element.
OperationType Activation Generic activation function application.
OperationType AdaptivePooling Adaptive pooling.
OperationType Add Element-wise addition of two tensors.
OperationType AffineGrid Affine grid generation for spatial transformers.
OperationType And Logical AND.
OperationType AnomalyScore Anomaly score computation.
OperationType Attention Generic attention mechanism operation.
OperationType AveragePooling Average pooling operation.
OperationType AvgPool2D 2D average pooling.
OperationType BatchNorm Batch normalization.
OperationType BatchNormalization Batch normalization.
OperationType BentIdentity Bent Identity - (sqrt(x² + 1) - 1) / 2 + x, smooth alternative to ReLU.
OperationType Broadcast Broadcast operation - expands tensor dimensions to match target shape.
OperationType CELU Continuously Differentiable ELU - max(0, x) + min(0, α * (exp(x/α) - 1)).
OperationType CRFForward CRF forward algorithm for sequence labeling.
OperationType Cast Type cast operation.
OperationType CliffordInnerProduct Inner (contraction) product of multivectors.
OperationType Clip Clip values to range.
OperationType ComplexMatMul Complex matrix multiplication for quantum operations.
OperationType ComplexMultiply Element-wise complex multiplication.
OperationType Concat Concatenate multiple tensors along an axis.
OperationType Constant Constant node - represents a constant value that doesn’t require gradients.
OperationType Conv2D 2D convolution operation.
OperationType ConvTranspose2D 2D transposed convolution (deconvolution).
OperationType Convolution General convolution operation.
OperationType Convolution2D 2D convolution operation.
OperationType Convolution3D 3D convolution operation.
OperationType Crop Crop tensor by removing border elements.
OperationType CrossAttention Cross-attention operation.
OperationType Custom Custom user-defined operation for extensibility.
OperationType Deconvolution Deconvolution (transposed convolution) operation.
OperationType DeformableConv2D 2D deformable convolution with learnable offsets and optional modulation.
OperationType Dense Dense (fully connected) layer.
OperationType DepthwiseConv2D 2D depthwise convolution.
OperationType DepthwiseConvolution Depthwise convolution operation.
OperationType DilatedConv2D 2D dilated (atrous) convolution.
OperationType DilatedConvolution Dilated convolution operation.
OperationType Divide Element-wise division of two tensors.
OperationType DropPath DropPath regularization.
OperationType Dropout Dropout regularization operation - randomly zeros elements during training.
OperationType ELU Exponential Linear Unit - ELU(x) = x if x > 0, alpha * (exp(x) - 1) otherwise.
OperationType Embedding Embedding lookup operation.
OperationType Equal Element-wise equality.
OperationType Exp Element-wise exponential function - e^x for each element.
OperationType Expand Expand tensor dimensions.
OperationType FakeQuantization Fake quantization operation with Straight-Through Estimator (STE) for differentiable quantization.
OperationType Flatten Flatten tensor to 1D.
OperationType FullyConnected Fully connected layer.
OperationType FusedAddReLU Fused addition + ReLU.
OperationType FusedConvBatchNorm Fused convolution + batch normalization.
OperationType FusedConvBatchNormReLU Fused Conv + BatchNorm + ReLU.
OperationType FusedLayerNormAttention Fused LayerNorm + Attention.
OperationType FusedLinearReLU Fused linear layer with ReLU (MatMul + Add + ReLU).
OperationType FusedMatMulAdd Fused matrix multiplication + addition (MatMul + Add).
OperationType FusedMatMulBias Fused MatMul + Bias.
OperationType FusedMatMulBiasGELU Fused MatMul + Bias + GELU.
OperationType FusedMatMulBiasReLU Fused MatMul + Bias + ReLU.
OperationType FusedMultiHeadAttention Fused MultiHead Attention.
OperationType GELU Gaussian Error Linear Unit - x * Φ(x) where Φ is standard normal CDF.
OperationType GRU GRU recurrent layer.
OperationType GRUCell GRU cell operation for recurrent networks.
OperationType Gather Gather operation - selects elements from a tensor using indices.
OperationType Gaussian Gaussian activation - exp(-x²), bell-shaped response curve.
OperationType Gemm General Matrix Multiplication.
OperationType GeometricProduct Geometric product of multivectors.
OperationType GlobalAveragePooling Global average pooling.
OperationType GlobalMaxPooling Global max pooling.
OperationType GradeProject Grade projection of multivectors.
OperationType GraphConv Graph convolutional operation for GNNs.
OperationType Greater Element-wise greater than.
OperationType GreaterOrEqual Element-wise greater or equal.
OperationType GridSample Grid sampling for spatial transformers.
OperationType GroupNormalization Group normalization.
OperationType GumbelSoftmax Gumbel-Softmax for differentiable discrete sampling (used in stochastic layers).
OperationType HardSigmoid Hard Sigmoid - piecewise linear approximation of sigmoid: clip((x + 1) / 2, 0, 1).
OperationType HardTanh Hard Tanh - piecewise linear approximation of tanh: clip(x, -1, 1).
OperationType HierarchicalSoftmax Hierarchical Softmax - tree-based efficient softmax for large vocabularies.
OperationType HyperboloidDistance Hyperboloid distance metric.
OperationType ISRU Inverse Square Root Unit - x / sqrt(1 + alpha * x²).
OperationType Input Input node - represents a variable or parameter in the computation graph.
OperationType InstanceNormalization Instance normalization.
OperationType LSTM LSTM recurrent layer.
OperationType LSTMCell LSTM cell operation for recurrent networks.
OperationType LayerNorm Layer normalization.
OperationType LayerNormalization Layer normalization.
OperationType LeakyReLU Leaky Rectified Linear Unit - max(alpha * x, x) where alpha is typically 0.01.
OperationType LeakyStateUpdate Leaky state update for reservoir/echo state networks.
OperationType Less Element-wise less than.
OperationType LessOrEqual Element-wise less or equal.
OperationType LiSHT Linearly Scaled Hyperbolic Tangent - x * tanh(x).
OperationType LocallyConnectedConv2D 2D locally connected convolution.
OperationType Log Element-wise natural logarithm.
OperationType LogSoftmax Log-Softmax - log(softmax(x)), numerically stable for cross-entropy loss.
OperationType LogSoftmin Log-Softmin - log(softmin(x)) = log(softmax(-x)).
OperationType MatMul Matrix multiplication (not element-wise).
OperationType MaxPool2D 2D max pooling.
OperationType MaxPool3D 3D max pooling.
OperationType MaxPooling Max pooling operation.
OperationType Maxout Maxout activation - maximum over multiple linear pieces.
OperationType Mean Mean operation (reduces all dimensions).
OperationType Mish Mish activation - x * tanh(softplus(x)).
OperationType MobiusAdd Mobius addition in Poincare ball model.
OperationType MultiHeadAttention Multi-head attention operation.
OperationType Multiply Element-wise multiplication (Hadamard product) of two tensors.
OperationType MultivectorAdd Multivector addition.
OperationType MultivectorReverse Multivector reverse operation.
OperationType Negate Element-wise negation - multiplies each element by -1.
OperationType Norm L2 norm computation along an axis - sqrt(sum(x²)).
OperationType Not Logical NOT.
OperationType OctonionAdd Octonion addition.
OperationType OctonionConjugate Octonion conjugation.
OperationType OctonionMatMul Octonion matrix multiplication for neural networks.
OperationType OctonionMultiply Octonion multiplication (non-associative).
OperationType Or Logical OR.
OperationType Output Output node in computation graph.
OperationType PReLU Parametric Rectified Linear Unit - max(0, x) + alpha * min(0, x) where alpha is learned.
OperationType Pad Pad tensor with values.
OperationType Permute Permute tensor dimensions (general transpose).
OperationType PixelShuffle Pixel shuffle operation for upsampling.
OperationType PoincareDistance Poincare ball distance metric.
OperationType PoincareExpMap Poincare exponential map.
OperationType PoincareLogMap Poincare logarithmic map.
OperationType PositionalEncoding Positional encoding for transformers.
OperationType Power Element-wise power operation - raises each element to a specified exponent.
OperationType RBFKernel RBF (Radial Basis Function) kernel operation.
OperationType RNN Basic RNN layer.
OperationType RReLU Randomized Leaky ReLU - LeakyReLU with random alpha during training.
OperationType ReLU Rectified Linear Unit - max(0, x).
OperationType ReduceLogVariance Log-variance reduction along specified axes.
OperationType ReduceMax Maximum value reduction along specified axes.
OperationType ReduceMean Mean reduction along specified axes.
OperationType ReduceMin Minimum value reduction.
OperationType ReduceSum Sum reduction along specified axes.
OperationType Reshape Reshape tensor to new dimensions.
OperationType SELU Scaled Exponential Linear Unit - self-normalizing activation with fixed lambda and alpha.
OperationType SQRBF Square Radial Basis Function - smooth bell-shaped activation.
OperationType ScaledDotProductAttention Scaled dot-product attention.
OperationType ScaledTanh Scaled Tanh - parameterized tanh with adjustable steepness β.
OperationType Scatter Scatter values to indices.
OperationType Se3Exp SE(3) exponential map.
OperationType Se3Log SE(3) logarithmic map.
OperationType SelfAttention Self-attention operation.
OperationType Sigmoid Sigmoid activation - 1 / (1 + e^(-x)).
OperationType Sign Sign function with surrogate gradient for training - returns -1, 0, or 1.
OperationType Slice Slice tensor along an axis - extract a portion with optional stride.
OperationType So3Exp SO(3) exponential map.
OperationType So3Log SO(3) logarithmic map.
OperationType SoftKNN Soft K-Nearest Neighbors operation for differentiable instance-based learning.
OperationType SoftLocallyWeighted Soft locally-weighted regression operation for differentiable instance-based learning.
OperationType SoftPlus SoftPlus activation - ln(1 + e^x), smooth approximation of ReLU.
OperationType SoftSign SoftSign activation - x / (1 + |x|), alternative to tanh with polynomial tails.
OperationType SoftSplit Soft split operation for differentiable decision trees.
OperationType Softmax Softmax activation - converts logits to probability distribution.
OperationType Softmin Softmin - softmax(-x), assigns higher probability to lower values.
OperationType SpMM Sparse matrix-matrix multiplication.
OperationType SpMV Sparse matrix-vector multiplication.
OperationType SparseGather Sparse gather operation.
OperationType SparseScatter Sparse scatter operation.
OperationType Sparsemax Sparsemax - projects onto probability simplex, can produce sparse outputs.
OperationType SphericalSoftmax Spherical Softmax - L2 normalization followed by softmax.
OperationType Split Split tensor along an axis into multiple tensors.
OperationType Sqrt Element-wise square root.
OperationType Square Element-wise square - x² for each element.
OperationType Squash Squashing activation for capsule networks - s(v) = ||v||² / (1 + ||v||²) * (v / ||v||).
OperationType Squeeze Remove dimensions of size 1.
OperationType Stack Stack tensors along new axis.
OperationType StraightThroughThreshold Straight-through threshold for HTM-style sparse activations.
OperationType Subtract Element-wise subtraction of two tensors.
OperationType SurrogateSpike Surrogate spike function for spiking neural networks with gradient estimation.
OperationType Swish Swish/SiLU activation - x * sigmoid(x).
OperationType Tanh Hyperbolic tangent activation.
OperationType TaylorSoftmax Taylor Softmax - softmax using Taylor series approximation of exp.
OperationType ThresholdedReLU Thresholded Rectified Linear Unit - x if x > threshold, 0 otherwise.
OperationType TopKSoftmax Top-K softmax for mixture-of-experts routing.
OperationType Transpose Matrix transpose - swaps rows and columns.
OperationType Unknown Unknown operation type.
OperationType Unsqueeze Add dimension of size 1.
OperationType Upsample Upsample tensor by repeating elements.
OperationType Upsample3D 3D upsampling operation for volumetric data.
OperationType WedgeProduct Wedge (outer) product of multivectors.
OperationType Xor Logical XOR.
int value__