PReLULayer<T>

Layers · AiDotNet.NeuralNetworks.Layers

Implements a Parametric ReLU (PReLU) layer with learnable negative-slope coefficients.

Inheritance: LayerBase<T> → PReLULayer<T>

Implements: IShapeContract

For Beginners

PReLU is like Leaky ReLU except the “leakiness” is learned from data instead of being set by you. For positive inputs the output is unchanged; for negative inputs the output is the input scaled by α. The network adjusts α during training to control how much signal to let through on the negative side. Per-channel α lets different channels learn different leaks, which often works better than a single shared α for convolutional networks.

How It Works

PReLU is a learnable variant of Leaky ReLU where the negative-slope coefficient α is a trainable parameter rather than a fixed hyperparameter. The function is:

f(x) = max(0, x) + α * min(0, x) = ReLU(x) - α * ReLU(-x)

Two parameterization modes are supported, matching PyTorch’s nn.PReLU:

Introduced in He et al., “Delving Deep into Rectifiers” (2015), which also introduced the Kaiming initialization. The paper’s recommended initial α is 0.25, which is this layer’s default.

Constructors

Constructor Summary
PReLULayer(int numParameters, int channelAxis, double initialAlpha) Initializes a new PReLULayer.

Properties

Property Summary
bool SupportsTraining { get; } Gets a value indicating that this layer has trainable parameters (α).
int ChannelAxis { get; } Gets the tensor axis used for per-channel broadcasting.
int NumParameters { get; } Gets the number of independently learned negative slopes.

Methods

Method Summary
Dictionary<string, string> GetParameterRoles() Returns parameter roles for per-role optimizer configuration (e.g., weight decay exemption for biases).
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank) Derived from this type’s [TensorLayout] declarations.
IReadOnlyList<Tensor<T>> GetTrainableParameters() Returns all trainable parameter tensors marked with [TrainableParameter].
Tensor<T> GetAlphaTensor() Gets the current α tensor.
void ResetState()
void SetTrainableParameters(IReadOnlyList<Tensor<T>> parameters) Replaces trainable parameter tensors (e.g., with ParameterBuffer views).
void ZeroGrad() Clears all gradient fields discovered by convention ({paramName}Gradient).