HighwayLayer<T>

Layers · AiDotNet.NeuralNetworks.Layers

Represents a Highway Neural Network layer that allows information to flow unchanged through the network.

Inheritance: LayerBase<T> → HighwayLayer<T>

Implements: IAuxiliaryLossLayer<T>, IShapeContract

For Beginners

This layer helps solve a common problem in deep neural networks: difficulty in training very deep networks.

Think of the Highway Layer like a road with two lanes:

  • The “transform lane” processes the data like a regular neural network layer
  • The “bypass lane” lets information pass through unchanged
  • A “gate” controls how much information flows through each lane

For example, when processing an image, the gate might let basic features like edges pass through directly while sending more complex features through the transform lane for further processing.

This helps the network train more effectively because important information can flow more easily through many layers without being lost or distorted.

How It Works

A Highway Layer enables networks to effectively train even when they are very deep by introducing “gating units” which learn to selectively pass or transform information. Unlike regular feed-forward layers, highway layers have two “lanes”: the transform lane that processes input data and the bypass lane that passes information unchanged. The balance between these two lanes is controlled by a learned gating mechanism.

Constructors

Constructor Summary
HighwayLayer(int inputDimension, IActivationFunction<T> transformActivation, IActivationFunction<T> gateActivation, IInitializationStrategy<T> initializationStrategy) Initializes a new instance of the HighwayLayer class with the specified dimensions and element-wise activation functions.
HighwayLayer(int inputDimension, IVectorActivationFunction<T> transformActivation, IVectorActivationFunction<T> gateActivation) Initializes a new instance of the HighwayLayer class with the specified dimensions and vector activation functions.

Properties

Property Summary
T AuxiliaryLossWeight { get; set; } Gets or sets the weight for the auxiliary loss contribution.
bool SupportsTraining { get; } Gets a value indicating whether this layer supports training.
bool UseAuxiliaryLoss { get; set; } Gets or sets a value indicating whether auxiliary loss is enabled for this layer.

Methods

Method Summary
Dictionary<string, string> GetAuxiliaryLossDiagnostics() Gets diagnostic information about the auxiliary loss computation.
Dictionary<string, string> GetDiagnostics() Gets diagnostic information about this component’s state and behavior.
Dictionary<string, string> GetParameterRoles() Returns parameter roles for per-role optimizer configuration (e.g., weight decay exemption for biases).
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank) Derived from this type’s [TensorLayout] declarations.
IReadOnlyList<Tensor<T>> GetTrainableParameters() Returns all trainable parameter tensors marked with [TrainableParameter].
T ComputeAuxiliaryLoss() Computes the auxiliary loss for this layer based on gate balance regularization.
void ClearGradients()
void ResetState()
void SetTrainableParameters(IReadOnlyList<Tensor<T>> parameters) Replaces trainable parameter tensors (e.g., with ParameterBuffer views).
void UpdateParameters(T learningRate) Updates the parameters of the layer using the calculated gradients.
void ZeroGrad() Clears all gradient fields discovered by convention ({paramName}Gradient).