NeuralNetworkKernel<T>
Models & Types · AiDotNet.Kernels
Implements the Neural Network (Arc-Cosine) kernel that corresponds to infinitely wide neural networks.
Implements: IKernelFunction<T>
For Beginners
The Neural Network kernel (also called Arc-Cosine kernel) is fascinating because it exactly corresponds to the behavior of infinitely wide neural networks with specific activation functions.
Key insight: As a neural network gets wider and wider (more neurons in each layer), its behavior becomes more predictable and can be described by a Gaussian Process with this kernel. This connection is called the “Neural Network Gaussian Process” (NNGP).
The kernel depends on the activation function:
- Order 0: Step function (Heaviside) - Measures angle between inputs
- Order 1: ReLU - Captures piecewise linear behavior
- Order 2: ReQU (Rectified Quadratic Unit) - Smoother, more expressive
How It Works
Mathematical form: k_n(x, x’) = (1/π) × ||x||^n × ||x’||^n × J_n(θ)
Where:
- θ = angle between x and x’ (arccos of normalized dot product)
- J_n(θ) = (-1)^n × (sin(θ))^(2n+1) × (∂/∂t)^n [(π-θ)/sin(θ)] evaluated at t=cos(θ)
For n=1 (ReLU): J_1(θ) = sin(θ) + (π-θ)cos(θ)
Why use this kernel?
- Neural Network Connection: Analyze what infinite neural networks can learn
- Deep Architectures: Can be composed for “deep” kernel behavior
- Non-stationary: Unlike RBF, it’s not translation-invariant
- Theoretical Insights: Helps understand deep learning through GP lens
Example
using AiDotNet.Kernels;
using AiDotNet.Tensors.LinearAlgebra;
var kernel = new NeuralNetworkKernel<double>();
var a = new Vector<double>(new[] { 1.0, 2.0, 3.0 });
var b = new Vector<double>(new[] { 1.5, 1.0, 2.5 });
double similarity = kernel.Calculate(a, b);
Console.WriteLine($"NeuralNetworkKernel: K(a, b) = {similarity:F4}");
Constructors
| Constructor | Summary |
|---|---|
NeuralNetworkKernel(int order, double weightVariance, double biasVariance) |
Initializes a new Neural Network kernel. |
Properties
| Property | Summary |
|---|---|
double BiasVariance { get; } |
Gets the bias variance. |
double WeightVariance { get; } |
Gets the weight variance. |
int Order { get; } |
Gets the kernel order. |
Methods
| Method | Summary |
|---|---|
DeepNeuralNetworkKernel<T> ToDeep(int numLayers) |
Creates a “deep” version of the kernel by composing multiple layers. |
T Calculate(Vector<T> x1, Vector<T> x2) |
Calculates the Neural Network kernel value between two vectors. |