BinaryCrossEntropyWithLogitsLoss<T>

Models & Types · AiDotNet.LossFunctions

Implements Binary Cross-Entropy loss that accepts raw logits (not probabilities).

Inheritance: LossFunctionBase<T> → BinaryCrossEntropyWithLogitsLoss<T>

For Beginners

This loss function is equivalent to PyTorch’s nn.BCEWithLogitsLoss. It combines a sigmoid activation and binary cross-entropy into a single numerically stable computation.

How It Works

Unlike BinaryCrossEntropyLoss which expects probability inputs (after sigmoid), this version accepts raw logits (unbounded model outputs) and applies the sigmoid internally. This is the correct choice when your model’s final layer outputs raw scores without sigmoid activation — for example, the classification heads in DenseNet/EfficientNet/etc. emit logits because applying sigmoid in the model and then again inside the loss would compose two non-linear functions and produce wrong gradients.

The numerically stable form (avoids exp overflow for large positive x and log(0) for very negative x) is:

loss = max(x, 0) - x * y + log(1 + exp(-|x|))

Derivation: BCE on probability p = sigmoid(x) is -(y log p + (1-y) log(1-p)). Substituting p = 1/(1+exp(-x)) and using log(1-sigmoid(x)) = -x - log(1+exp(-x)), the expression simplifies to x - x*y + log(1+exp(-x)). The max(x, 0) + log(1 + exp(-|x|)) rewrite is the standard log-sum-exp trick that keeps both the very-positive and very-negative tails well-conditioned.

The gradient with respect to logits has the elegant form d(loss)/d(x) = sigmoid(x) - y, just like cross-entropy with logits.

Example

using AiDotNet.LossFunctions;
using AiDotNet.Tensors.LinearAlgebra;

var loss = new BinaryCrossEntropyWithLogitsLoss<float>();
var predicted = new Vector<float>(new[] { 0.9f, 0.2f, 0.7f });
var actual = new Vector<float>(new[] { 1.0f, 0.0f, 1.0f });

float value = loss.CalculateLoss(predicted, actual);
Console.WriteLine($"BinaryCrossEntropyWithLogitsLoss = {value:F4}");

Constructors

Constructor Summary
BinaryCrossEntropyWithLogitsLoss() Initializes a new instance of the BinaryCrossEntropyWithLogitsLoss class.

Methods

Method Summary
T CalculateLoss(Vector<T> predicted, Vector<T> actual) Calculates BCE loss directly from raw logits using the numerically stable form.
Tensor<T> ComputeTapeLoss(Tensor<T> predicted, Tensor<T> target)