NoiseContrastiveEstimationLoss<T>
Models & Types · AiDotNet.LossFunctions
Implements the Noise Contrastive Estimation (NCE) loss function for efficient training with large output spaces.
Inheritance: LossFunctionBase<T> → NoiseContrastiveEstimationLoss<T>
For Beginners
Noise Contrastive Estimation (NCE) is a loss function designed to efficiently train models with very large output spaces, such as language models with large vocabularies.
Instead of computing probabilities for all possible outputs (which could be millions in language models), NCE transforms the problem into a binary classification task: distinguishing the true data from noise samples.
The key idea is to:
- Sample a small number of “negative” examples from a noise distribution
- Train the model to distinguish between true data points and these negative samples
This approach is much more computationally efficient than computing full softmax probabilities over all possible outputs, especially when the output space is very large.
NCE is commonly used in:
- Word embedding models like Word2Vec
- Neural language models with large vocabularies
- Any model with a very large output space
Example
using AiDotNet.LossFunctions;
using AiDotNet.Tensors.LinearAlgebra;
var loss = new NoiseContrastiveEstimationLoss<float>();
var predicted = new Vector<float>(new[] { 0.9f, 0.2f, 0.7f });
var actual = new Vector<float>(new[] { 1.0f, 0.0f, 1.0f });
float value = loss.CalculateLoss(predicted, actual);
Console.WriteLine($"NoiseContrastiveEstimationLoss = {value:F4}");
Constructors
| Constructor | Summary |
|---|---|
NoiseContrastiveEstimationLoss(int numNoiseSamples) |
Initializes a new instance of the NoiseContrastiveEstimationLoss class. |
Methods
| Method | Summary |
|---|---|
T Calculate(Vector<T> targetLogits, Matrix<T> noiseLogits) |
Calculates the NCE loss between target and noise samples. |
T CalculateLoss(Vector<T> predicted, Vector<T> actual) |
This method is not used for NCE Loss as it requires specific input formats. |
Tensor<T> ComputeTapeLoss(Tensor<T> predicted, Tensor<T> target) |