NeRF<T>

Models & Types · AiDotNet.NeuralRadianceFields.Models

Implements Neural Radiance Fields (NeRF) for novel view synthesis.

Inheritance: NeuralNetworkBase<T> → DeclaredModelLayoutBase<T> → VectorModelLayoutBase<T> → NeRF<T>

Implements: IRadianceField<T>

For Beginners

NeRF is a groundbreaking method for creating photorealistic 3D scenes from 2D images.

How It Works

What NeRF does:

  • Input: Collection of photos of a scene from different angles
  • Training: Learn a neural network that represents the 3D scene
  • Output: Ability to render the scene from any new viewpoint

Key innovation:

  • Represents entire 3D scene as a continuous 5D function
  • Input: (x, y, z, θ, φ) - position and viewing direction
  • Output: (r, g, b, σ) - color and volume density

Architecture:

  1. Positional encoding: Transform (x,y,z) to higher-dimensional space
  • Why: Helps network learn high-frequency details
  • Example: (x,y,z) → [sin(x), cos(x), sin(2x), cos(2x), …, sin(2^Lx), cos(2^Lx)]
  • Similar encoding for direction (θ, φ)
  1. Coarse network (8 layers, 256 units):
  • Input: Encoded position
  • Output: Density + intermediate features
  • Input: Intermediate features + encoded direction
  • Output: RGB color
  1. Fine network (same structure):
  • Resamples based on coarse network predictions
  • Focuses samples where density is high
  • Produces final high-quality output

Why positional encoding matters:

  • Neural networks naturally learn low-frequency functions (smooth, blurry)
  • Real scenes have high-frequency details (sharp edges, textures)
  • Positional encoding enables learning high-frequency details
  • Without it: Blurry reconstructions
  • With it: Sharp, detailed reconstructions

Training process:

  1. Sample random rays from training images
  2. Sample points along each ray
  3. Query network at each sample point
  4. Render ray using volume rendering
  5. Compare rendered color to actual pixel color
  6. Backpropagate error and update network weights
  7. Repeat for thousands of iterations

Hierarchical sampling:

  • Coarse sampling: Uniform samples along ray
  • Analyze coarse results: Where is density high?
  • Fine sampling: More samples where density is high (near surfaces)
  • Final rendering: Use both coarse and fine samples
  • Result: Better quality with fewer total samples

Rendering equation (volume rendering): C(r) = Σ T(t_i) * (1 - exp(-σ_i * δ_i)) * c_i where:

  • C(r): Final color of ray r
  • T(t_i): Transmittance (how much light reaches point i)
  • σ_i: Density at sample point i
  • δ_i: Distance between sample points
  • c_i: Color at sample point i
  • T(t_i) = exp(-Σ(j<i) σ_j * δ_j)

Applications:

  • Virtual reality: Create immersive 3D environments from photos
  • Film industry: Digitize real locations for CGI
  • Real estate: Virtual property tours
  • Cultural heritage: Preserve historical sites digitally
  • Robotics: Build 3D maps for navigation
  • Medical imaging: Reconstruct 3D anatomy from scans

Limitations of original NeRF:

  • Slow training: Hours to days per scene
  • Slow rendering: Seconds per image
  • Scene-specific: Must retrain for each new scene
  • Static only: Can’t handle moving objects

These limitations led to many improved variants:

  • Instant-NGP: 100x faster training and rendering
  • Plenoxels: No neural network, faster optimization
  • TensoRF: Tensor decomposition for efficiency
  • Dynamic NeRF: Handle time-varying scenes
  • Mip-NeRF: Better handling of scale/blur

Reference: “NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis” by Mildenhall et al., ECCV 2020

Example

// Create a NeRF model with custom architecture parameters
var nerf = new NeRF<float>(options: new NeRFOptions { PositionEncodingLevels = 10, DirectionEncodingLevels = 4, HiddenDim = 256, NumLayers = 8 });
// Render a novel view from camera position and direction
var positions = Tensor<float>.CreateRandom(1024, 3);           // sample points along the rays
var viewingDirections = Tensor<float>.CreateRandom(1024, 3);   // the direction each was seen from
var (rgb, density) = nerf.QueryField(positions, viewingDirections);

Constructors

Constructor Summary
NeRF(ILossFunction<T> lossFunction, NeRFOptions options) Initializes the model from an already-resolved options instance.

Properties

Property Summary
bool SupportsTraining { get; } Gets whether this network supports training.

Methods

Method Summary
ModelMetadata<T> GetModelMetadata() Gets metadata about the model.
T TrainOnImageBatch(IDataLoader<ImageView<T>, PixelBatch<T>> loader, int raysPerBatch, OptimizationAlgorithmOptions<T, Tensor<T>, Tensor<T>> optimizerOptions, ImageTrainingOptions imageTrainingOptions) Image-space photometric training (#1834).
Tensor<T> ForwardForTraining(Tensor<T> input) Tape-aware forward pass used by TrainWithTape.
Tensor<T> ForwardWithMemory(Tensor<T> input) Performs forward pass with memory for backpropagation.
Tensor<T> RenderImage(Vector<T> cameraPosition, Matrix<T> cameraRotation, int imageWidth, int imageHeight, T focalLength) Renders an image from a camera viewpoint.
Tensor<T> RenderRays(Tensor<T> rayOrigins, Tensor<T> rayDirections, int numSamples, T nearBound, T farBound) Renders colors for a batch of rays.
ValueTuple<Tensor<T>, Tensor<T>> QueryField(Tensor<T> positions, Tensor<T> viewingDirections) Queries the radiance field at given positions and viewing directions.