NeRF<T>
Models & Types · AiDotNet.NeuralRadianceFields.Models
Implements Neural Radiance Fields (NeRF) for novel view synthesis.
Inheritance: NeuralNetworkBase<T> → DeclaredModelLayoutBase<T> → VectorModelLayoutBase<T> → NeRF<T>
Implements: IRadianceField<T>
For Beginners
NeRF is a groundbreaking method for creating photorealistic 3D scenes from 2D images.
How It Works
What NeRF does:
- Input: Collection of photos of a scene from different angles
- Training: Learn a neural network that represents the 3D scene
- Output: Ability to render the scene from any new viewpoint
Key innovation:
- Represents entire 3D scene as a continuous 5D function
- Input: (x, y, z, θ, φ) - position and viewing direction
- Output: (r, g, b, σ) - color and volume density
Architecture:
- Positional encoding: Transform (x,y,z) to higher-dimensional space
- Why: Helps network learn high-frequency details
- Example: (x,y,z) → [sin(x), cos(x), sin(2x), cos(2x), …, sin(2^Lx), cos(2^Lx)]
- Similar encoding for direction (θ, φ)
- Coarse network (8 layers, 256 units):
- Input: Encoded position
- Output: Density + intermediate features
- Input: Intermediate features + encoded direction
- Output: RGB color
- Fine network (same structure):
- Resamples based on coarse network predictions
- Focuses samples where density is high
- Produces final high-quality output
Why positional encoding matters:
- Neural networks naturally learn low-frequency functions (smooth, blurry)
- Real scenes have high-frequency details (sharp edges, textures)
- Positional encoding enables learning high-frequency details
- Without it: Blurry reconstructions
- With it: Sharp, detailed reconstructions
Training process:
- Sample random rays from training images
- Sample points along each ray
- Query network at each sample point
- Render ray using volume rendering
- Compare rendered color to actual pixel color
- Backpropagate error and update network weights
- Repeat for thousands of iterations
Hierarchical sampling:
- Coarse sampling: Uniform samples along ray
- Analyze coarse results: Where is density high?
- Fine sampling: More samples where density is high (near surfaces)
- Final rendering: Use both coarse and fine samples
- Result: Better quality with fewer total samples
Rendering equation (volume rendering): C(r) = Σ T(t_i) * (1 - exp(-σ_i * δ_i)) * c_i where:
- C(r): Final color of ray r
- T(t_i): Transmittance (how much light reaches point i)
- σ_i: Density at sample point i
- δ_i: Distance between sample points
- c_i: Color at sample point i
- T(t_i) = exp(-Σ(j<i) σ_j * δ_j)
Applications:
- Virtual reality: Create immersive 3D environments from photos
- Film industry: Digitize real locations for CGI
- Real estate: Virtual property tours
- Cultural heritage: Preserve historical sites digitally
- Robotics: Build 3D maps for navigation
- Medical imaging: Reconstruct 3D anatomy from scans
Limitations of original NeRF:
- Slow training: Hours to days per scene
- Slow rendering: Seconds per image
- Scene-specific: Must retrain for each new scene
- Static only: Can’t handle moving objects
These limitations led to many improved variants:
- Instant-NGP: 100x faster training and rendering
- Plenoxels: No neural network, faster optimization
- TensoRF: Tensor decomposition for efficiency
- Dynamic NeRF: Handle time-varying scenes
- Mip-NeRF: Better handling of scale/blur
Reference: “NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis” by Mildenhall et al., ECCV 2020
Example
// Create a NeRF model with custom architecture parameters
var nerf = new NeRF<float>(options: new NeRFOptions { PositionEncodingLevels = 10, DirectionEncodingLevels = 4, HiddenDim = 256, NumLayers = 8 });
// Render a novel view from camera position and direction
var positions = Tensor<float>.CreateRandom(1024, 3); // sample points along the rays
var viewingDirections = Tensor<float>.CreateRandom(1024, 3); // the direction each was seen from
var (rgb, density) = nerf.QueryField(positions, viewingDirections);
Constructors
| Constructor | Summary |
|---|---|
NeRF(ILossFunction<T> lossFunction, NeRFOptions options) |
Initializes the model from an already-resolved options instance. |
Properties
| Property | Summary |
|---|---|
bool SupportsTraining { get; } |
Gets whether this network supports training. |
Methods
| Method | Summary |
|---|---|
ModelMetadata<T> GetModelMetadata() |
Gets metadata about the model. |
T TrainOnImageBatch(IDataLoader<ImageView<T>, PixelBatch<T>> loader, int raysPerBatch, OptimizationAlgorithmOptions<T, Tensor<T>, Tensor<T>> optimizerOptions, ImageTrainingOptions imageTrainingOptions) |
Image-space photometric training (#1834). |
Tensor<T> ForwardForTraining(Tensor<T> input) |
Tape-aware forward pass used by TrainWithTape. |
Tensor<T> ForwardWithMemory(Tensor<T> input) |
Performs forward pass with memory for backpropagation. |
Tensor<T> RenderImage(Vector<T> cameraPosition, Matrix<T> cameraRotation, int imageWidth, int imageHeight, T focalLength) |
Renders an image from a camera viewpoint. |
Tensor<T> RenderRays(Tensor<T> rayOrigins, Tensor<T> rayDirections, int numSamples, T nearBound, T farBound) |
Renders colors for a batch of rays. |
ValueTuple<Tensor<T>, Tensor<T>> QueryField(Tensor<T> positions, Tensor<T> viewingDirections) |
Queries the radiance field at given positions and viewing directions. |