GaussianSplatting<T>

Models & Types · AiDotNet.NeuralRadianceFields.Models

Implements 3D Gaussian Splatting for real-time novel view synthesis.

Inheritance: NeuralNetworkBase<T> → DeclaredModelLayoutBase<T> → VectorModelLayoutBase<T> → GaussianSplatting<T>

Implements: IRadianceField<T>

For Beginners

3D Gaussian Splatting is the newest breakthrough in novel view synthesis, achieving real-time rendering (30+ FPS) with state-of-the-art quality.

How It Works

Speed comparison:

  • NeRF: ~30 seconds per frame
  • Instant-NGP: ~30 milliseconds per frame (~1000× faster than NeRF)
  • Gaussian Splatting: ~10 milliseconds per frame (~3000× faster than NeRF)
  • Gaussian Splatting can render at 100+ FPS on modern GPUs!

Key innovation - Explicit representation:

  • NeRF/Instant-NGP: Implicit (neural network represents the scene)
  • Gaussian Splatting: Explicit (scene is a collection of 3D Gaussians)

What’s a 3D Gaussian?

  • A “blob” of color in 3D space
  • Has a center position (x, y, z)
  • Has a size and shape (covariance matrix)
  • Has a color (RGB)
  • Has opacity (alpha)

Think of it like this:

  • Traditional rendering: Triangles (hard edges)
  • Volume rendering (NeRF): Sample continuous field
  • Gaussian Splatting: “Paint” with fuzzy 3D blobs

How it works:

  1. Start with point cloud (from SfM like COLMAP)
  2. Place a Gaussian at each point
  3. Optimize Gaussian parameters:
  • Position: Where is the Gaussian?
  • Covariance: What shape/size is it?
  • Color: What color is it?
  • Opacity: How transparent is it?
  1. Add/remove Gaussians as needed (adaptive densification)
  2. Render by “splatting” Gaussians onto image

Gaussian Splatting rendering:

  1. For each Gaussian:
  • Project to 2D image plane
  • Compute 2D Gaussian on screen
  • Determines which pixels it affects
  1. Sort Gaussians by depth (back to front)
  2. For each pixel:
  • Blend Gaussians that affect it (alpha blending)
  • Front-to-back or back-to-front blending
  1. Result: Final pixel color

This is MUCH faster than ray marching because:

  • No network evaluation (explicit representation)
  • Highly parallelizable (each Gaussian independent)
  • Efficient GPU rasterization (like traditional graphics)

Why Gaussians?

  • Smooth gradients for optimization
  • Can be rasterized efficiently (like triangles)
  • 2D projection is also Gaussian (mathematically elegant)
  • Adaptive: Can represent sharp edges with many small Gaussians

or smooth surfaces with fewer large Gaussians

Gaussian parameters:

  • Position μ: Center of Gaussian (3D point)
  • Covariance Σ: Shape and orientation (3×3 matrix)
  • Can represent ellipsoids (stretched in different directions)
  • Encoded as rotation + scale for easier optimization
  • Color c: RGB values (often with spherical harmonics for view-dependent effects)
  • Opacity α: Transparency (0 = invisible, 1 = opaque)

Total per Gaussian: 3 (pos) + 4 (rotation) + 3 (scale) + 3 (color) + 1 (opacity) = 14 values With spherical harmonics: Can be 40-60 values per Gaussian

Adaptive densification: During optimization, Gaussians are dynamically added/removed:

  • Clone: Copy Gaussians in high-gradient regions (need more detail)
  • Split: Large Gaussians → multiple smaller ones (refine detail)
  • Prune: Remove transparent Gaussians (save memory)

Example:

  • Start: 100K Gaussians from point cloud
  • After optimization: 500K-5M Gaussians
  • High detail areas: Many small Gaussians (e.g., edges, textures)
  • Smooth areas: Fewer large Gaussians (e.g., walls, sky)

Training process:

  1. Initialize from Structure-from-Motion point cloud
  2. For each training iteration:
  • Render view using current Gaussians
  • Compute loss (difference from ground truth image)
  • Backpropagate to Gaussian parameters
  • Update positions, colors, shapes, opacities
  • Every N iterations: Densify/prune Gaussians
  1. Converges in ~10-30 minutes (similar to Instant-NGP)

Advantages over NeRF:

  • Faster rendering: 100+ FPS vs 0.03 FPS (NeRF)
  • Explicit representation: Easy to edit, manipulate
  • No network evaluation: Simpler deployment
  • Better quality: Often sharper details
  • Easier to understand: Physical interpretation (colored blobs)

Disadvantages:

  • Memory: More than NeRF (millions of Gaussians)
  • NeRF: ~5-50MB
  • Gaussian Splatting: ~100-500MB
  • Requires good initialization (SfM point cloud)
  • Can have “floating” artifacts in empty regions
  • File size for storage

Rendering pipeline (simplified):

For each view:

1. Transform Gaussians to camera space
2. Project 3D Gaussians to 2D screen space
3. Compute 2D Gaussian parameters (center, covariance)
4. Determine which tiles/pixels each Gaussian affects
5. Sort Gaussians by depth within each tile
6. For each pixel:
- Accumulate color from affecting Gaussians
- Alpha blending: C = Σ α_i * c_i * Π(1 - α_j) for j < i
7. Output: Rendered image

Applications (especially suited for):

  • Real-time VR/AR: Low latency is critical
  • Gaming: Integration with game engines
  • Digital twins: Interactive 3D models of real places
  • Telepresence: Realistic remote environments
  • Film pre-visualization: Fast preview of captured scenes
  • Live events: Real-time volumetric capture

Comparison table:

Feature NeRF Instant-NGP Gaussian Splatting
Rendering speed 30 s 30 ms 10 ms (100+ FPS)
Training time 1-2 days 5-10 min 10-30 min
Quality High High Very High
Memory usage 5 MB 50 MB 200-500 MB
Editability Hard Hard Easy
Real-time rendering No Borderline Yes
GPU requirement Any CUDA Any (faster w/ CUDA)

Reference: “3D Gaussian Splatting for Real-Time Radiance Field Rendering” by Kerbl et al., SIGGRAPH 2023

Example

// Create a 3D Gaussian Splatting model from an initial point cloud
var options = new GaussianSplattingOptions { MaxGaussians = 100000, ShDegree = 3 };
var pointCloud = new Matrix<float>(1000, 3); // 1000 initial 3D points
var colors = new Matrix<float>(1000, 3);     // RGB colors per point
var gs = new GaussianSplatting<float>(options, pointCloud, colors);

Constructors

Constructor Summary
GaussianSplatting()
GaussianSplatting(GaussianSplattingOptions options, Matrix<T> initialPointCloud, Matrix<T> initialColors, ILossFunction<T> lossFunction)
GaussianSplatting(Matrix<T> initialPointCloud, Matrix<T> initialColors, ILossFunction<T> lossFunction) Initializes a new instance of the GaussianSplatting class.

Properties

Property Summary
bool EnableDensification { get; set; }
bool EnableSpatialIndex { get; set; }
bool SupportsTraining { get; }
double ColorLearningRate { get; set; }
double DefaultPointSpacing { get; set; }
double InitialNeighborSearchScale { get; set; }
double InitialScaleMultiplier { get; set; }
double MinScale { get; set; }
double OpacityLearningRate { get; set; }
double PositionLearningRate { get; set; }
double PruneOpacityThreshold { get; set; }
double RotationLearningRate { get; set; }
double ScaleLearningRate { get; set; }
double SplitGradientThreshold { get; set; }
double SplitOpacityFactor { get; set; }
double SplitOpacityMax { get; set; }
double SplitPositionJitter { get; set; }
double SplitScaleFactor { get; set; }
int DensificationInterval { get; set; }
int GaussianCount { get; } Gets the number of Gaussians currently in the scene.
int MaxGaussians { get; set; }
int SpatialIndexRadius { get; set; }
int TileSize { get; set; }

Methods

Method Summary
Dictionary<string, Tensor<T>> GetNamedLayerActivations(Tensor<T> input) Exposes inspectable activations for this explicit-representation model.
IReadOnlyList<ValueTuple<double, double, double, double, double, double>> GetGaussianDiagnostics() Diagnostic snapshot of the Gaussian cloud state — returns per-index summary (position, opacity-after-sigmoid, max-scale, mean-color).
ModelMetadata<T> GetModelMetadata()
T TrainOnImageBatch(IDataLoader<ImageView<T>, PixelBatch<T>> loader, int raysPerBatch, OptimizationAlgorithmOptions<T, Tensor<T>, Tensor<T>> optimizerOptions, ImageTrainingOptions imageTrainingOptions) Image-space photometric training (#1834).
Tensor<T> ForwardWithMemory(Tensor<T> input)
Tensor<T> RenderImage(Vector<T> cameraPosition, Matrix<T> cameraRotation, int imageWidth, int imageHeight, T focalLength)
Tensor<T> RenderRays(Tensor<T> rayOrigins, Tensor<T> rayDirections, int numSamples, T nearBound, T farBound)
ValueTuple<Tensor<T>, Tensor<T>> QueryField(Tensor<T> positions, Tensor<T> viewingDirections)
void ApplyOptimizerHyperparameters(OptimizationAlgorithmOptions<T, Tensor<T>, Tensor<T>> options) Consume the configured facade optimizer’s hyperparameters into GS’s per-attribute learning-rate schedule (#1833).
void RunCompressionPass() Excellence goal #3 — post-training compression pass: prunes low-opacity Gaussians, merges overlapping ellipses, and quantizes spherical-harmonics coefficients.