ShapEModel<T>

Models & Types · AiDotNet.Diffusion.ThreeD

Shap-E model for text-to-3D and image-to-3D generation with implicit neural representations.

Inheritance: DiffusionModelBase<T> → LatentDiffusionModelBase<T> → ThreeDDiffusionModelBase<T> → ShapEModel<T>

For Beginners

Shap-E creates 3D objects that you can view from any angle:

What is an Implicit Neural Representation (NeRF)?

  • A neural network that knows the 3D shape
  • Input: 3D coordinates (x, y, z)
  • Output: Color and density at that point
  • Can render views from ANY angle without artifacts

Comparison with Point-E:

Feature Point-E Shap-E
Output Point cloud Neural field
Quality Good Better
Rendering Fast Slower
Mesh export Reconstruction Direct SDF
Memory Lower Higher

Example: “A red chair”

  1. Shap-E generates network weights (latent representation)
  2. These weights define a neural network
  3. Query (x,y,z) -> neural network -> color, density
  4. Render from any view or extract mesh via marching cubes

Use cases:

  • High-quality 3D assets
  • Novel view synthesis
  • Direct mesh export with SDF
  • View-consistent 3D models

How It Works

Shap-E is OpenAI’s model for generating 3D objects as implicit neural representations (NeRFs). Unlike Point-E which generates point clouds, Shap-E generates parameters for a neural network that represents the 3D shape, which can then be rendered from any angle or converted to meshes.

Technical specifications:

  • Latent dimension: 1024 parameters per shape
  • Output: NeRF weights or SDF (Signed Distance Function)
  • Rendering: Differentiable volumetric rendering
  • Mesh export: Marching cubes on SDF
  • Inference: ~64 steps

Example

// Create a Shap-E model
var shapE = new ShapEModel<float>();
            
// Generate a 3D shape from text
var latent = shapE.GenerateLatent(
    prompt: "A wooden chair",
    numInferenceSteps: 64,
    guidanceScale: 15.0);
            
// Render from a specific view
var image = shapE.RenderView(latent, cameraPosition: (0, 0, 2), lookAt: (0, 0, 0));
            
// Export to mesh
var (vertices, faces) = shapE.ExtractMesh(latent, resolution: 64);
            
// Or use the high-level API
var mesh = shapE.GenerateMesh(
    prompt: "A red sports car",
    resolution: 128);
            
// mesh is the result tensor — write it out with your own encoder.

Constructors

Constructor Summary
ShapEModel(NeuralNetworkArchitecture<T> architecture, DiffusionModelOptions<T> options, INoiseScheduler<T> scheduler, DiTNoisePredictor<T> latentPredictor, IConditioningModule<T> conditioner, bool useSDFMode, int defaultPointCount, int? seed) Initializes a new Shap-E model with full customization support.

Properties

Property Summary
IConditioningModule<T> Conditioner { get; }
INoisePredictor<T> NoisePredictor { get; }
IVAEModel<T> VAE { get; }
bool SupportsMesh { get; }
bool SupportsNovelView { get; }
bool SupportsPointCloud { get; }
bool SupportsScoreDistillation { get; }
bool SupportsTexture { get; }
bool UseSDFMode { get; } Gets whether this model uses SDF mode.
int LatentChannels { get; }
int LatentDimension { get; } Gets the latent dimension.

Methods

Method Summary
Mesh3D<T> GenerateMesh(string prompt, string negativePrompt, int resolution, int numInferenceSteps, double guidanceScale, int? seed) Generates a mesh directly from a text prompt.
Mesh3D<T> ImageTo3D(Tensor<T> inputImage, int numViews, int numInferenceSteps, double guidanceScale, int? seed) Image-to-3D with Shap-E’s image-conditioned pipeline (Jun and Nichol, 2023, image300M): the image conditions the latent diffusion (GenerateLatentFromImage), and the latent’s implicit SDF is decoded and its zero level set extracted on a 64…
ModelMetadata<T> GetModelMetadata()
Tensor<T> GenerateLatent(string prompt, string negativePrompt, int numInferenceSteps, double guidanceScale, int? seed) Generates a latent representation of a 3D shape from text.
Tensor<T> GenerateLatentFromImage(Tensor<T> image, int numInferenceSteps, double guidanceScale, int? seed) Generates a latent from an image.
Tensor<T> GeneratePointCloud(string prompt, string negativePrompt, int? numPoints, int numInferenceSteps, double guidanceScale, int? seed)
Tensor<T> RenderView(Tensor<T> latent, ValueTuple<double, double, double> cameraPosition, ValueTuple<double, double, double> lookAt, int imageSize, int numSamples) Renders a view of the shape from a camera position.
Tensor<T> SamplePointCloud(Tensor<T> latent, int? numPoints, int? seed) Samples a point cloud from the shape.
ValueTuple<Tensor<T>, Tensor<T>> ExtractMesh(Tensor<T> latent, int resolution) Extracts a mesh from the latent using marching cubes.