TensorRTConfiguration

Options & Configuration · AiDotNet.Deployment.TensorRT

Configuration for TensorRT model conversion and execution.

Constructors

Constructor Summary
TensorRTConfiguration()

Properties

Property Summary
List<OptimizationProfileConfig> OptimizationProfiles { get; set; } Gets or sets optimization profiles for dynamic shapes.
List<string> CustomPluginPaths { get; set; } Gets or sets custom plugin library paths.
TensorRTPrecision Precision { get; set; } Gets or sets the inference precision mode (default: FP32).
bool CleanupIntermediateFiles { get; set; } Gets or sets whether to cleanup intermediate files (ONNX, etc.) (default: true).
bool EnableCudaGraphs { get; set; } Gets or sets whether to enable CUDA graph capture (default: false).
bool EnableDLA { get; set; } Gets or sets whether to enable Deep Learning Accelerator (DLA) on Jetson devices.
bool EnableDynamicShapes { get; set; } Gets or sets whether to enable dynamic shapes (default: false).
bool EnableMultiStream { get; set; } Gets or sets whether to use multi-stream execution (default: false).
bool EnableProfiling { get; set; } Gets or sets whether to enable profiling (default: false).
bool StrictTypeConstraints { get; set; } Gets or sets whether to enable strict type constraints (default: false).
int BuilderOptimizationLevel { get; set; } Gets or sets the builder optimization level (0-5, higher is more optimization).
int DLACore { get; set; } Gets or sets the DLA core to use when EnableDLA is true (-1 = disabled, 0+ = specific core).
int DeviceId { get; set; } Gets or sets the GPU device ID to use (default: 0).
int MaxBatchSize { get; set; } Gets or sets the maximum batch size for inference (default: 1).
int NumStreams { get; set; } Gets or sets the number of streams for multi-stream execution (default: 2).
long MaxWorkspaceSize { get; set; } Gets or sets the maximum workspace size in bytes for TensorRT (default: 1GB).
string CalibrationDataPath { get; set; } Gets or sets the path to calibration data for INT8 quantization.
string EngineCachePath { get; set; } Gets or sets the engine cache path for faster reloading.

Methods

Method Summary
TensorRTConfiguration ForHighThroughput(int batchSize, string calibrationDataPath) Creates a configuration optimized for high throughput.
TensorRTConfiguration ForInt8(string calibrationDataPath) Creates a configuration with INT8 quantization.
TensorRTConfiguration ForLowLatency() Creates a configuration optimized for low latency (batch size 1).
TensorRTConfiguration ForMaxPerformance() Creates a configuration optimized for maximum performance.
void Validate() Validates the configuration and throws exceptions for invalid settings.