ConvTasNet<T>

Models & Types · AiDotNet.Audio.Enhancement

Conv-TasNet: A fully-convolutional time-domain audio separation network.

Inheritance: NeuralNetworkBase<T> → AudioNeuralNetworkBase<T> → ConvTasNet<T>

Implements: IAudioEnhancer<T>

For Beginners

Conv-TasNet is like having multiple microphones that each focus on one speaker in a noisy room. Give it a recording with multiple people talking, and it separates them into individual clean tracks!

Traditional methods convert audio to frequency domain, process it, then convert back. Conv-TasNet works directly on the waveform, which avoids problems with phase reconstruction and often produces cleaner results.

Common use cases:

  • Separating speakers in meeting recordings
  • Isolating vocals from music
  • Removing background noise
  • Speech enhancement for hearing aids
  • Denoising phone calls

How It Works

Conv-TasNet (Convolutional Time-domain Audio Separation Network) is a pioneering neural network architecture that operates directly in the time domain, avoiding the phase reconstruction problems of frequency-domain methods.

The architecture consists of three main components:

  • Encoder: Converts waveform to a learned representation using 1D convolutions
  • Separator: Temporal Convolutional Network (TCN) that estimates source masks
  • Decoder: Reconstructs separated waveforms from masked representations

Reference: Luo, Y., & Mesgarani, N. (2019). Conv-TasNet: Surpassing Ideal Time-Frequency Magnitude Masking for Speech Separation.

Example

// Use AiModelBuilder facade for audio source separation
var architecture = new NeuralNetworkArchitecture<float>(
    inputType: InputType.OneDimensional,
    taskType: NeuralNetworkTaskType.Regression,
    inputSize: 16000,
    outputSize: 16000);
            
var builder = new AiModelBuilder<float, Tensor<float>, Tensor<float>>()
    .ConfigureModel(new ConvTasNet<float>(architecture, "conv_tasnet.onnx",
        new ConvTasNetOptions { SampleRate = 8000, NumSources = 2 }));
            
var trainingData = Tensor<float>.CreateRandom(4, 16000);
var trainingLabels = Tensor<float>.CreateRandom(4, 16000);
var mixedAudioTensor = Tensor<float>.CreateRandom(1, 16000);

// Build and use the model through the facade
var result = builder.Build(trainingData, trainingLabels);
var prediction = result.Predict(mixedAudioTensor);

Constructors

Constructor Summary
ConvTasNet(NeuralNetworkArchitecture<T> architecture, ConvTasNetOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer, ILossFunction<T> lossFunction) Initializes a new instance of the ConvTasNet class for native training mode.
ConvTasNet(NeuralNetworkArchitecture<T> architecture, string modelPath, ConvTasNetOptions options, OnnxModelOptions onnxOptions) Initializes a new instance of the ConvTasNet class for native training mode.

Properties

Property Summary
double EnhancementStrength { get; set; }
int EncoderDimension { get; } Gets the encoder dimension (number of basis functions).
int EncoderKernelSize { get; } Gets the encoder kernel size (window length in samples).
int LatencySamples { get; set; }
int NumChannels { get; }
int NumSources { get; } Gets the number of sources the network separates.

Methods

Method Summary
Dictionary<string, Tensor<T>> GetNamedLayerActivations(Tensor<T> input) Captures every named stage of the network for a single input.
ModelMetadata<T> GetModelMetadata()
Tensor<T> Enhance(Tensor<T> audio)
Tensor<T> EnhanceWithReference(Tensor<T> audio, Tensor<T> reference)
Tensor<T> ForwardForTraining(Tensor<T> input)
Tensor<T> ProcessChunk(Tensor<T> audioChunk)
void EstimateNoiseProfile(Tensor<T> noiseOnlyAudio)
void ResetState()