ConvTasNet<T>
Models & Types · AiDotNet.Audio.Enhancement
Conv-TasNet: A fully-convolutional time-domain audio separation network.
Inheritance: NeuralNetworkBase<T> → AudioNeuralNetworkBase<T> → ConvTasNet<T>
Implements: IAudioEnhancer<T>
For Beginners
Conv-TasNet is like having multiple microphones that each focus on one speaker in a noisy room. Give it a recording with multiple people talking, and it separates them into individual clean tracks!
Traditional methods convert audio to frequency domain, process it, then convert back. Conv-TasNet works directly on the waveform, which avoids problems with phase reconstruction and often produces cleaner results.
Common use cases:
- Separating speakers in meeting recordings
- Isolating vocals from music
- Removing background noise
- Speech enhancement for hearing aids
- Denoising phone calls
How It Works
Conv-TasNet (Convolutional Time-domain Audio Separation Network) is a pioneering neural network architecture that operates directly in the time domain, avoiding the phase reconstruction problems of frequency-domain methods.
The architecture consists of three main components:
- Encoder: Converts waveform to a learned representation using 1D convolutions
- Separator: Temporal Convolutional Network (TCN) that estimates source masks
- Decoder: Reconstructs separated waveforms from masked representations
Reference: Luo, Y., & Mesgarani, N. (2019). Conv-TasNet: Surpassing Ideal Time-Frequency Magnitude Masking for Speech Separation.
Example
// Use AiModelBuilder facade for audio source separation
var architecture = new NeuralNetworkArchitecture<float>(
inputType: InputType.OneDimensional,
taskType: NeuralNetworkTaskType.Regression,
inputSize: 16000,
outputSize: 16000);
var builder = new AiModelBuilder<float, Tensor<float>, Tensor<float>>()
.ConfigureModel(new ConvTasNet<float>(architecture, "conv_tasnet.onnx",
new ConvTasNetOptions { SampleRate = 8000, NumSources = 2 }));
var trainingData = Tensor<float>.CreateRandom(4, 16000);
var trainingLabels = Tensor<float>.CreateRandom(4, 16000);
var mixedAudioTensor = Tensor<float>.CreateRandom(1, 16000);
// Build and use the model through the facade
var result = builder.Build(trainingData, trainingLabels);
var prediction = result.Predict(mixedAudioTensor);
Constructors
| Constructor | Summary |
|---|---|
ConvTasNet(NeuralNetworkArchitecture<T> architecture, ConvTasNetOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer, ILossFunction<T> lossFunction) |
Initializes a new instance of the ConvTasNet class for native training mode. |
ConvTasNet(NeuralNetworkArchitecture<T> architecture, string modelPath, ConvTasNetOptions options, OnnxModelOptions onnxOptions) |
Initializes a new instance of the ConvTasNet class for native training mode. |
Properties
| Property | Summary |
|---|---|
double EnhancementStrength { get; set; } |
|
int EncoderDimension { get; } |
Gets the encoder dimension (number of basis functions). |
int EncoderKernelSize { get; } |
Gets the encoder kernel size (window length in samples). |
int LatencySamples { get; set; } |
|
int NumChannels { get; } |
|
int NumSources { get; } |
Gets the number of sources the network separates. |
Methods
| Method | Summary |
|---|---|
Dictionary<string, Tensor<T>> GetNamedLayerActivations(Tensor<T> input) |
Captures every named stage of the network for a single input. |
ModelMetadata<T> GetModelMetadata() |
|
Tensor<T> Enhance(Tensor<T> audio) |
|
Tensor<T> EnhanceWithReference(Tensor<T> audio, Tensor<T> reference) |
|
Tensor<T> ForwardForTraining(Tensor<T> input) |
|
Tensor<T> ProcessChunk(Tensor<T> audioChunk) |
|
void EstimateNoiseProfile(Tensor<T> noiseOnlyAudio) |
|
void ResetState() |