UniAudio<T>

Models & Types · AiDotNet.TextToSpeech.CodecBased

UniAudio: a universal neural codec and multi-scale autoregressive audio Transformer.

Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → UniAudio<T>

Implements: ICodecTts<T>

How It Works

The native path implements the released zero-shot TTS sequence, the 16 kHz universal SEANet/RVQ codec, and the paper’s two-scale Transformer. Every condition is serialized as three-token patches. The global Transformer models frames; its previous-frame state conditions a local Transformer that generates the three RVQ levels within a frame.

Constructors

Constructor Summary
UniAudio(NeuralNetworkArchitecture<T> architecture, UniAudioOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) Creates the native codec and paper/release multi-scale Transformer.
UniAudio(NeuralNetworkArchitecture<T> architecture, string modelPath, UniAudioOptions options) Creates the native codec and paper/release multi-scale Transformer.

Properties

Property Summary
bool CodecCodebooksInitialized { get; } Whether all universal-codec codebooks have been initialized or restored.
bool HasPaperPipeline { get; } Whether this instance owns the released universal codec and both Transformer scales.
bool TrainsOnSampleObjective { get; }
double LastCodecDiscriminatorLoss { get; set; } The last universal-codec discriminator loss.
double LastCodecGeneratorLoss { get; set; } The last universal-codec generator loss, including commitment.
double MaxGradNormValue { get; }
int CodebookSize { get; }
int CodecFrameRate { get; }
int CodecTokenCodebooks { get; }
int CodecTokenVocabulary { get; }
int GlobalLayerCount { get; } The global Transformer’s layer count.
int LocalLayerCount { get; } The local Transformer’s layer count.
int MaxTextLength { get; }
int NumCodebooks { get; }

Methods

Method Summary
LayerInputDomain GetInputDomain(int[] inputShape)
ModelMetadata<T> GetModelMetadata()
T EvaluateTrainingObjective(TtsTrainingSample<T> sample)
Tensor<T> DecodeFromTokens(Tensor<T> tokens) Decodes flattened, codebook-offset tokens through the universal SEANet decoder.
Tensor<T> EncodeToTokens(Tensor<T> audio) Encodes 16 kHz audio into flattened, codebook-offset UniAudio tokens.
Tensor<T> EncodeToTokens(Tensor<T> audio, int sampleRate) Encodes audio after band-limited resampling to the release’s 16 kHz codec rate.
Tensor<T> Synthesize(string text) Serializes phonemes and speaker prompt, autoregressively generates three RVQ streams, then decodes them.
Tensor<T> SynthesizeWithVoice(string text, Tensor<T> recording, int sampleRate) Synthesizes with a recording without changing the configured voice.
Tensor<T> TokenizePhones(string text) Runs the configured raw-text frontend.
Tensor<T> TokenizeReleasedPhoneSequence(string phones) Encodes a whitespace-separated alignment-phone sequence using the released inventory and TTS duplicate removal.
TtsVoice<T> CreateVoice(Tensor<T> recording, int sampleRate) Creates a three-second-compatible zero-shot speaker prompt from a recording.
void LoadAuthorCodecDiscriminatorWeights(string path) Loads the original universal-codec checkpoint’s trained mel discriminators.
void LoadAuthorWeights(string languageModelPath, string codecPath, string codecPrefix) Loads original named UniAudio multi-scale LM and universal SoundStream codec tensors.
void LoadConvertedWeights(string path) Copies {ConvertedParameterTensor} from source straight into a parameter vector.
void SetCodecTrainingEpoch(int completedEpochs) Sets the completed codec epoch and applies the released epoch-wise exponential scheduler.