UniAudio<T>
Models & Types · AiDotNet.TextToSpeech.CodecBased
UniAudio: a universal neural codec and multi-scale autoregressive audio Transformer.
Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → UniAudio<T>
Implements: ICodecTts<T>
How It Works
The native path implements the released zero-shot TTS sequence, the 16 kHz universal SEANet/RVQ codec, and the paper’s two-scale Transformer. Every condition is serialized as three-token patches. The global Transformer models frames; its previous-frame state conditions a local Transformer that generates the three RVQ levels within a frame.
Constructors
| Constructor | Summary |
|---|---|
UniAudio(NeuralNetworkArchitecture<T> architecture, UniAudioOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
Creates the native codec and paper/release multi-scale Transformer. |
UniAudio(NeuralNetworkArchitecture<T> architecture, string modelPath, UniAudioOptions options) |
Creates the native codec and paper/release multi-scale Transformer. |
Properties
| Property | Summary |
|---|---|
bool CodecCodebooksInitialized { get; } |
Whether all universal-codec codebooks have been initialized or restored. |
bool HasPaperPipeline { get; } |
Whether this instance owns the released universal codec and both Transformer scales. |
bool TrainsOnSampleObjective { get; } |
|
double LastCodecDiscriminatorLoss { get; set; } |
The last universal-codec discriminator loss. |
double LastCodecGeneratorLoss { get; set; } |
The last universal-codec generator loss, including commitment. |
double MaxGradNormValue { get; } |
|
int CodebookSize { get; } |
|
int CodecFrameRate { get; } |
|
int CodecTokenCodebooks { get; } |
|
int CodecTokenVocabulary { get; } |
|
int GlobalLayerCount { get; } |
The global Transformer’s layer count. |
int LocalLayerCount { get; } |
The local Transformer’s layer count. |
int MaxTextLength { get; } |
|
int NumCodebooks { get; } |
Methods
| Method | Summary |
|---|---|
LayerInputDomain GetInputDomain(int[] inputShape) |
|
ModelMetadata<T> GetModelMetadata() |
|
T EvaluateTrainingObjective(TtsTrainingSample<T> sample) |
|
Tensor<T> DecodeFromTokens(Tensor<T> tokens) |
Decodes flattened, codebook-offset tokens through the universal SEANet decoder. |
Tensor<T> EncodeToTokens(Tensor<T> audio) |
Encodes 16 kHz audio into flattened, codebook-offset UniAudio tokens. |
Tensor<T> EncodeToTokens(Tensor<T> audio, int sampleRate) |
Encodes audio after band-limited resampling to the release’s 16 kHz codec rate. |
Tensor<T> Synthesize(string text) |
Serializes phonemes and speaker prompt, autoregressively generates three RVQ streams, then decodes them. |
Tensor<T> SynthesizeWithVoice(string text, Tensor<T> recording, int sampleRate) |
Synthesizes with a recording without changing the configured voice. |
Tensor<T> TokenizePhones(string text) |
Runs the configured raw-text frontend. |
Tensor<T> TokenizeReleasedPhoneSequence(string phones) |
Encodes a whitespace-separated alignment-phone sequence using the released inventory and TTS duplicate removal. |
TtsVoice<T> CreateVoice(Tensor<T> recording, int sampleRate) |
Creates a three-second-compatible zero-shot speaker prompt from a recording. |
void LoadAuthorCodecDiscriminatorWeights(string path) |
Loads the original universal-codec checkpoint’s trained mel discriminators. |
void LoadAuthorWeights(string languageModelPath, string codecPath, string codecPrefix) |
Loads original named UniAudio multi-scale LM and universal SoundStream codec tensors. |
void LoadConvertedWeights(string path) |
Copies {ConvertedParameterTensor} from source straight into a parameter vector. |
void SetCodecTrainingEpoch(int completedEpochs) |
Sets the completed codec epoch and applies the released epoch-wise exponential scheduler. |