MetaVoice1B<T>

Models & Types · AiDotNet.TextToSpeech.VoiceCloning

MetaVoice’s learned-position causal GPT, noncausal codebook GPT, speaker LSTM and EnCodec-conditioned four-band diffusion waveform generator.

Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → MetaVoice1B<T>

Implements: IEndToEndTts<T>, IVoiceCloner<T>

How It Works

The released output includes loudness normalization and DeepFilterNet3 enhancement at 48 kHz. RawDecoderOutput selects the original 24 kHz neural decoder waveform.

Constructors

Constructor Summary
MetaVoice1B(NeuralNetworkArchitecture<T> architecture, MetaVoice1BOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) Constructs the actual native transformer and speaker graphs.
MetaVoice1B(NeuralNetworkArchitecture<T> architecture, string modelPath, MetaVoice1BOptions options) Constructs the actual native transformer and speaker graphs.

Properties

Property Summary
bool HasPaperPipeline { get; } Whether the actual native language and speaker graphs are present.
bool TrainsOnSampleObjective { get; }
double MinReferenceDuration { get; }
int CodecTokenCodebooks { get; }
int CodecTokenVocabulary { get; }
int HiddenDim { get; }
int MaxTextLength { get; }
int NumFlowSteps { get; }
int SpeakerEmbeddingDim { get; }

Methods

Method Summary
LayerInputDomain GetInputDomain(int[] inputShape)
ModelMetadata<T> GetModelMetadata()
T EvaluateTrainingObjective(TtsTrainingSample<T> sample)
Tensor<T> ExtractSpeakerEmbedding(Tensor<T> referenceAudio)
Tensor<T> ExtractSpeakerEmbedding(Tensor<T> referenceAudio, int sampleRate) Runs the original speaker frontend and LSTM, without imposing the synthesis UI’s duration threshold.
Tensor<T> Synthesize(string text)
Tensor<T> SynthesizeWithVoice(string text, Tensor<T> referenceAudio)
Tensor<T> SynthesizeWithVoice(string text, Tensor<T> referenceAudio, int sampleRate) Clones a mono recording at an explicitly specified sample rate.
int[] EncodeText(string text) Encodes normalized text using the actual first-stage checkpoint tokenizer.
void LoadDecoderCheckpoints(string encodecSafetensors, string multibandCheckpoint) Loads actual HF EnCodec and all four original AudioCraft PTH bands.
void LoadEnhancerCheckpoint(string checkpoint) Loads the unmodified DeepFilterNet3 model_120.ckpt.best archive.
void LoadLanguageModelCheckpoints(string firstStage, string secondStage) Loads both unmodified actor PTH files, including their trained byte-BPE metadata.
void LoadPretrainedWeights(string modelDirectory, string encodecSafetensors, string multibandCheckpoint) Loads the original MetaVoice actor directory and its original external decoder assets.
void LoadPretrainedWeights(string modelDirectory, string encodecSafetensors, string multibandCheckpoint, string deepFilterCheckpoint) Loads all actors, the neural decoder and the release’s DeepFilterNet3 enhancer.
void LoadSpeakerEncoderCheckpoint(string checkpoint) Loads all 14 active speaker tensors; the author’s two unused similarity scalars are explicitly recognized.
void SetTrainingMode(bool isTraining)