Voicebox<T>

Models & Types · AiDotNet.TextToSpeech.CodecBased

Voicebox: a text-guided speech infilling model with separate regression-duration and conditional-flow audio models.

Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → Voicebox<T>

Implements: ITtsModel<T>

How It Works

The native path implements Le et al. (2023): position-dependent phones with ghost silences, a duration-context Transformer, an 80-bin normalized log-mel optimal-transport conditional flow, midpoint ODE integration with classifier-free guidance, and the paper’s 16 kHz HiFi-GAN V1 adaptation. It supports text-only diverse synthesis, reference-conditioned zero-shot synthesis, and arbitrary mel infilling. Voicebox is not a neural-codec language model; the previous amplitude quantizer and sinusoidal “decoder” were placeholders and have been removed.

Meta did not release Voicebox weights or source code. Native checkpoints therefore use an explicit AiDotNet conversion schema and are never described as official weights. Architecture details stated in the paper are exact; unpublished ordinary Transformer details use the documented pre-norm implementation in this class.

Constructors

Constructor Summary
Voicebox(NeuralNetworkArchitecture<T> architecture, VoiceboxOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) Creates the paper-native duration, audio-flow, mel, and vocoder graph.
Voicebox(NeuralNetworkArchitecture<T> architecture, string modelPath, VoiceboxOptions options) Creates the paper-native duration, audio-flow, mel, and vocoder graph.

Properties

Property Summary
Func<string, IReadOnlyList<IReadOnlyList<string>>> MfaWordPhonemizer { get; set; } Supplies MFA-compatible phones for each word, before word-position suffixes and ghost silences are added.
bool HasPaperPipeline { get; } Whether the native instance owns the complete paper pipeline.
bool TrainsOnSampleObjective { get; }
int AudioLayerCount { get; } Audio vector-field Transformer block count.
int CodecTokenCodebooks { get; } Voicebox predicts continuous mels and has no codec codebooks.
int CodecTokenVocabulary { get; } Voicebox predicts continuous mels and has no codec vocabulary.
int DurationLayerCount { get; } Regression duration Transformer block count.
int MaxTextLength { get; } Maximum number of phones accepted by the text front end.
int NumberOfFunctionEvaluations { get; } Default vector-field evaluations (two per midpoint step).
int PhoneTableSize { get; } The configured phone inventory size, or zero until a text frontend is selected.

Methods

Method Summary
IReadOnlyList<string> Phonemize(string text) Returns the paper’s ghost-silence and word-position-dependent phone sequence.
LayerInputDomain GetInputDomain(int[] inputShape)
ModelMetadata<T> GetModelMetadata()
T EvaluateTrainingObjective(TtsTrainingSample<T> sample)
Tensor<T> Infill(Tensor<T> normalizedMel, IReadOnlyList<int> framePhoneIds, IReadOnlyList<bool> mask) Infills the frames selected by mask while preserving all context frames exactly.
Tensor<T> Synthesize(string text) Synthesizes either diverse speech or zero-shot speech when Voice is set.
TtsVoice<T> CreateVoice(Tensor<T> recording, int sampleRate, string transcript, int[] durations) Creates the complete zero-shot context.
void LoadConvertedWeights(string path) Copies {ConvertedParameterTensor} from source straight into a parameter vector.
void SetPhoneVocabulary(IReadOnlyDictionary<string, int> vocabulary) Sets a dataset/checkpoint’s exact MFA phone inventory without changing its embedding dimensions.