VoiceCraft<T>

Models & Types · AiDotNet.TextToSpeech.CodecBased

VoiceCraft: causal-masked, delayed-stacked neural-codec speech infilling and zero-shot TTS.

Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → VoiceCraft<T>

Implements: ICodecTts<T>

How It Works

The native graph implements Peng et al. (2024) and the authors’ release: the frozen 16 kHz GigaSpeech EnCodec (four 2048-entry RVQ codebooks at 50 Hz), the released IPA front end, causal masking, delayed stacking, learned mask embeddings, a pre-norm decoder-only Transformer, four independent two-layer codebook heads and the weighted likelihood objective. Text-to-speech is the paper’s insertion-at-the-end case; EditSpeech performs general infilling while preserving unmasked codec frames exactly.

Constructors

Constructor Summary
VoiceCraft(NeuralNetworkArchitecture<T> architecture, VoiceCraftOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) Creates the native EnCodec and VoiceCraft Transformer.
VoiceCraft(NeuralNetworkArchitecture<T> architecture, string modelPath, VoiceCraftOptions options) Creates the native EnCodec and VoiceCraft Transformer.

Properties

Property Summary
bool CodecCodebooksInitialized { get; }
bool HasPaperPipeline { get; }
bool TrainsOnSampleObjective { get; }
int CodebookSize { get; }
int CodecFrameRate { get; }
int CodecTokenCodebooks { get; }
int CodecTokenVocabulary { get; }
int MaxTextLength { get; }
int NumCodebooks { get; }
int PhoneVocabularySize { get; }
int TransformerLayerCount { get; }

Methods

Method Summary
LayerInputDomain GetInputDomain(int[] inputShape)
ModelMetadata<T> GetModelMetadata()
T EvaluateTrainingObjective(TtsTrainingSample<T> sample)
Tensor<T> DecodeFromTokens(Tensor<T> tokens) Decodes [frames, four codebooks] or flattened frame-major codes through EnCodec.
Tensor<T> DelayCodecTokens(Tensor<T> tokens) Exposes the paper’s delayed stacking for diagnostics: codebook q is shifted by q rows.
Tensor<T> EditSpeech(string targetTranscript, Tensor<T> originalAudio, IReadOnlyList<ValueTuple<int, int>> maskSpans, int sampleRate) Infills the specified codec-frame spans and returns a decoded waveform.
Tensor<T> EncodeToTokens(Tensor<T> audio) Encodes audio as [frames, four codebooks] VoiceCraft tokens.
Tensor<T> EncodeToTokens(Tensor<T> audio, int sampleRate)
Tensor<T> Synthesize(string text) Performs the paper’s zero-shot TTS case: an insertion after the enrolled prompt.
Tensor<T> SynthesizeWithVoice(string text, Tensor<T> recording, string transcript, int sampleRate) Synthesizes with a prompt without changing the configured voice.
Tensor<T> TokenizeText(string text) Runs the release’s eSpeak-style IPA front end and exact checkpoint phn2num table.
TtsVoice<T> CreateVoice(Tensor<T> recording, string transcript, int sampleRate) Enrolls the prompt recording and its transcript, both required by VoiceCraft continuation.
void LoadAuthorWeights(string transformerPath, string codecPath, string codecPrefix) Loads the authors’ named Transformer and GigaSpeech codec state dictionaries.
void LoadConvertedWeights(string path) Copies {ConvertedParameterTensor} from source straight into a parameter vector.
void SetPhoneVocabulary(IReadOnlyDictionary<string, int> vocabulary) Sets the original checkpoint’s phn2num mapping without changing its token IDs.