VoiceCraft<T>
Models & Types · AiDotNet.TextToSpeech.CodecBased
VoiceCraft: causal-masked, delayed-stacked neural-codec speech infilling and zero-shot TTS.
Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → VoiceCraft<T>
Implements: ICodecTts<T>
How It Works
The native graph implements Peng et al. (2024) and the authors’ release: the frozen 16 kHz GigaSpeech EnCodec
(four 2048-entry RVQ codebooks at 50 Hz), the released IPA front end, causal masking, delayed stacking, learned mask
embeddings, a pre-norm decoder-only Transformer, four independent two-layer codebook heads and the weighted
likelihood objective. Text-to-speech is the paper’s insertion-at-the-end case; EditSpeech performs
general infilling while preserving unmasked codec frames exactly.
Constructors
| Constructor | Summary |
|---|---|
VoiceCraft(NeuralNetworkArchitecture<T> architecture, VoiceCraftOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
Creates the native EnCodec and VoiceCraft Transformer. |
VoiceCraft(NeuralNetworkArchitecture<T> architecture, string modelPath, VoiceCraftOptions options) |
Creates the native EnCodec and VoiceCraft Transformer. |
Properties
| Property | Summary |
|---|---|
bool CodecCodebooksInitialized { get; } |
|
bool HasPaperPipeline { get; } |
|
bool TrainsOnSampleObjective { get; } |
|
int CodebookSize { get; } |
|
int CodecFrameRate { get; } |
|
int CodecTokenCodebooks { get; } |
|
int CodecTokenVocabulary { get; } |
|
int MaxTextLength { get; } |
|
int NumCodebooks { get; } |
|
int PhoneVocabularySize { get; } |
|
int TransformerLayerCount { get; } |
Methods
| Method | Summary |
|---|---|
LayerInputDomain GetInputDomain(int[] inputShape) |
|
ModelMetadata<T> GetModelMetadata() |
|
T EvaluateTrainingObjective(TtsTrainingSample<T> sample) |
|
Tensor<T> DecodeFromTokens(Tensor<T> tokens) |
Decodes [frames, four codebooks] or flattened frame-major codes through EnCodec. |
Tensor<T> DelayCodecTokens(Tensor<T> tokens) |
Exposes the paper’s delayed stacking for diagnostics: codebook q is shifted by q rows. |
Tensor<T> EditSpeech(string targetTranscript, Tensor<T> originalAudio, IReadOnlyList<ValueTuple<int, int>> maskSpans, int sampleRate) |
Infills the specified codec-frame spans and returns a decoded waveform. |
Tensor<T> EncodeToTokens(Tensor<T> audio) |
Encodes audio as [frames, four codebooks] VoiceCraft tokens. |
Tensor<T> EncodeToTokens(Tensor<T> audio, int sampleRate) |
|
Tensor<T> Synthesize(string text) |
Performs the paper’s zero-shot TTS case: an insertion after the enrolled prompt. |
Tensor<T> SynthesizeWithVoice(string text, Tensor<T> recording, string transcript, int sampleRate) |
Synthesizes with a prompt without changing the configured voice. |
Tensor<T> TokenizeText(string text) |
Runs the release’s eSpeak-style IPA front end and exact checkpoint phn2num table. |
TtsVoice<T> CreateVoice(Tensor<T> recording, string transcript, int sampleRate) |
Enrolls the prompt recording and its transcript, both required by VoiceCraft continuation. |
void LoadAuthorWeights(string transformerPath, string codecPath, string codecPrefix) |
Loads the authors’ named Transformer and GigaSpeech codec state dictionaries. |
void LoadConvertedWeights(string path) |
Copies {ConvertedParameterTensor} from source straight into a parameter vector. |
void SetPhoneVocabulary(IReadOnlyDictionary<string, int> vocabulary) |
Sets the original checkpoint’s phn2num mapping without changing its token IDs. |