SpiritLMOptions

Options & Configuration · AiDotNet.TextToSpeech.MultiModal

Options for SpiritLM.

Inheritance: ModelOptions → SpiritLMOptions

For Beginners

SpiritLM is a text language model that also learned a “language” of speech tokens. Text is marked [Text], speech [Speech], and it can switch between them mid-sentence, so it can continue speech as speech or text, read text aloud, or transcribe.

How It Works

The defaults are the paper’s SpiritLM Base 7B: LLaMA 2 7B (32 layers, d 4096, 32 heads, SwiGLU 11008, RMSNorm ε 1e-5) with RoPE’s base raised to 100,000 (App. A) and its vocabulary extended by [Text], [Speech] and 501 HuBERT tokens (32,512 rows in the release); the 25 Hz mHuBERT’s layer 11 for units; the unit HiFi-GAN trained on Expresso with 4 speakers (and 34 styles); generation at temperature 0.8 and top-p 0.95 with at most 50 tokens of text, 200 of speech from text and 300 of speech from speech (App. D); continued pre-training at a constant rate of 3e-5 over 4,096-token sequences (App. A).

Expressive is SpiritLM Expressive: 64 pitch and 100 style tokens added (32,768 rows), FCPE F0 and a pitch VQ-VAE, the released wav2vec 2.0 style classifier, and a vocoder that reads both.

Constructors

Constructor Summary
SpiritLMOptions() Creates SpiritLM Base 7B.

Properties

Property Summary
EnCodecOptions SpeechPropCodec { get; set; } SONAR Expressive’s EnCodec (§3.1): 16 kHz, SEANet ratios 8·5·4·4 (25 frames a second), 128-dimensional latents, 8 codebooks of 1024 (2 kbps).
OnnxModelOptions OnnxOptions { get; set; } Settings for the exported ONNX graph runtime.
SpiritLmPitchExtractor PitchExtractor { get; set; }
SpiritLmStyleTokenizer StyleTokenizer { get; set; } The style tokenizer: the release’s classifier by default, or the paper’s SpeechProp features.
SpiritLmUnitQuantizer UnitQuantizer { get; set; }
SpiritLmVariant Variant { get; set; } Base or Expressive.
bool IncludeSonarTextEncoder { get; set; } Whether to build SONAR’s text encoder (NLLB-200’s 1.3B encoder) for the decoders’ semantic conditioning; without it, training takes SONAR embeddings computed elsewhere.
bool IncludeVocoderDiscriminators { get; set; } Whether to build HiFi-GAN’s discriminators, which training the vocoder needs.
double DropoutRate { get; set; } The SONAR Expressive training dropout.
double LearningRate { get; set; } The language model’s AdamW learning rate.
double MaxGradientNorm { get; set; } The language model’s optimizer: LLaMA 2’s AdamW (β 0.9 and 0.95, ε 1e-5, weight decay 0.1, clipping 1.0) at the constant LearningRate.
double RmsEpsilon { get; set; } RMSNorm ε (1e-5) and the rotary base (100,000, App.
double RopeTheta { get; set; }
double SpeechPropLearningRate { get; set; } The rate of SONAR Expressive’s and the style fine-tuning’s AdamW (1e-4; neither paper states one).
double Temperature { get; set; } Sampling (App.
double TopP { get; set; }
double VocoderLearningRate { get; set; } The vocoder’s and the pitch VQ-VAE’s rate (2e-4, AdamW β 0.8 and 0.99), decayed by 0.999 per epoch of VocoderUpdatesPerEpoch updates.
double VocoderLearningRateDecay { get; set; }
double WeightDecay { get; set; } The language model’s AdamW weight decay.
int ExpressoStyles { get; set; } Expresso’s style classes the SpeechProp features are fine-tuned to predict (the release’s 34-style list).
int FcpeBins { get; set; }
int FcpeLayers { get; set; }
int FcpeMels { get; set; } FCPE’s mel bands (128), width (512), layers (6) and salience bins (360).
int FcpeWidth { get; set; }
int HiddenDim { get; set; } The LLaMA decoder’s hidden width.
int HubertConvChannels { get; set; }
int HubertHeads { get; set; }
int HubertIntermediate { get; set; }
int HubertLayer { get; set; }
int HubertWidth { get; set; } The 25 Hz mHuBERT’s width (768), layer read (11), heads (12), feed-forward width (3072) and feature-encoder channels (512); its convolutions stride 5·2⁷ = 640 samples.
int Intermediate { get; set; }
int Layers { get; set; } LLaMA 2 layers (32) and SwiGLU width (11008).
int MaxNewSpeechTokensFromSpeech { get; set; }
int MaxNewSpeechTokensFromText { get; set; }
int MaxNewTextTokens { get; set; } The most tokens a generation adds (App.
int MaxSpeechSpanWords { get; set; }
int MaxTextLength { get; set; } The training and generation context limit in tokens.
int MaxTextSpanWords { get; set; }
int MinSpeechSpanWords { get; set; }
int MinTextSpanWords { get; set; } The word spans of interleaved training (App.
int NumHeads { get; set; } The LLaMA decoder’s attention heads.
int PitchQuantizerLatent { get; set; }
int PitchQuantizerWidth { get; set; } The pitch VQ-VAE’s width (32) and latent width (128).
int PitchTokens { get; set; } Pitch tokens (64) and the F0 extractor feeding the pitch VQ-VAE.
int SampleRate { get; set; } The speech tokenizer’s waveform sample rate.
int SonarFeedForward { get; set; }
int SonarHeads { get; set; }
int SonarLayers { get; set; }
int SonarVocabSize { get; set; } SONAR’s text encoder: vocabulary (256,206), width (1024), layers (24), heads (16), feed-forward (8192).
int SonarWidth { get; set; }
int Speaker { get; set; }
int SpeechPropDecoderLayers { get; set; }
int SpeechPropDim { get; set; } The SpeechProp embedding’s size (1024, as SONAR’s).
int SpeechPropEncoderLayers { get; set; } The SpeechProp encoder’s layers (12) and each decoder’s (16).
int SpeechPropFeedForward { get; set; }
int SpeechPropHeads { get; set; }
int SpeechPropWidth { get; set; } The SpeechProp encoder’s and the decoders’ width (1024), heads (16) and feed-forward width (4096); the paper states only the depths (12 and 16).
int StyleEncoderConvChannels { get; set; }
int StyleEncoderHeads { get; set; }
int StyleEncoderIntermediate { get; set; }
int StyleEncoderLayers { get; set; }
int StyleEncoderProjection { get; set; }
int StyleEncoderWidth { get; set; }
int StyleTokens { get; set; } Style tokens (100) and the style classifier’s wav2vec 2.0 width (768), layers (12), heads (12), feed-forward width (3072), convolution channels (512) and projection (256).
int TextVocabulary { get; set; } LLaMA’s text vocabulary (32,000); the markers and speech tokens follow it.
int TopK { get; set; } Top-k before top-p: the release builds a fresh GenerationConfig, so transformers’ default of 50 applies (the paper names only top-p); 0 samples from the nucleus alone.
int Units { get; set; } HuBERT units (500) and their quantizer; the token vocabulary holds one more (the CTC blank).
int VocabSize { get; set; } The joint text and speech token vocabulary size.
int VocoderChannels { get; set; }
int VocoderEmbeddingDim { get; set; } The vocoder’s embedding width (128), first width (512), upsampling (5·4·4·4·2 = 640 samples per unit, 25 Hz) and kernels.
int VocoderSegmentSize { get; set; } The vocoder’s training segment in samples (the Base config’s 8960, 14 units; the Expressive config’s 17920).
int VocoderSpeakers { get; set; } The vocoder’s speakers (Expresso’s 4) and the speaker synthesis uses (the release’s default, 2).
int VocoderStyle { get; set; }
int VocoderStyles { get; set; } The Base vocoder’s utterance styles (34, its styles.txt; 0 for none) and the style synthesis uses (the release’s default, 8, “conv-default”).
int VocoderUpdatesPerEpoch { get; set; }
int[] VocoderUpsampleKernels { get; set; }
int[] VocoderUpsampleRates { get; set; }
string ModelPath { get; set; } The exported ONNX graph path, when a graph is loaded.
string SonarTokenizerDirectory { get; set; } A directory with NLLB’s sentencepiece.bpe.model and special_tokens_map.json.
string TokenizerPath { get; set; } The release’s SentencePiece tokenizer.model, whose user-defined pieces are the modality markers and speech tokens; text needs it.

Methods

Method Summary
SpiritLMOptions Expressive() SpiritLM Expressive 7B: pitch and style tokens, a 32,768-row vocabulary, the expressive vocoder.