SpiritLMOptions
Options & Configuration · AiDotNet.TextToSpeech.MultiModal
Options for SpiritLM.
Inheritance: ModelOptions → SpiritLMOptions
For Beginners
SpiritLM is a text language model that also learned a “language” of speech tokens. Text is
marked [Text], speech [Speech], and it can switch between them mid-sentence, so it can continue speech
as speech or text, read text aloud, or transcribe.
How It Works
The defaults are the paper’s SpiritLM Base 7B: LLaMA 2 7B (32 layers, d 4096, 32 heads, SwiGLU 11008, RMSNorm
ε 1e-5) with RoPE’s base raised to 100,000 (App. A) and its vocabulary extended by [Text], [Speech] and
501 HuBERT tokens (32,512 rows in the release); the 25 Hz mHuBERT’s layer 11 for units; the unit HiFi-GAN trained on
Expresso with 4 speakers (and 34 styles); generation at temperature 0.8 and top-p 0.95 with at most 50 tokens of text,
200 of speech from text and 300 of speech from speech (App. D); continued pre-training at a constant rate of 3e-5 over
4,096-token sequences (App. A).
Expressive is SpiritLM Expressive: 64 pitch and 100 style tokens added (32,768 rows), FCPE F0 and a
pitch VQ-VAE, the released wav2vec 2.0 style classifier, and a vocoder that reads both.
Constructors
| Constructor | Summary |
|---|---|
SpiritLMOptions() |
Creates SpiritLM Base 7B. |
Properties
| Property | Summary |
|---|---|
EnCodecOptions SpeechPropCodec { get; set; } |
SONAR Expressive’s EnCodec (§3.1): 16 kHz, SEANet ratios 8·5·4·4 (25 frames a second), 128-dimensional latents, 8 codebooks of 1024 (2 kbps). |
OnnxModelOptions OnnxOptions { get; set; } |
Settings for the exported ONNX graph runtime. |
SpiritLmPitchExtractor PitchExtractor { get; set; } |
|
SpiritLmStyleTokenizer StyleTokenizer { get; set; } |
The style tokenizer: the release’s classifier by default, or the paper’s SpeechProp features. |
SpiritLmUnitQuantizer UnitQuantizer { get; set; } |
|
SpiritLmVariant Variant { get; set; } |
Base or Expressive. |
bool IncludeSonarTextEncoder { get; set; } |
Whether to build SONAR’s text encoder (NLLB-200’s 1.3B encoder) for the decoders’ semantic conditioning; without it, training takes SONAR embeddings computed elsewhere. |
bool IncludeVocoderDiscriminators { get; set; } |
Whether to build HiFi-GAN’s discriminators, which training the vocoder needs. |
double DropoutRate { get; set; } |
The SONAR Expressive training dropout. |
double LearningRate { get; set; } |
The language model’s AdamW learning rate. |
double MaxGradientNorm { get; set; } |
The language model’s optimizer: LLaMA 2’s AdamW (β 0.9 and 0.95, ε 1e-5, weight decay 0.1, clipping 1.0) at the constant LearningRate. |
double RmsEpsilon { get; set; } |
RMSNorm ε (1e-5) and the rotary base (100,000, App. |
double RopeTheta { get; set; } |
|
double SpeechPropLearningRate { get; set; } |
The rate of SONAR Expressive’s and the style fine-tuning’s AdamW (1e-4; neither paper states one). |
double Temperature { get; set; } |
Sampling (App. |
double TopP { get; set; } |
|
double VocoderLearningRate { get; set; } |
The vocoder’s and the pitch VQ-VAE’s rate (2e-4, AdamW β 0.8 and 0.99), decayed by 0.999 per epoch of VocoderUpdatesPerEpoch updates. |
double VocoderLearningRateDecay { get; set; } |
|
double WeightDecay { get; set; } |
The language model’s AdamW weight decay. |
int ExpressoStyles { get; set; } |
Expresso’s style classes the SpeechProp features are fine-tuned to predict (the release’s 34-style list). |
int FcpeBins { get; set; } |
|
int FcpeLayers { get; set; } |
|
int FcpeMels { get; set; } |
FCPE’s mel bands (128), width (512), layers (6) and salience bins (360). |
int FcpeWidth { get; set; } |
|
int HiddenDim { get; set; } |
The LLaMA decoder’s hidden width. |
int HubertConvChannels { get; set; } |
|
int HubertHeads { get; set; } |
|
int HubertIntermediate { get; set; } |
|
int HubertLayer { get; set; } |
|
int HubertWidth { get; set; } |
The 25 Hz mHuBERT’s width (768), layer read (11), heads (12), feed-forward width (3072) and feature-encoder channels (512); its convolutions stride 5·2⁷ = 640 samples. |
int Intermediate { get; set; } |
|
int Layers { get; set; } |
LLaMA 2 layers (32) and SwiGLU width (11008). |
int MaxNewSpeechTokensFromSpeech { get; set; } |
|
int MaxNewSpeechTokensFromText { get; set; } |
|
int MaxNewTextTokens { get; set; } |
The most tokens a generation adds (App. |
int MaxSpeechSpanWords { get; set; } |
|
int MaxTextLength { get; set; } |
The training and generation context limit in tokens. |
int MaxTextSpanWords { get; set; } |
|
int MinSpeechSpanWords { get; set; } |
|
int MinTextSpanWords { get; set; } |
The word spans of interleaved training (App. |
int NumHeads { get; set; } |
The LLaMA decoder’s attention heads. |
int PitchQuantizerLatent { get; set; } |
|
int PitchQuantizerWidth { get; set; } |
The pitch VQ-VAE’s width (32) and latent width (128). |
int PitchTokens { get; set; } |
Pitch tokens (64) and the F0 extractor feeding the pitch VQ-VAE. |
int SampleRate { get; set; } |
The speech tokenizer’s waveform sample rate. |
int SonarFeedForward { get; set; } |
|
int SonarHeads { get; set; } |
|
int SonarLayers { get; set; } |
|
int SonarVocabSize { get; set; } |
SONAR’s text encoder: vocabulary (256,206), width (1024), layers (24), heads (16), feed-forward (8192). |
int SonarWidth { get; set; } |
|
int Speaker { get; set; } |
|
int SpeechPropDecoderLayers { get; set; } |
|
int SpeechPropDim { get; set; } |
The SpeechProp embedding’s size (1024, as SONAR’s). |
int SpeechPropEncoderLayers { get; set; } |
The SpeechProp encoder’s layers (12) and each decoder’s (16). |
int SpeechPropFeedForward { get; set; } |
|
int SpeechPropHeads { get; set; } |
|
int SpeechPropWidth { get; set; } |
The SpeechProp encoder’s and the decoders’ width (1024), heads (16) and feed-forward width (4096); the paper states only the depths (12 and 16). |
int StyleEncoderConvChannels { get; set; } |
|
int StyleEncoderHeads { get; set; } |
|
int StyleEncoderIntermediate { get; set; } |
|
int StyleEncoderLayers { get; set; } |
|
int StyleEncoderProjection { get; set; } |
|
int StyleEncoderWidth { get; set; } |
|
int StyleTokens { get; set; } |
Style tokens (100) and the style classifier’s wav2vec 2.0 width (768), layers (12), heads (12), feed-forward width (3072), convolution channels (512) and projection (256). |
int TextVocabulary { get; set; } |
LLaMA’s text vocabulary (32,000); the markers and speech tokens follow it. |
int TopK { get; set; } |
Top-k before top-p: the release builds a fresh GenerationConfig, so transformers’ default of 50 applies (the paper names only top-p); 0 samples from the nucleus alone. |
int Units { get; set; } |
HuBERT units (500) and their quantizer; the token vocabulary holds one more (the CTC blank). |
int VocabSize { get; set; } |
The joint text and speech token vocabulary size. |
int VocoderChannels { get; set; } |
|
int VocoderEmbeddingDim { get; set; } |
The vocoder’s embedding width (128), first width (512), upsampling (5·4·4·4·2 = 640 samples per unit, 25 Hz) and kernels. |
int VocoderSegmentSize { get; set; } |
The vocoder’s training segment in samples (the Base config’s 8960, 14 units; the Expressive config’s 17920). |
int VocoderSpeakers { get; set; } |
The vocoder’s speakers (Expresso’s 4) and the speaker synthesis uses (the release’s default, 2). |
int VocoderStyle { get; set; } |
|
int VocoderStyles { get; set; } |
The Base vocoder’s utterance styles (34, its styles.txt; 0 for none) and the style synthesis uses (the release’s default, 8, “conv-default”). |
int VocoderUpdatesPerEpoch { get; set; } |
|
int[] VocoderUpsampleKernels { get; set; } |
|
int[] VocoderUpsampleRates { get; set; } |
|
string ModelPath { get; set; } |
The exported ONNX graph path, when a graph is loaded. |
string SonarTokenizerDirectory { get; set; } |
A directory with NLLB’s sentencepiece.bpe.model and special_tokens_map.json. |
string TokenizerPath { get; set; } |
The release’s SentencePiece tokenizer.model, whose user-defined pieces are the modality markers and speech tokens; text needs it. |
Methods
| Method | Summary |
|---|---|
SpiritLMOptions Expressive() |
SpiritLM Expressive 7B: pitch and style tokens, a 32,768-row vocabulary, the expressive vocoder. |