SpiritLM<T>
Models & Types · AiDotNet.TextToSpeech.MultiModal
SpiritLM (Nguyen et al., TACL 2025): an interleaved spoken and written language model. LLaMA 2 7B, continuously
pre-trained on text, speech and word-level interleaved speech-text sequences, reads and writes both modalities, each
span marked [Text] or [Speech].
Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → SpiritLM<T>
For Beginners
SpiritLM writes text and speech in one stream: it can continue a recording as speech or as text, or read text aloud, because it learned both “languages” at once.
How It Works
Speech tokens (§3.1): the 25 Hz mHuBERT’s layer-11 units, consecutive repeats merged, [Hu0] …
[Hu500]. SpiritLM Expressive (§3.2) adds pitch tokens (a VQ-VAE over FCPE’s F0, 12.5 a second) and style tokens
(one a second), deduplicated and interleaved with the units by time.
Speech is voiced by a unit HiFi-GAN trained on Expresso with a duration predictor (Base: 4 speakers and 34 styles; Expressive: the pitch and style tokens too).
Training (§3.3, App. A): next-token prediction over text-only ([Text]…), speech-only
([Speech]…) and aligned sequences whose modality switches at word boundaries, text spans of 10–30 words and
speech spans of 5–15, at a constant 3e-5 with RoPE’s base at 100,000.
Example
var architecture = new NeuralNetworkArchitecture<float>(InputType.OneDimensional, NeuralNetworkTaskType.Regression, inputSize: 1, outputSize: 1);
var model = new SpiritLM<float>(architecture, new SpiritLMOptions());
model.LoadReleasedWeights("spiritlm_model/spirit-lm-base-7b", "speech_tokenizer");
var speech = model.Synthesize("The largest country in the world is");
var continuation = model.Generate(new[] { SpiritLmContent<float>.FromSpeech(recording16kHz) }, SpiritLmModality.Text);
Constructors
| Constructor | Summary |
|---|---|
SpiritLM(NeuralNetworkArchitecture<T> architecture) |
Calls the fully specified overload with the omitted parameters’ defaults. |
SpiritLM(NeuralNetworkArchitecture<T> architecture, SpiritLMOptions options) |
Creates a native, trainable SpiritLM. |
SpiritLM(NeuralNetworkArchitecture<T> architecture, string modelPath, SpiritLMOptions options) |
Creates an ONNX-backed SpiritLM for inference. |
Properties
| Property | Summary |
|---|---|
EnCodec<T> SpeechPropCodec { get; } |
SONAR Expressive’s 25 Hz EnCodec (with SpeechProp; otherwise null): load or train it as a codec model of its own. |
bool IsExpressive { get; } |
Whether this is SpiritLM Expressive. |
bool TrainsOnSampleObjective { get; } |
|
int MaxTextLength { get; } |
The longest sequence the language model takes (4096). |
Methods
| Method | Summary |
|---|---|
IReadOnlyList<SpiritLmContent<T>> Generate(IReadOnlyList<SpiritLmContent<T>> inputs, SpiritLmModality output) |
Generates output after interleaved text and speech (the release’s Spiritlm.generate): the prompt built, the other modality’s tokens forbidden unless the output is arbitrary, and the result decoded — text as it is, speech voiced. |
IReadOnlyList<SpiritLmContent<T>> Generate(string prompt, SpiritLmModality output, int maxNewTokens) |
Generates after a prompt already written as a string ([Text]…, [Speech][Hu…]…). |
LayerInputDomain GetInputDomain(int[] inputShape) |
|
ModelMetadata<T> GetModelMetadata() |
|
T EvaluateTrainingObjective(TtsTrainingSample<T> sample) |
|
T TrainInterleaved(Tensor<T> recording, IReadOnlyList<SpiritLmWord> words) |
One step on an aligned recording interleaved at word boundaries (§3.1’s speech+text data). |
T TrainPitchQuantizer(Tensor<T> recording, SpiritLmPitchExtractor extractor) |
One step of SpiritLM Expressive’s F0 VQ-VAE (§3.2) on 16 kHz speech: its F0 by pYAAPT (the paper’s extractor for this model; extractor to change it), interpolated and normalized by the voiced mean, with the voicing, cut to a multiple of 1… |
T TrainSequence(string sequence) |
One continued pre-training step on a sequence (TextSequence, SpeechSequence, InterleavedSequence), tokenized with <s>. |
T TrainSonarExpressive(Tensor<T> recording, Tensor<T> semantic, bool withSpeechProp) |
One step of SONAR Expressive (§4.1) on a recording and the SONAR embedding semantic ([1024], of its transcript or translation): the decoders’ cross-entropy on its EnCodec units. |
T TrainSonarExpressive(Tensor<T> recording, string transcript, string language, bool withSpeechProp) |
The step of TrainSonarExpressive with the semantic embedding of transcript in language (e.g. |
T TrainSpeech(Tensor<T> recording) |
One step on speech alone (§3.1’s speech-only data). |
T TrainSpeechPropStyle(Tensor<T> recording, int style) |
One step of the style fine-tuning (§3.2): the SpeechProp features of each second of a recording classified into Expresso’s styles by a linear head, cross-entropy against style; SpeechProp’s encoder trains with the head. |
T TrainText(string text) |
One step on text alone (§3.1’s text-only data). |
T TrainVocoder(Tensor<T> recording) |
Calls the fully specified overload with the omitted parameters’ defaults. |
T TrainVocoder(Tensor<T> recording, int speaker) |
Calls the fully specified overload with the omitted parameters’ defaults. |
T TrainVocoder(Tensor<T> recording, int speaker, int? style) |
One GAN step of the unit vocoder (speech-resynthesis’ and Expresso’s train.py) on 16 kHz speech of a speaker: its frame-level units (with the per-frame pitch and style tokens for Expressive, or the utterance style for Base) and audio cu… |
T TrainVocoder(Tensor<T> recording, int? style) |
Calls the fully specified overload with the omitted parameters’ defaults. |
T TrainVocoder(Tensor<T> recording, int[] codes, int[] pitch, int[] styles, int speaker, int? style) |
The vocoder step on a recording whose speech tokens are already extracted (SpeechUnits), as the release’s training reads them from a manifest: frame-level HuBERT units and, for Expressive, the pitch and style tokens at their own rates. |
Tensor<T> SonarEmbedding(string text, string language) |
SONAR’s sentence embedding [1024] of text. |
Tensor<T> SonarExpressiveSpeech(Tensor<T> semantic, Tensor<T> styleReference, int maxFrames) |
Speech of a SONAR embedding and a SpeechProp embedding (from styleReference, or none): SONAR Expressive’s units (top-k 10, guidance 3), decoded by its EnCodec. |
Tensor<T> Speak(string speechTokens, int? speaker) |
The speech of a token string (the release’s decode): its HuBERT units, with the pitch and style each carries, voiced with duration prediction as speaker (Speaker by default) and, for Base, VocoderStyle. |
Tensor<T> SpeechPropStyleFeatures(Tensor<T> recording) |
The SpeechProp style features of 16 kHz audio, without gradients. |
Tensor<T> Synthesize(string text) |
The speech SpiritLM writes after [Text]{text}[Speech] (text-to-speech continuation, App. |
ValueTuple<int[], int[], int[]> SpeechUnits(Tensor<T> recording) |
The speech tokens of 16 kHz audio (the release’s encode_units): per chunk of at most 100 seconds (chunks under 1280 samples skipped), the HuBERT units and, for Expressive, the pitch and style tokens. |
int[] Encode(string text, bool addBos) |
The ids of a string as the release’s LlamaTokenizer (legacy, add_bos_token) encodes it. |
int[] TokenizeText(string text) |
The token ids of text (no <s>). |
string Decode(IEnumerable<int> ids) |
The string of ids (<s> and </s> skipped). |
string InterleavedSequence(Tensor<T> recording, IReadOnlyList<SpiritLmWord> words, Random random) |
An interleaved training sequence (§3.1, App. |
string Prompt(IReadOnlyList<SpiritLmContent<T>> inputs, SpiritLmModality output) |
The prompt of interleaved inputs (the release’s _build_prompt): each run of a modality opened by its marker, speech as its token string, and the output modality’s marker when the last input differs. |
string SpeechSequence(Tensor<T> recording) |
A speech-only training sequence (§3.1): [Speech]{speech tokens}. |
string SpeechString(Tensor<T> recording) |
The speech-token string of 16 kHz audio (the release’s encode_string). |
string TextSequence(string text) |
A text-only training sequence (§3.1): [Text]{text}. |
void FitStyleCentroids(IEnumerable<Tensor<T>> recordings, int iterations) |
Fits the style k-means (§3.2: 100 clusters of the normalized features) on the SpeechProp style features of recordings: k-means++ seeding, then Lloyd’s iterations until no assignment changes or iterations have run. |
void LoadFcpe(string path) |
Loads FCPE’s weights, torchfcpe’s bundled assets/fcpe_c_v001.pt (SpiritLM Expressive with Fcpe). |
void LoadReleasedWeights(string modelDirectory, string speechTokenizerDirectory) |
Loads a released SpiritLM: the language model of modelDirectory (spirit-lm-base-7b or spirit-lm-expressive-7b; its tokenizer.model unless TokenizerPath is set) and, with speechTokenizerDirectory, the release’s speech_tokenizer… |
void LoadSonarTextEncoder(string directory) |
Loads SONAR’s text encoder (transformers’ M2M100Encoder, e.g. |
void LoadSpeechTokenizer(string directory) |
Loads the release’s speech_tokenizer directory: the 25 Hz mHuBERT and its linear quantizer (hubert_25hz), the variant’s vocoder (hifigan_spiritlm_base or hifigan_spiritlm_expressive_w2v2) and, for Expressive, the F0 VQ-VAE (`vqvae_f… |
void LoadUnitKMeans(string path) |
Loads the k-means centroids of KMeans: a joblib-pickled scikit-learn model (as TWIST’s and textlesslib’s tokenizers ship them) or a torch file with cluster_centers (the release’s KmeansModel format). |