MARS5TTS<T>
Models & Types · AiDotNet.TextToSpeech.CodecBased
MARS5 (Camb.ai, 2024): two-stage voice cloning over EnCodec’s 6 kbps codes. An autoregressive Mistral decoder writes the first codebook as BPE-merged speech tokens after the text, conditioned on an implicit speaker embedding of the reference’s codes; a non-autoregressive encoder-decoder Transformer fills the other seven codebooks by multinomial diffusion; Vocos decodes them.
Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → MARS5TTS<T>
For Beginners
MARS5 reads text in the voice of a short recording; with the recording’s transcript it copies the voice more closely.
How It Works
The pipeline is the release’s Mars5TTS.tts: the text in minbpe tokens between start and end tokens (the
reference’s transcript first for a deep clone), the reference’s EnCodec codes for the speaker embedding (and, deep
cloned, its first codebook’s speech tokens continued), AR sampling with frequency, presence and early-end penalties,
then 200 steps of guided diffusion with the first codebook held (revisable in the last 20 steps), Vocos at the 3 kbps
class, and the output’s silences trimmed.
MARS5 has no paper, and the release ships no training code. Training follows its architecture notes (the AR’s cross-entropy with text tokens down-weighted, the NAR’s multinomial diffusion with the first codebook given) and its diffusion code (the variational loss), with the recipes its checkpoints’ metadata records.
Example
var architecture = new NeuralNetworkArchitecture<float>(InputType.OneDimensional, NeuralNetworkTaskType.Regression, inputSize: 1, outputSize: 1);
var model = new MARS5TTS<float>(architecture, MARS5TTSOptions.FromCheckpoint("mars5_ar.safetensors"));
model.LoadReferenceWeights(new Mars5Checkpoints { Autoregressive = "mars5_ar.safetensors", NonAutoregressive = "mars5_nar.safetensors",
Codec = "encodec_24khz/model.safetensors", Vocoder = "vocos-encodec-24khz/pytorch_model.bin" });
model.Voice = model.CreateVoice(referenceAudio, 24000, "What the reference says.");
var audio = model.Synthesize("Text to speak in that voice."); // 24 kHz
Constructors
| Constructor | Summary |
|---|---|
MARS5TTS(NeuralNetworkArchitecture<T> architecture) |
Calls the fully specified overload with the omitted parameters’ defaults. |
MARS5TTS(NeuralNetworkArchitecture<T> architecture, MARS5TTSOptions options) |
Creates a native, trainable MARS5. |
MARS5TTS(NeuralNetworkArchitecture<T> architecture, string modelPath, MARS5TTSOptions options) |
Creates an ONNX-backed MARS5 for inference. |
Properties
| Property | Summary |
|---|---|
EnCodec<T> Codec { get; } |
The codec (EnCodec 24 kHz at 6 kbps), frozen. |
bool TrainsOnSampleObjective { get; } |
|
int MaxTextLength { get; } |
Methods
| Method | Summary |
|---|---|
LayerInputDomain GetInputDomain(int[] inputShape) |
|
ModelMetadata<T> GetModelMetadata() |
|
T EvaluateTrainingObjective(TtsTrainingSample<T> sample) |
|
Tensor<T> Synthesize(string text) |
|
TtsVoice<T> CreateVoice(Tensor<T> recording, int sampleRate, string transcript) |
A voice from a recording of the speaker at sampleRate (resampled to 24 kHz, averaged to mono) and, for a deep clone, its transcript. |
void LoadReferenceWeights(Mars5Checkpoints checkpoints) |
Loads the release’s weights: the AR and NAR models (their F16 tensors read as the model’s type), EnCodec’s 24 kHz checkpoint and Vocos’s EnCodec model. |