MARS5TTS<T>

Models & Types · AiDotNet.TextToSpeech.CodecBased

MARS5 (Camb.ai, 2024): two-stage voice cloning over EnCodec’s 6 kbps codes. An autoregressive Mistral decoder writes the first codebook as BPE-merged speech tokens after the text, conditioned on an implicit speaker embedding of the reference’s codes; a non-autoregressive encoder-decoder Transformer fills the other seven codebooks by multinomial diffusion; Vocos decodes them.

Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → MARS5TTS<T>

For Beginners

MARS5 reads text in the voice of a short recording; with the recording’s transcript it copies the voice more closely.

How It Works

The pipeline is the release’s Mars5TTS.tts: the text in minbpe tokens between start and end tokens (the reference’s transcript first for a deep clone), the reference’s EnCodec codes for the speaker embedding (and, deep cloned, its first codebook’s speech tokens continued), AR sampling with frequency, presence and early-end penalties, then 200 steps of guided diffusion with the first codebook held (revisable in the last 20 steps), Vocos at the 3 kbps class, and the output’s silences trimmed.

MARS5 has no paper, and the release ships no training code. Training follows its architecture notes (the AR’s cross-entropy with text tokens down-weighted, the NAR’s multinomial diffusion with the first codebook given) and its diffusion code (the variational loss), with the recipes its checkpoints’ metadata records.

Example

var architecture = new NeuralNetworkArchitecture<float>(InputType.OneDimensional, NeuralNetworkTaskType.Regression, inputSize: 1, outputSize: 1);
var model = new MARS5TTS<float>(architecture, MARS5TTSOptions.FromCheckpoint("mars5_ar.safetensors"));
model.LoadReferenceWeights(new Mars5Checkpoints { Autoregressive = "mars5_ar.safetensors", NonAutoregressive = "mars5_nar.safetensors",
    Codec = "encodec_24khz/model.safetensors", Vocoder = "vocos-encodec-24khz/pytorch_model.bin" });
model.Voice = model.CreateVoice(referenceAudio, 24000, "What the reference says.");
var audio = model.Synthesize("Text to speak in that voice.");   // 24 kHz

Constructors

Constructor Summary
MARS5TTS(NeuralNetworkArchitecture<T> architecture) Calls the fully specified overload with the omitted parameters’ defaults.
MARS5TTS(NeuralNetworkArchitecture<T> architecture, MARS5TTSOptions options) Creates a native, trainable MARS5.
MARS5TTS(NeuralNetworkArchitecture<T> architecture, string modelPath, MARS5TTSOptions options) Creates an ONNX-backed MARS5 for inference.

Properties

Property Summary
EnCodec<T> Codec { get; } The codec (EnCodec 24 kHz at 6 kbps), frozen.
bool TrainsOnSampleObjective { get; }
int MaxTextLength { get; }

Methods

Method Summary
LayerInputDomain GetInputDomain(int[] inputShape)
ModelMetadata<T> GetModelMetadata()
T EvaluateTrainingObjective(TtsTrainingSample<T> sample)
Tensor<T> Synthesize(string text)
TtsVoice<T> CreateVoice(Tensor<T> recording, int sampleRate, string transcript) A voice from a recording of the speaker at sampleRate (resampled to 24 kHz, averaged to mono) and, for a deep clone, its transcript.
void LoadReferenceWeights(Mars5Checkpoints checkpoints) Loads the release’s weights: the AR and NAR models (their F16 tensors read as the model’s type), EnCodec’s 24 kHz checkpoint and Vocos’s EnCodec model.