StepAudio<T>

Models & Types · AiDotNet.TextToSpeech.MultiModal

Step-Audio (Huang et al. 2025): speech understanding and generation for voice interaction. A dual-codebook tokenizer turns speech into linguistic and semantic tokens, the 130B Step-1 chat model answers in text (AQTA), and the 3B Step-1 TTS model writes the answer’s dual-codebook tokens, which CosyVoice’s flow and HiFT turn into 22.05 kHz speech.

Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → StepAudio<T>

For Beginners

Step-Audio listens, thinks in text with a very large language model, and answers aloud in a voice you give it a sample of; tags like “(高兴1)” or “(RAP)” at the start of the text set the style.

How It Works

Built as the release (stepfun-ai/Step-Audio) runs it; see StepAudioOptions and docs/text-to-speech/paper-fidelity-decisions.md. Speech goes in at any rate (resampled to 16 kHz for the tokenizer) and comes out at 22.05 kHz. The TTS speaks as SpeakerVoice, a speaker made from a prompt recording and its transcript (CreateSpeaker).

Constructors

Constructor Summary
StepAudio(NeuralNetworkArchitecture<T> architecture) Calls the fully specified overload with the omitted parameters’ defaults.
StepAudio(NeuralNetworkArchitecture<T> architecture, StepAudioOptions options) Creates Step-Audio with its networks (randomly initialized).
StepAudio(NeuralNetworkArchitecture<T> architecture, string modelPath, StepAudioOptions options) Creates Step-Audio running an ONNX export.

Properties

Property Summary
StepAudioSpeaker<T> SpeakerVoice { get; set; } The speaker the TTS speaks as; Synthesize needs one.
bool TrainsOnSampleObjective { get; }
int MaxTextLength { get; } The longest answer, in tokens.

Methods

Method Summary
LayerInputDomain GetInputDomain(int[] inputShape)
ModelMetadata<T> GetModelMetadata()
StepAudioSpeaker<T> CreateSpeaker(Tensor<T> recording, int sampleRate, string transcript, string name) A speaker from a prompt recording (any rate) and its transcript, named as the release’s system prompt names speakers (empty for the prompt without a name).
T EvaluateTrainingObjective(TtsTrainingSample<T> sample)
T TrainChat(Tensor<T> question, int sampleRate, string answer, IReadOnlyList<ValueTuple<string, string>> history) One AQTA fine-tuning step (§5.2.2) on the chat model: the chat prompt with earlier turns and the spoken question, then the answer’s tokens and <|EOT|>, which alone are scored.
T TrainContinuation(Tensor<T> recording, int sampleRate, bool tts) A pre-training step on a recording’s audio continuation (its dual-codebook tokens).
T TrainPpo(Tensor<T> question, int sampleRate) One PPO rollout (§5.2.6) on a spoken question: the policy (the chat model) samples an answer; the per-token reward is −β (log π − log π_ref) with the reward model’s score added at the last token; GAE (γ, λ) gives the advantages (whitened) a…
T TrainReward(Tensor<T> question, int sampleRate, string chosen, string rejected) One reward-model step (§5.2.4): the Bradley–Terry loss −log σ(r(chosen) − r(rejected)) on two answers to the same spoken question.
T TrainSequence(string sequence, bool tts) One pre-training step (§4.2) on a sequence of text and <audio_N> tokens (pure audio continuation: AudioTokenString; interleaved, ASR and TTS data: the caller’s layout, whose format the paper does not state), on the chat model or, with `tt…
T TrainTts(string text, Tensor<T> recording, int sampleRate, StepAudioSpeaker<T> speaker) One TTS fine-tuning step (§5.1.2) on the TTS model: the two-turn prompt with the speaker (their transcript and audio tokens) and the human turn’s text, then the recording’s dual-codebook tokens and <|EOT|>, which alone are scored.
Tensor<T> Speak(string text) Speech for text in SpeakerVoice (the release’s TTS), 22.05 kHz.
Tensor<T> Synthesize(string text)
ValueTuple<Tensor<T>, string> Chat(Tensor<T> speech, int sampleRate) The full voice interaction (stepaudio.py’s __call__): the spoken answer to a spoken question, and its text.
double Reward(Tensor<T> question, int sampleRate, string answer) The reward model’s score of an answer to a spoken question (its value at the last token).
int[] TokenizeText(string text) Step-1’s token ids of text (with , as the release encodes).
string Answer(Tensor<T> speech, int sampleRate, IReadOnlyList<ValueTuple<string, string>> history) The chat model’s text answer to a spoken question (AQTA), the question at sampleRate, after optional earlier turns (role, text).
string Answer(string question) The chat model’s text answer to a text question (TQTA).
string AudioTokenString(Tensor<T> recording, int sampleRate) The dual-codebook token string of a recording (“<audio_N>…”, tokenizer.py’s __call__).
void InitializeCriticFromReward() Starts the critic from the trained reward model.
void InitializeRlhfFromChat() Starts the reward model from the current chat model (§5.2.4: “initializes from a SFT model”) and the critic and the frozen reference policy likewise.