IrohaStudios Release article
Iroha

How we built
Iroha’s seven controls

Iroha Studios
October 2026
Listen to samplesExplore the methods

Technical notes from October 29, 2026, with the October 30 listening-study snapshot added.

Emotion samples

[happy]

あっ、つい話し込んじゃってたわね。

Oh, I got carried away chatting.
Fish S2-Pro
Iroha-TTS V1

Whole-line emotion

Data

Keep the text and fixed. A generates happy, angry, sad and surprised versions. Retain candidates with the requested emotion, correct words, consistent voice and acceptable quality.

Training

pairs tagged text and reference audio with the filtered target speech. Holding content and voice constant helps the model learn the tag’s acoustic effect instead of guessing emotion from the words.

Inference

The interface maps joy to [happy] and the other emotions to their tags, then uses the same generator. none and neutral add no tag. A leading inline tag overrides the whole-line setting.

Reliability varies by emotion. In the release evaluation, happy speech reached 76.67% emotion accuracy, below the other three classes.

Training

Iroha is based on Fish S2-Pro. Its slow and fast models predict , which a decoder converts to audio. Adaptation focuses on paired data, LoRA fine-tuning and policy optimization.

Output: 44.1 kHz
AaText + reference audioContent and voice
→
≈Predict each frame’s semantic code
→
≋Add 9 residual codes
→
▂▅▇▃▆Audio decoderDecode codes to audio
01

Paired data

A happy tag needs happy audio; [sigh] needs an actual sigh. Constructing and filtering these pairs provides a clear training signal. Relabeling the same recording does not teach a new vocal expression.

02

Supervised fine-tuning

trains the model to predict target speech from the input. adds a small set of trainable parameters to both transformers; these are later merged into the weights. Emotion, vocal events and whispering use different demonstration recipes.

03

Policy optimization

generates 8 candidates per request and scores emotion, text accuracy, voice similarity and quality. It raises the probability of better candidates while a limits drift from the learned behavior. Only the slow model’s MLP is updated; the fast model keeps its SFT weights.

Training configuration

SFT: LoRA on attention and MLP layers in both transformers, rank 32 and alpha 16; 2,700 steps followed by 1,600 more. Policy stage: rsLoRA on the slow MLP only, rank 16 and alpha 64; 12 requests × 8 candidates per batch, one update, KL weight 0.02. Pronunciation control runs before model input; whisper guidance and intensity steering run during inference.

Controls

Emotion transitions

Inline tags control the emotion of the following segment.

[happy] やった、ついに完成したよ！
[sad] …でも、これを見せたかった人は、もういないんだ。

It’s finally finished! …But the person I wanted to show it to is no longer here.

Same-speaker concatenation

Concatenate tagged text and audio codes from two differently expressed clips by the same speaker. Record the boundary and use another clip from that speaker as the reference. The final mixture includes 5,000 general transitions and 4,000 clearer transitions between non-neutral emotions.

Segment rewards

Training checks each segment’s emotion and penalizes boundary pauses longer than 0.6 seconds. At inference, the complete tagged text produces one continuous utterance. Concatenated training examples teach this continuous output.

Not yet evaluated: a complete segment-level evaluation of the release package. Classifying the whole clip as happy does not establish a correct happy-to-sad transition.

Whispering

Use a [whisper] tag or a whispered reference recording.

[whisper] …私の声…どうして、こんな声に………My voice… Why does it sound like this…?

Two input conditions

Each of 1,652 whispered targets has two inputs: (1) an ordinary reference plus [whisper], and (2) a whispered reference with untagged text. Both predict the same target, transferring reference-following behavior to tag-based control.

Guidance

At every step, compute predictions with and without the whisper tag, then use to amplify their difference. The default strength is 1.5 for whispering and 0 otherwise. Both branches share weights and maintain separate generation histories.

Trade-off: whisper detection improves, but voice similarity and predicted quality can decline. This changes vocal production rather than playback volume, and success is not consistent across requests.

Intensity Experimental

Direction extraction

For each emotion, collect with and without its tag on 500 training-side inputs. Average their differences to obtain the direction used for steering.

Activation steering

Add the scaled direction at layers 20 and 22, then restore the original . The four coefficients are −0.06, −0.03, 0 and 0.045. Zero is ordinary tagged generation.

Limits: nonzero intensity supports only whole-line emotion settings and cannot be combined with inline emotion or event tags. Only 70.83% of 72 contexts followed the full four-level ordering. Stronger settings can shift the voice, so this remains experimental.

Anger intensity默认
LowestLowDefaultHigh

こんなことをして、良心が痛まないのか！

Voice cloning

Reference audio
Generated audio

Reference: えっと…こちらのお客様は？
Generated: あっ、つい話し込んじゃってたわね。

Reference conditioning

The reference recording is encoded into audio codes and included in the prompt alongside its transcript. A matching speaker marker on the new text conditions the model to continue that voice. Preset voices select different reference recordings; all use the same model weights.

Voice preservation

SFT preserves reference conditioning and mixes in ordinary untagged speech. Policy training uses for rewards and filtering. Whispering changes vocal characteristics, so its similarity threshold is handled separately.

Reference preparation: the transcript should match the recording. Robustness to incorrect transcripts has not been measured. Strong emotions can also reduce voice similarity.

Pronunciation

水神[すいじん]様すいじん様

水神[すいじん]さんから、素敵なお祝いの手紙が届きました！

OmniVoice · reading-control preview

Text preprocessing

The parser replaces the contiguous kanji immediately before the brackets with the kana inside them. In this example, the model receives 「すいじん様」; the remaining generation pipeline is unchanged.

Reading-tag experiments

Candidates trained to interpret bracketed readings fell short and degraded whispering and sighing. Explicit rewriting achieved 94.70% pronunciation accuracy on 264 outputs; the base model with the same frontend reached 93.90%. Most of the gain comes from explicit input.

Scope: explicit kana readings. This does not perform automatic word segmentation or control Japanese pitch accent.

Evaluation

Overview

All our experiments are based on Fish S2 Pro.
6 voices × 20 lines × joy, anger, sadness and surprise, for 480 clips. All models use the same reference audio and scoring procedure. Emotion accuracy is judged by two classifiers: emo2vec large (open source, not used in our training) and EmoRuler (our fine-tuned emo2vec large).
Fish Audio Drama 3 uses the same bracketed emotion tags, which it treats as natural-language instructions. Of 720 clips, 7 stopped mid-sentence and remained silent until the 95-second limit; they were scored as generated.
We also tested well-known multilingual TTS models, including Qwen3-TTS, OmniVoice and Ming 3.0 TTS. They work well in Chinese and English, but we heard more pronunciation errors and less natural delivery in Japanese, so this table compares models that performed better in Japanese.

ModelWith emotion (joy / anger / sadness / surprise)Without emotion
Accuracyemo2vec largeAccuracyEmoRulerJoy / Anger / Sadness / Surprise %EmoRulerVoice similarityspeaker similarityQualityUTMOSCharacter error rateVoice similarityspeaker similarityQualityUTMOSCharacter error rate
Fish S2 Pro OriginalStarting point, without training17.9%6.0%6 / 0 / 9 / 90.6853.515.6%0.6843.576.2%
OursFish S2 Pro + SFT + reinforcement learning85.6%93.3%91 / 94 / 95 / 930.6363.555.8%0.6943.784.8%
Fish Audio Drama 3Fish’s latest open-source model · 35B MoE (3B active)32.1%21.2%32 / 14 / 21 / 180.6283.267.0%0.6533.439.3%
IndexTTS-2.5Open source · bilibili40.2%30.6%65 / 18 / 19 / 200.6122.266.9%0.7142.916.0%
Irodori-TTS · Text descriptionOpen source · Aratako30.8%20.6%34 / 23 / 8 / 170.6493.036.0%0.6623.095.7%
Irodori-TTS · emojiOpen source · Aratako20.0%9.8%17 / 2 / 8 / 120.6513.036.5%0.6623.095.7%
ChatterboxOpen source · ResembleEmotion cannot be specified with tags0.6753.346.5%

Subjective listening test: MOS (Japanese)

We selected 15 groups × 5 emotions = 75 lines. Each listener heard all 75 lines.

Model Naturalness ↑ Emotion match ↑ Voice similarity ↑ Overall score ↑ Win share ↑
Ours 3.84 4.20 3.52 3.85 31.7%
Fish Audio Drama 3 3.97 3.64 3.45 3.69 23.2%
Irodori-TTS v4 large · Text description 3.61 3.95 3.32 3.63 28.1%
Fish S2 Pro Original 3.75 2.93 3.72 3.47 12.1%
IndexTTS-2.5 2.35 3.49 2.81 2.88 4.9%

All three dimensions use human ratings on a five-point scale (MOS). The overall score is their unweighted mean. Win share compares the five models for each listener–line pair, splitting credit equally when models tie for first.

Pairwise win rates

Ours’ pairwise win rates, based on the overall score, with ties counted as half a win:

Comparison Ours win rate ↑
vs. Fish S2 Pro Original 69.3%
vs. Fish Audio Drama 3 58.0%
vs. Irodori-TTS v4 large · Text description 58.7%
vs. IndexTTS-2.5 82.7%

Vocal events

Release evaluation detected sighing in 56/60 clips and crying in 16/60. Laughter scored 26/60 under the local check and 44/59 under an external check, indicating a need to examine the evaluation criteria.

Whisper guidance

On 60 matched whisper requests, guidance raised detections from 4 to 32, while SIM fell from 0.587 to 0.494 and UTMOS from 2.80 to 1.78. Intensity steering shows similar voice trade-offs.

Failed approaches

01

Limited gains from more unpaired data

+

Scaling unpaired data early on brought limited emotion gains. Paired examples made the relationship between condition and sound clearer by fixing content and voice. Other conditions changed between runs, so the full improvement cannot be attributed to one factor.

02

Intensity labels need acoustic differences

+

Changing intensity labels while reusing the same audio provides no acoustic distinction. Hidden-state steering showed better trends, but no positive setting passed every quality gate in a strict sweep of 880 outputs.

03

Emotion rewards induced extra laughter

+

Emotion rewards can favor easily recognized shortcuts. One inspection found unrequested laughter in 11.7% of happy samples. A higher reward does not necessarily mean a delivery that better matches the request.

04

Standard quality gates can reject whispers

+

Whispering changes acoustic characteristics. Ordinary speech-similarity thresholds may reject valid whispers; pushing whisper detection higher can degrade quality. A strong event detector also does not guarantee the generation policy will learn that event.

05

Explicit pronunciation rewriting worked better

+

Across 264 pronunciation outputs, explicit kana raised accuracy from 39.40% to 94.70%; the base model with the same frontend reached 93.90%. This gain belongs to explicit pronunciation input, not a new acoustic capability.

About Iroha-TTS-V1

Release articleModel