A single rough clip can ruin the entire style — How to choose 5 good ones

📝 Originally published (in Japanese) at forge.workstyle.tech.

The Roughness of a Single Clip Can Ruin an Entire Style

The TTS system I’m using builds emotional styles from “a few representative clips.” For example, if you provide several clips of a “joyful” voice, the system registers the average style vector from those samples.

At first, I naively passed the first five clips of the group:

clips = corpus_clips[emotion][:5]
register_style(model_id, style_name=emotion, clips=clips)

Enter fullscreen mode Exit fullscreen mode

The resulting synthesis sounded raspy in every sentence.

One Bad Clip Out of 200 Can Make a Difference

The issue was that one of the five clips I selected was of poor quality.

Generated audio occasionally produces low-quality outputs—raspy voices, unstable pitch fluctuations, or octave jumps. The quality gate (which checks alignment with the script) passes these because the content is read correctly; the roughness in the audio isn’t detected.

The full corpus contains about 200 clips, so a few rough ones have little impact on overall training. However, style registration only uses five clips. If one of those is rough, it accounts for 20% of the input.

Additionally, averaging doesn’t dilute the problem as much as expected. Style vectors are averages in embedding space, but rough audio often lies far from the norm in the direction of “raspiness.” When you average four clean clips with one outlier, the centroid shifts significantly toward the outlier.

As a result, every synthesized voice using that style ends up sounding raspy. One rough clip propagates its roughness across the entire style’s output.

How to Measure “Roughness”

Breaking down what sounds “rough” to the ear, I identified two main factors:

Jitter (periodic instability): Unstable vocal fold vibration cycles, creating a raspy impression.

Octave jumps: F0 estimation jumps to double or half the frequency between adjacent frames. This can happen due to actual voice breaks or estimation errors, but either way, it results in an unstable sound.

I measure these two factors and select the clips with the lowest values.

def stability_score(wav_bytes):
    """Returns (jitter, octave_jump_rate). Lower is more stable."""
    f0 = extract_f0_series(wav_bytes)          # F0 series every 10ms
    voiced = f0[f0 > 0]
    if len(voiced) < 10:
        return None                            # Too short for evaluation

    # Jitter: average relative difference between adjacent periods
    periods = 1.0 / voiced
    jitter = float(np.mean(np.abs(np.diff(periods))) / np.mean(periods))

    # Octave jumps: proportion of adjacent frames where ratio is near 2x or 0.5x
    ratio = voiced[1:] / voiced[:-1]
    jump  = float(np.mean((ratio > 1.8) | (ratio < 0.55)))

    return jitter, jump

def stable_top(clips, n=5):
    """Selects the n most stable clips"""
    scored = []
    for c in clips:
        s = stability_score(c.wav)
        if s is None:
            continue
        jitter, jump = s
        scored.append((jitter + jump * 2.0, c))    # Weight jumps more heavily
    scored.sort(key=lambda x: x[0])
    return [c for _, c in scored[:n]]

Enter fullscreen mode Exit fullscreen mode

I weight octave jumps more heavily (×2.0) because their perceptual impact is greater. Jitter creates a continuous roughness, while jumps create a discrete breakdown—like a voice cracking—which is more noticeable.

The registration process was updated accordingly:

# Before
clips = corpus_clips[emotion][:5]

# After
clips = stable_top(corpus_clips[emotion], n=5)

Enter fullscreen mode Exit fullscreen mode

This eliminated the raspy quality.

Filtering Out Clips With Short Voiced Segments

The line if len(voiced) < 10: return None is quietly important.

Short exclamations (“Yes!”, “Alright!”) or phrases with many unvoiced sounds (“desu shi”, “shikkari”) yield only a few voiced frames. Calculating jitter from such clips is meaningless, and they can accidentally receive low scores and be misclassified as “most stable.”

Clips that can’t be evaluated are excluded. If fewer than five stable clips are found, the system selects as many as possible.

⚠️ Low-Pitched Voices Overestimate Jitter

This metric has a known weakness: when F0 is estimated using autocorrelation, low-pitched voices are more likely to be misestimated. Harmonics may be confused with the fundamental frequency, or F0 estimates may jump between frames—both contributing to inflated jitter scores.

In practice, male voices showed jitter values between 31% and 56%. Normal human speech typically has jitter below 1%, so these measurements are clearly dominated by estimation errors.

Therefore:

• Relative comparisons within the same speaker are usable. Since errors are uniform, rankings remain meaningful.
• Comparisons between speakers, especially across genders, are not usable. Low-pitched voices are systematically disadvantaged.

Since style registration selects clips from the same speaker, it falls under the first case and works fine.

When I used the same metric (with weighting) to select the best voice from multiple candidates (Machine-Selected “Narrator-like Voices” from 24 Candidates), all male candidates received negative scores—a clear example of applying the metric outside its valid range. In that case, I reduced the weight and incorporated other evaluation criteria.

Separating Quality Gates into Layers

This experience revealed that “content correctness” and “audio quality” require separate gates.

Gate What It Checks What It Rejects Script Alignment (Whisper) Content match, non-script sounds Hallucinations, misreadings, omissions Tail Fidelity Unscripted elongated endings Clips where tail habits are baked in Acoustic Normalization Frequency response, loudness Channel characteristic variation Stability Selection Jitter, octave jumps Roughness, raspiness

Script alignment only checks content, so it allows rough audio through. Acoustic normalization (as discussed in TTS That Changes Recording Room Every Time) aligns spectral shapes but cannot fix temporal instability like jitter. Each gate checks a different layer, and none can substitute for another.

Moreover, the stability selection gate isn’t just a “pass/fail” filter—it’s a selection mechanism. While other gates exclude failures, this one selects the top performers from the passing pool. When working with a small, elite set of resources, filtering isn’t enough—you need selection.

Determining Strictness Based on Impact Range

Generalizing this idea:

The stricter the selection, the greater the impact of each individual clip.

• Training corpus of 200 clips → one clip has 0.5% impact. A simple quality gate is sufficient.
• Style registration using 5 clips → one clip has 20% impact. A gate alone isn’t enough; you need top-N selection.

Even when drawing from the same pool of material, the required quality depends on how the clips are used. Clips suitable for inclusion in the corpus aren’t necessarily suitable for representing a style. They require different selection criteria.

The mistake I made was assuming that “passing the quality gate means it’s usable.” A gate only guarantees a minimum standard of usability, not that a clip is appropriate as a representative.

Summary

• When working with a small set of clips, one outlier can ruin the entire style. With 5 clips, one bad one has a 20% impact that averaging can’t fully dilute.
• Roughness can be measured by jitter and octave jumps. Since octave jumps have a stronger perceptual impact, they should be weighted more heavily.
• Clips with short voiced segments should be excluded. Otherwise, they may be incorrectly judged as “most stable.”
• Low-pitched voices overestimate jitter. Use this metric only for relative comparisons within the same speaker.
• Exclusion gates and selection gates serve different purposes. For small, elite sets, selection is essential.
• Determine strictness based on impact range. The same material may require different quality standards depending on its use.

Series: Producing Practical Voices from Diffusion TTS

A record of designing voices from single-line captions, manufacturing training corpora, and mass-producing role-specific practical voices. This article is Part 2: Manufacturing.

← Previous: TTS That Changes Recording Room Every Time
→ Next: Where Did the AI Learn to Elongate “Konnichiwa~”?

Full Series (18 Parts)

1. Sound-Quality-Selected TTS Was Too Slow for Conversation
2. Drawing Voices Like a Gacha
3. Machine-Selected “Narrator-like Voices” from 24 Candidates
4. The Harder the Quality Gate, the More Flat Reads Survive
5. You Can’t Change Speaking Rate After Training
6. TTS That Changes Recording Room Every Time 7. One Rough Clip Can Ruin an Entire Style ← You are here
7. Where Did the AI Learn to Elongate “Konnichiwa~”?
8. 「少々」が「しょも」になる — Permission Character Lists Were Truncating Japanese
9. Code That Only Worked During Hallucinations
10. The “3-Character” That the Quality Gate Allowed Became the Model’s Tics
11. We Were Rejecting Candidates for Fixable Flaws
12. There Are Defects That Transcription Can’t Catch
13. 70 Minutes of Training Material Lost to a Network Blink
14. It Took “ja” to Become “JP” to Create a Babbling Model
15. Four Registration Paths, Zero Management UI
16. Deployments Kept Overwriting Each Other’s Outputs
17. Chasing Unmeasurable Traits with Thresholds Always Fails

The knowledge in this article is based on the manufacturing pipeline: Producing Practical Voices from Diffusion TTS.