Learning from Failure: Why Couldn't 'Breathiness' Be an Independent Parameter?

📝 Originally published (in Japanese) at forge.workstyle.tech.

Introduction to the “Breathy Voice” Parameter

We added several sliders to our “Voice Design” app to adjust voice qualities, including speed, pitch, and breathiness. The breathiness slider is intended to add a whispery, gentle voice quality or a breathy expression to the voice.

Conclusion

This slider could not be implemented. We tried various methods, including addition, band-limited noise, cross-synthesis, increasing non-periodicity with WORLD, differential addition, high-frequency limitation, cross-fading with whispering, and finally generating it on the model side. None of these methods achieved a “natural breathiness without noise.”

Why It Failed

The reason for the failure was obvious from the start, but it took nearly 10 attempts to articulate it. Breathiness equals respiratory noise and is physically inseparable from noise. This article honestly records the failures in chronological order, following the commit logs from Git.

Premise: What is a Breathy Voice?

First, let’s clarify what “breathy voice” or “breathiness” means in terms of acoustics. Voiced sounds are created by periodic vibrations of the vocal cords (pitch), but in breathy voices, the glottis is not completely closed, and turbulent noise of exhalation is mixed with the periodic vibration. Whispering is an extreme case where the periodic component is almost eliminated, and only turbulent noise remains.

In other words, the essence of breathiness is turbulent noise itself. This sentence anticipates the conclusion of this article. At the time, however, we didn’t fully understand this and thought that if we could shape the noise properly, we could create a “refined breathiness” that didn’t sound like noise.

Record of Trials

1. Adding Noise → High-Frequency Boost

Initially, we added Gaussian noise. As expected, this resulted in the addition of hiss noise. We then changed our approach to simply lightly boosting the high frequencies instead of adding noise.

sos = butter(2, min(3000.0, sr / 2 * 0.9) / (sr / 2), btype="high", output="sos")
hf = sosfilt(sos, wav).astype(np.float32)
return wav + hf * (0.22 * amt)

Enter fullscreen mode Exit fullscreen mode

The noise sensation disappeared, but this merely made the sound brighter without adding a sense of breath. It was more like an equalizer than a breathiness control.

2. Band-Limited (2-8kHz) Respiratory Noise

Since the high-frequency boost didn’t produce a breathy sound, we decided to simulate the turbulent flow of exhalation more directly by band-limiting the noise to 2-8kHz and shaping it with the volume envelope to only apply to voiced parts. This way, the noise wouldn’t apply to silent parts, avoiding a flat hiss.

env = np.abs(wav).astype(np.float32)          # Volume envelope (breath applies to speaking parts)
...
noise = sosfilt(lp, sosfilt(hp, noise))       # Band-limit to 2-8kHz
breath = noise * env * (0.35 * amt)

Enter fullscreen mode Exit fullscreen mode

The high-frequency component (>2kHz) ratio increased from 2.1% to 5.8%, indicating an increase in breathy components. However, to the ear, it still sounded like noise from a separate layer rather than part of the voice.

3. Cross-Synthesis: Shaping Noise with Voice Envelope

We thought that the flat noise sounded like a separate layer because it wasn’t passing through the vocal tract’s resonance (formant). So, we tried shaping the noise with the short-term spectral envelope of the voice and then mixing this “whisper component” into the original voice. The idea was that by passing the noise through the vocal tract’s resonance, it would sound like breath mixed with the voice rather than separate noise.

4. Increasing Non-Periodicity (AP) with WORLD

Here, we significantly changed our approach. Instead of trying to “add” breathiness, we attempted to recreate the voice itself to be more breathy. Using the WORLD vocoder, we analyzed the signal into F0 (pitch), spectral envelope (SP), and non-periodicity (AP), and then increased the non-periodicity before re-synthesizing. Increasing non-periodicity makes the voice sound more like breath.

f0, t = pw.harvest(x, sr)
sp = pw.cheaptrick(x, f0, t, sr)
ap = pw.d4c(x, f0, t, sr)
ap2 = np.clip(ap + (0.45 * amt) * (1.0 - ap), 0.0, 1.0)   # Increase non-periodicity towards 1 = more breath
y = pw.synthesize(f0, sp, ap2, sr, frame_period=5.0)

Enter fullscreen mode Exit fullscreen mode

This approach seemed promising but introduced new issues.

5. Double Codification Causes Distortion → Add Only the Difference

The input was already a voice generated by Seed-VC. When we re-synthesized it entirely with WORLD, it resulted in double coding, causing the sound to distort. To counter this, we decided to synthesize the voice twice (once with increased AP and once without) and only add the difference (the increased breath component) back to the original clean voice. The idea was that the WORLD-induced degradation would be canceled out by taking only the difference.

6. Low Frequencies Become Muddy → High-Frequency Limitation

Next, we found that increasing non-periodicity muddied the low frequencies (harmonics), making the voice sound gravelly. Since breathiness primarily affects high frequencies, we decided to limit the increase in non-periodicity to frequencies above 1.5kHz, keeping the lower frequencies (and thus the voiced parts) clean. This resolved the muddiness and distortion but led to another realization: the effect becomes weaker when limited to higher frequencies.

7. Strengthening → Cross-Fading with Whispering

Acknowledging the limitations, we tried strengthening the effect (with a high-frequency cut at 1.0kHz and a strength of 0.85) and then created a whisper version of the voice by fully non-periodicizing it across all frequency bands. We then cross-faded this whisper version with the original voice. At 100%, it’s a full whisper, and in the middle, it simulates breathy voice. Each component remains clean without muddying.

By this point, we had exhausted most post-processing methods. The common failure point was that as we tried to make the breathiness stronger, the noise increased, and as we reduced the noise, the breathiness became weaker.

8. Giving Up and Generating on the Model Side

If post-processing breathiness was limited, we thought, maybe the model itself could generate breathy voices cleanly. We adjusted the reference prompt (cref) in Seed-VC to blend with a whisper version (created using WORLD) based on the desired breathiness amount and had the model generate the voice.

# Blend reference prompt with whisper version based on breathiness → Model generates breathy voice
cref = (cref * (1.0 - 0.85 * b) + _world_whisper(cref, sr) * (0.85 * b)).astype(np.float32)

Enter fullscreen mode Exit fullscreen mode

Upon verification, the voiced rate decreased from 0.64 to 0.11 (at 100% breathiness), and the spectral flatness remained at 0.028 — indicating that breathiness increased without introducing noise. We even set torch.manual_seed(1234) to deterministically generate the voice for A/B comparisons.

However, even this approach had its limits. A strong breathy voice still sounds like it has noise. It was obvious in hindsight: since breathiness is essentially noise, a strong, noise-free breathy voice is a contradictory goal.

9. Removal

We concluded that breathiness equals respiratory noise and is physically inseparable from noise. Thus, a “strong breathy voice without noise” is impossible to achieve. We removed the breathiness slider from the UI, leaving only speed and pitch variation.

Why We Didn’t See It from the Start

The root of the failure was not technical but how we initially framed the premise. The UI label “breathiness” had transformed in our minds into an unachievable goal of “a refined, noise-free breathy voice.” Acoustically, breathiness is defined as turbulent noise. If you remove the noise, you also remove the breathiness. This is a matter of physics, not implementation technique.

Another lesson learned is that improving intermediate metrics does not necessarily mean achieving the goal. High-frequency ratio, voiced rate, spectral flatness — all these metrics “improved,” but we were moving further away from the inherently contradictory goal of “noise-free breathiness.” The better these metrics became, the less noticeable the fact that we were chasing an impossible target.

Summary

• Breathiness is essentially turbulent noise. A “noise-free breathy voice” is acoustically self-contradictory and not a matter of implementation but physics.
• We tried various methods: adding noise, high-frequency boosting, band-limited noise, cross-synthesis, increasing non-periodicity with WORLD, differential addition, high-frequency limitation, cross-fading with whispering, and model-side generation. All ended in the dilemma of “stronger breathiness introduces noise, weaker breathiness lacks effect.”
• Increasing non-periodicity with WORLD and model-side generation improved intermediate metrics (voiced rate, flatness) but did not achieve the unachievable goal of noise-free breathiness. Improving metrics does not necessarily mean success.
• The final judgment was to remove the breathiness slider. It’s more honest to the product to remove parameters that are ineffective or unnatural rather than leaving them.
• The root cause of failure was that the UI label implicitly defined an unachievable goal. Before creating a feature, we should define what the term means acoustically and ensure it’s physically possible.