Automating the Selection of Natural, High-Quality Single-Speaker Anchors

📝 Originally published (in Japanese) at forge.workstyle.tech.

Selecting High-Quality Voice Anchors for Voice Conversion

When building a voice conversion app, I often find that the quality of the output depends less on the conversion model itself and more on what target voice (anchor) you choose. In our app, we initially extracted 4–12 second segments from speaker or narration recordings to use as anchors. At first, we relied on “sounding good to the ear” to pick these segments—but this approach doesn’t scale.

Manually listening to dozens of recordings to hand-select good segments is not only tedious, but the judgment varies with mood. Plus, audio quality varies wildly: overlapping voices in a conversation, over-the-top emotional delivery with exaggerated intonation, background noise. Continuously filtering out “unusable segments” by hand is simply unsustainable.

This article documents how we replaced that “ear-based selection” with a three-stage filtering pipeline. Our goal: mechanically select segments that are single-speaker, natural, and high-quality. The result? The minimum PESQ score of our anchor set improved from 1.24 to 2.31, and we ended up with over 70 usable anchors.

Prerequisites: Three Requirements for a Good Anchor

What makes a good anchor for voice conversion? To be usable as a target voice, a segment must satisfy three conditions simultaneously:

1. Single-speaker: If a 12-second segment contains another person’s voice, the speaker embedding becomes an average of two voices—resulting in a “no one’s voice.”
2. Natural and calm delivery: Shouting, falsetto, singing, or exaggerated intonation deviate from the speaker’s “natural voice.” For conversion, neutral speech is ideal.
3. High audio quality: Segments with noise or distortion will carry over into the converted output.

The challenge is that these three conditions are independent properties. Trying to evaluate them with a single score leads to failure. So we designed independent filters applied in series, each targeting one property.

Overall Flow: Apply Cheap Filters First

We arrange filters in order of increasing computational cost, progressively narrowing down candidates. For each file, we first place 48 evenly spaced candidate windows, then filter them in stages:

48 candidates (evenly spaced)
  → Stage 1: Score "naturalness" using prosody → Top 16
  → Stage 2: Use campplus speaker embeddings to check single-speaker consistency
  → Quality Gate: Use SQUIM PESQ to assess cleanliness
  → Select the most natural window that passes both single-speaker and quality checks

Enter fullscreen mode Exit fullscreen mode

Lightweight analyses like pyin and RMS are applied to all candidates, while heavier operations like speaker embeddings and SQUIM are only applied to the top 16.

Stage 1: Measuring “Naturalness” via Prosody

The first filter quantifies how “calm” the delivery is using pitch (F0) and energy (RMS). We extract F0 using librosa.pyin and compute several indicators:

• Micro-instability (instab): Median semitone change between adjacent frames. Lower = more stable.
• Pitch spread (spread): p90–p10 of F0 in semitones. Too wide = over-expressive; too narrow = monotone.
• Voicing confidence (conf): Clarity of periodicity.
• Energy coefficient of variation (ecv): Std dev / mean of RMS. High values indicate emotional volume changes.
• Max hold duration (hold): Longest continuous stretch where F0 stays within ±0.5 semitones. Detects sustained vowels, singing, or elongated sounds.

An interesting twist: instead of treating low spread as “good,” we evaluate deviation from a target spread. We want to avoid both monotone and overly expressive speech, so we set a target spread of SPREAD_TARGET = 6.0 semitones and penalize deviations from it.

def _composite(c: dict, modal_f0: float) -> float:
    # Deviation from modal F0 [semitones] (detects falsetto/shouting/character voices)
    modal_dev = abs(np.log2(max(c["f0"], 1e-6) / max(modal_f0, 1e-6))) * 12.0
    pen = (W_SPREAD * abs(c["spread"] - SPREAD_TARGET)   # Penalize deviation from target spread
           + W_ECV * c["ecv"]                            # Energy variation
           + W_HOLD * max(0.0, c["hold"] - HOLD_FREE_S)  # Penalize holds >0.5s (weight 1.2)
           + W_MODAL * modal_dev)                        # Deviation from modal pitch
    base = c["conf"] / (1.0 + c["instab"])
    return base / (1.0 + pen)

Enter fullscreen mode Exit fullscreen mode

modal_f0 is the median F0 across all candidate windows in the file—i.e., the speaker’s usual pitch range. We heavily penalize windows that deviate significantly (e.g., falsetto, shouting), prioritizing segments closer to the speaker’s “natural voice.” Holds are allowed up to 0.5 seconds, but longer stretches are penalized to exclude singing or elongated endings.

We sort candidates by this score and pass only the top 16 to the next stage.

Stage 2: Speaker Embeddings to Verify Single-Speaker Consistency

This stage tackles dialogue or interview sources. We use the same campplus speaker embedding (192-dim) used in our anchor definition to check if a 12-second segment contains only one speaker.

The method is a simple two-speaker test. We divide the 12-second segment into 3-second chunks (with 1.5-second overlap), compute speaker embeddings for each chunk, then identify the most dissimilar chunk pair to seed a two-cluster assignment. We then compute the cosine similarity between the two cluster centroids.

E = E / (np.linalg.norm(E, axis=1, keepdims=True) + 1e-9)
S = E @ E.T
# Use most dissimilar chunk pair as seeds for 2 clusters
a, b = np.unravel_index(int(np.argmin(S)), S.shape)
assign = (S[a] < S[b]).astype(int)          # 0=close to seed a, 1=close to seed b
if assign.min() == assign.max():            # One-sided (effectively single cluster)
    return 1.0
c0 = E[assign == 0].mean(axis=0); c1 = E[assign == 1].mean(axis=0)
# ...L2 normalization...
return float(np.dot(c0, c1))               # Cosine similarity between centroids (high = single speaker)

Enter fullscreen mode Exit fullscreen mode

If the segment is single-speaker, chunk differences are minimal (mostly phonetic), so the centroids are close and cosine similarity is high. If speakers alternate, centroids diverge and similarity drops. We reject candidates where similarity falls below homo-thresh = 0.55.

Crucially, the embedding used for selection matches the one used to define anchors. This ensures the selection criteria and anchor definition are measured on the same scale—no drift in standards.

Quality Gate: Ensuring Clean Audio with SQUIM PESQ

The final gate checks recording quality. We use Torchaudio’s SQUIM_OBJECTIVE to estimate PESQ/STOI/SI-SDR without a reference—critical for raw, unpaired material.

def _pesq(squim, y16, center_s):
    seg = y16[i0:i0 + int(EXT_S * SR16)]   # 12s segment
    with torch.inference_mode():
        _stoi, pesq, _sisdr = squim(torch.tensor(seg).unsqueeze(0).float())
    return float(pesq.item())

Enter fullscreen mode Exit fullscreen mode

We set a threshold of pesq-thresh = 2.3. The final selection logic: among candidates that are single-speaker (homog ≥ 0.55) and high-quality (PESQ ≥ 2.3), pick the one with the highest Stage 1 score (i.e., most natural). If no candidates meet both criteria, we relax the quality constraint while still prioritizing single-speaker segments.

Results: Minimum Quality Bottoms Out

We determined thresholds by first auditing all anchors with audit_anchor_quality.py, ranking them by SQUIM score to see where to cut. Before the quality gate, the minimum PESQ in our anchor set dipped as low as 1.24—noise and distortion had slipped through. After applying the gate to real data (anchor_quality.json, 72 anchors), the distribution improved dramatically:

• Minimum PESQ: 2.31
• p25: 2.72 / Median: 3.08 / Max: 4.12

Raising the floor from 1.24 to 2.31 means the “worst anchor” is now far better—improving the overall user experience. A median above 3.0 also shows our selection process is now stable.

Pitfalls & Lessons Learned

• Don’t mix into a single score. Single-speaker consistency, naturalness, and audio quality are uncorrelated. Weighted sums tend to favor “mediocre in everything.” By separating concerns into independent filters and applying hard constraints (single-speaker) and soft priorities (naturalness), we achieved the desired outcome.
• “Smaller is better” isn’t always right. Pitch spread isn’t about minimizing variance—it’s about avoiding both monotony and over-expressiveness. Switching to deviation-from-target scoring finally allowed natural conversation to be selected.
• Use the same metric for selection and definition. We used campplus for both single-speaker detection and anchor embedding. Using a different model for selection risks passing segments that sound mismatched in production.
• Set thresholds based on audits, not guesses. We didn’t arbitrarily set 2.3—we ran the audit script, examined the distribution, and chose the cutoff. Guessing thresholds is just “ear-based selection” in disguise.

Summary

• Replaced “ear-based selection” with a three-stage quantitative filtering pipeline.
• Stage 1 (prosody) scores naturalness; Stage 2 (campplus embeddings) checks single-speaker consistency; the quality gate (SQUIM PESQ) ensures clean audio.
• Pitch spread is evaluated as deviation from a target (6 semitones), not minimized. Falsetto/shouting is penalized via deviation from the speaker’s modal pitch.
• Quality gate (PESQ ≥ 2.3) raised the minimum PESQ from 1.24 → 2.31, with a median of 3.08 and over 70 anchors.
• Thresholds were set using audit scripts, not intuition. Single-speaker detection uses the same embedding model as anchor definition to ensure consistency.