vocal_helper.diar module

vocal_helper.diar

Two diarization paths : online for live streams and offline for batch / file-based inputs.

  • OnlineDiarStage — consumes VoicedSegment as the VAD emits them, embeds each one, and runs a per-segment cosine running-mean clusterer. The current best online answer per the pdbms 2026-06-29 canonical study : matches hungarian_nemo / hungarian_pyannote in spirit, simpler because the VAD has already isolated each segment.

  • OfflineDiarStage — receives the full PCM buffer and hands it to the canonical offline backend (pyannote/speaker-diarization-3.1 by default, NeMo Sortformer as alternative). Runs whole-buffer by default : the 2026-07-14 offline map-reduce study found whole-buffer strictly best for DER, so pyannote only chunks past ideal_duration_s = 3600 s (a memory backstop), while NeMo keeps 60 s (Sortformer 90 s cap). When chunking does kick in, the stage overlaps chunks and stitches via cosine AHC (pdbms §10.5, AMI dev-slice median DER 0.116, inside Bredin 2023’s band).

Reliability — which path to use

The offline path is the reliable one and should be preferred for any batch / file input. A 2026-07-16 DER sweep (studies/diar_der_paths.py, pyannote.metrics, collar 0.25) measured every path against ground truth:

corpus

offline pyannote

offline nemo (Sortf)

online (no ref / ref)

AMI (20-40 min meetings)

0.122

0.242

0.497 / 0.351

bagarre (~30 s, <=4 spk)

0.338

0.177

0.586 / 0.592

Takeaways: (1) offline pyannote is literature-grade (Bredin 2023 ~ 0.188 uncollared) and wins on long meetings, running whole-buffer with global clustering and no speaker-count cap. (2) offline NeMo Sortformer (diar_sortformer_4spk-v1) is end-to-end and overlap-aware and nearly halves the DER on short <=4-speaker clips — but it is capped at 4 speakers and ~90 s per window, so it degrades once it must chunk long audio. (3) The online streaming path (nemo TitaNet embeddings) stays ~3x the offline DER — a latency-bound approximation that cannot model overlapped speech ; refine_on_close roughly halves its DER on meetings that over-segment (ES2011a 0.588 -> 0.296) and never hurts.

Default policy: pyannote is the offline default — robust across any length and speaker count, best on the long inputs that dominate file use. Pick --offline --diar-backend nemo for short <=4-speaker workloads where Sortformer wins. The CLI file --no-real-time auto-selects offline pyannote when the bundle is present ; reserve OnlineDiarStage for live streams. Downstream integrators embedding diarization in a larger pipeline should use OfflineDiarStage / OfflinePipeline for batch.

Online algorithm — minimal cosine-AHC online clusterer

Algorithm — minimal cosine-AHC online clusterer

We don’t carry the full pdbms HungarianDiar across this boundary. The full sliding-window Hungarian wrapper assumes the diarizer is fed raw PCM windows, but here the VAD already gives us isolated voiced segments — one embedding per segment is enough and the global stitching collapses to a 1-D nearest-centroid match on cosine distance with running-mean updates.

For each incoming VoicedSegment :

  1. Embed via the configured backend (pyannote/embedding or NVIDIA TitaNet). The embedding is L2-normalised.

  2. Compute cosine distance to every existing centroid.

  3. The minimum-distance centroid wins iff its distance is below join_threshold (default 0.30, calibrated on AMI dev-slice in the 2026-06-30 stitch-threshold sweep). Otherwise, mint a new speaker.

  4. Update the matched centroid by exponential moving average with coefficient ema_alpha (default 0.1) so the centroid adapts slowly to within-speaker variation.

The stage is meant to run online — every voiced segment is labelled at most embed_latency_ms after the speech ends.

Backend choice

backend='pyannote' is the default and only requires the pyannote extra (pip install vocal-helper[pyannote]). It uses pyannote/embedding (160 ms minimum input). backend='nemo' uses NVIDIA TitaNet via the nemo extra ; slower to load but better cosine separation on noisy mixes.

Author

Warith HARCHAOUI — https://linkedin.com/in/warith-harchaoui

class vocal_helper.diar.OfflineDiarStage(*, backend='pyannote', ideal_duration_s=None, overlap_s=10.0, stitch_threshold=0.35, device=None)[source]

Bases: object

Offline diarization on the full PCM buffer.

Designed for batch / file-based use : the upstream source is expected to drain end-to-end, the stage collects the full PCM, then hands it to the canonical offline backend.

Backends

  • "pyannote" — pyannote/speaker-diarization-3.1. The production default for any meeting / podcast / lecture (pdbms §10.5 : AMI dev-slice median DER 0.116, inside Bredin 2023’s 0.188 band).

  • "nemo" — NVIDIA Sortformer (the nvidia/diar_sortformer_v1 checkpoint). Better for short clips ≤ 60 s, struggles past its 90 s training cap. Auto-chunked when len > IDEAL_DURATION_S.

Long-form chunking

For inputs longer than ideal_duration_s the stage replicates the pdbms ChunkedOfflineDiarizer strategy at minimal cost :

  1. Split the audio into chunks of ideal_duration_s with overlap_s (default 10 s) of shared content at boundaries.

  2. Run the backend on each chunk.

  3. Embed each chunk-local speaker on its concatenated audio.

  4. Cluster all chunk-local embeddings via cosine AHC (stitch_threshold=0.35, the value selected in the 2026-06-30 stitch_threshold sweep on AMI dev-slice N=8 where t∈{0.30..0.40} forms the operating plateau).

The full pdbms variant adds VAD-aware cut-point selection and pink-noise pad ; vocal-helper trades these for a simpler hard-cut + zero-pad pair to keep the dependency surface small. For mission- critical AMI-style work, use pdbms.diar.offline_chunked.ChunkedOfflineDiarizer directly.

Chunking is a memory ceiling, not a quality lever. The 2026-07-14 offline map-reduce study (full stack VAD + ASR + diar on AMI, doc/studies/offline-mapreduce-study.md) found DER strictly monotone in chunk size — whole-buffer is best (median DER 0.143 vs 0.170 at 300 s, and cliffs to 0.31 / 0.50 at 120 s / 60 s as speaker fragmentation outruns the stitch) — and ASR destabilises when chunked (a long-window whisper loop drove one meeting to WER 1.17). So the pyannote default now runs whole-buffer for any realistic input (ideal_duration_s = 3600 s) and only chunks past ~1 h as a memory backstop. NeMo is the exception: its Sortformer 90 s training cap forces chunking at ideal_duration_s = 60 s.

param backend:

Backend to use. Default "pyannote". "sherpa" is the portable ONNX pipeline (community-1 seg + TitaNet-large emb, no torch) — see ADR 0002.

type backend:

“pyannote” | “nemo” | “sherpa”

param ideal_duration_s:

Whole-buffer ceiling : inputs longer than this are chunked + stitched, shorter ones run as a single call. Default depends on the backend — 3600 s for pyannote (effectively whole-buffer for any realistic meeting; chunking is a memory backstop only), 60 s for NeMo (forced by its Sortformer 90 s cap).

type ideal_duration_s:

float, optional

param overlap_s:

Overlap between adjacent chunks. Default 10 s.

type overlap_s:

float

param stitch_threshold:

Cosine-distance threshold for cross-chunk AHC stitching. Default 0.35.

type stitch_threshold:

float

param device:

Torch device for the pyannote pipeline + embedder. None (default) auto-picks CUDA > MPS > CPU. On Apple Silicon CPU is ~ 10× slower than MPS, so the auto-pick matters in practice. Has no effect on the NeMo backend.

type device:

“cpu” | “cuda” | “mps”, optional

Notes

Model weights load from the self-hosted diarization-engines bundle (see resolve_diarization_engines()) — no HuggingFace token is used or accepted.

diarize(pcm, sr)[source]

Return [(t0, t1, speaker), …] sorted by start time.

Parameters:

  • pcm (ndarray[tuple[Any, ...], dtype[float32]])

  • sr (int)

Return type:

list[tuple[float, float, str]]

async run(inbox, outbox)[source]

Drain inbox of PcmFrame ; emit DiarizedSegment.

Collects every frame until the upstream sends None, then runs diarize on the full buffer in a worker thread and emits one DiarizedSegment per identified speaker span.

Parameters:

  • inbox (Queue)

  • outbox (Queue)

Return type:

None

Parameters:

  • backend (BackendName)

  • ideal_duration_s (float | None)

  • overlap_s (float)

  • stitch_threshold (float)

  • device (str | None)

class vocal_helper.diar.OnlineDiarStage(*, backend='nemo', join_threshold=0.3, ema_alpha=0.1, min_segment_ms=500, device=None, max_speakers=None, refine_on_close=False, min_cluster_size=2, merge_threshold=None)[source]

Bases: object

Producer/consumer online speaker diarizer.

Parameters:

  • backend ("pyannote" | "nemo" | "sherpa") – Which embedding model to use. Default "pyannote". "sherpa" runs TitaNet-large through onnxruntime (no torch) — see _SherpaEmbedder.

  • join_threshold (float) – Cosine-distance threshold below which a new segment joins an existing centroid. Default 0.30 — calibrated on AMI dev-slice N=8 in 2026-06-30 stitch_threshold sweep, where the pyannote/embedding distribution exhibits a clear DER minimum at the 0.30-0.45 plateau.

  • ema_alpha (float) – Exponential-moving-average coefficient for centroid updates. Default 0.1.

  • min_segment_ms (int) – Minimum voiced-segment duration to attempt embedding. Default 500 ms (pyannote/embedding’s convolutional kernels choke on shorter inputs).

  • device ("cpu" | "cuda" | "mps", optional) – Torch device for the pyannote embedder. None (default) auto-picks CUDA > MPS > CPU. Has no effect on the NeMo backend.

  • max_speakers (int | None)

  • refine_on_close (bool)

  • min_cluster_size (int)

  • merge_threshold (float | None)

Notes

Model weights load from the self-hosted diarization-engines bundle (see resolve_diarization_engines()) — no HuggingFace token is used or accepted.

async run(inbox, outbox)[source]

Consume VoicedSegment from inbox, push DiarizedSegment.

In streaming mode (refine_on_close=False) each segment is labelled and emitted as it arrives. In batch mode (refine_on_close=True) segments are buffered, globally re-clustered when the stream closes, then emitted with corrected labels — see _run_refine().

Parameters:

  • inbox (Queue)

  • outbox (Queue)

Return type:

None

vocal_helper.diar.resolve_diarization_engines()[source]

Locate the HF-free diarization-engines bundle, or None.

Source order: the explicit $VH_DIARIZATION_ENGINES env var, then engines.diarization_url in settings.yaml (the canonical config), then DEFAULT_DIARIZATION_ENGINES_URL. A local dir is used as-is ; a URL to diarization-engines.zip is downloaded once and cached under $VH_CACHE_DIR (default ~/.cache/vocal-helper). Returns the directory that contains manifest.json.

Return type:

Path | None