vocal_helper.lid module

vocal_helper.lid

Spoken-language diarization — partition an audio into mono-language regions so a code-switching recording can be transcribed one language at a time.

Why a posterior curve, not a boundary detector

Running whisper in naive "auto" mode locks onto the first language it hears and transcribes (often translates) the whole file in it — so a text-level detector only ever sees one language. Language must be resolved before transcription. But a language switch, unlike a speaker change, has no sharp acoustic cue (same speaker, same channel): it cannot be detected as a boundary, only classified per unit of time and smoothed.

Following ASR-posterior language diarization (Wang et al., Interspeech 2019) with Gaussian-smoothed change points, using whisper’s own language head as the posterior source, detect_language_regions():

  1. samples a language-posterior curve over overlapping windows (language_posterior_curve()),

  2. Gaussian-smooths it over time,

  3. takes the per-frame argmax,

  4. places change points where the argmax flips,

  5. absorbs sub-min_region_s regions into a neighbour,

  6. locally refines each change point (fine re-scan + interpolated crossing),

  7. snaps it to the nearest silence.

detect_language() remains the single-window primitive. All codes are plain ISO-639-1 ("en", "fr", …); callers may restrict the candidate set so whisper never ranks an un-routable close relative (Galician gl / Catalan ca over Spanish es) on a short window.

Independent verification

cross_check_regions() corroborates each region with a second, fully independent audio classifier — an ECAPA-TDNN trained on VoxLingua107 (SpeechBrain) — which shares nothing with whisper but the signal. It is an optional dependency (pip install vocal-helper[lid]), imported lazily.

Author

Warith HARCHAOUI — https://linkedin.com/in/warith-harchaoui

class vocal_helper.lid.LangRegion(lang, t0, t1)[source]

Bases: object

A contiguous mono-language span of audio (seconds), ISO-639-1 lang.

Parameters:
lang: str
t0: float
t1: float
class vocal_helper.lid.RegionVerdict(t0, t1, primary, speechbrain, sb_prob, agree)[source]

Bases: object

One region cross-checked against the independent SpeechBrain LID.

Parameters:
agree: bool
primary: str
sb_prob: float
speechbrain: str
t0: float
t1: float
vocal_helper.lid.cross_check_regions(pcm, regions, sample_rate=16000, *, supported=None)[source]

Corroborate each region’s language with the independent SpeechBrain LID.

An agree=False verdict is a genuine disagreement between two models that share only the audio — a signal to inspect, not a code bug. Regions shorter than 1 s are too short to judge and pass through as agreeing.

Parameters:
Return type:

list[RegionVerdict]

vocal_helper.lid.detect_language(pcm, *, model='large-v3-turbo-q5_0', threads=6, offset_ms=0, supported=None)[source]

Discover the actual language of pcm (single window).

Runs whisper.cpp’s auto_detect_language and, by default, returns whisper’s true argmax over its full language head — the language the input actually is, with no default and no restriction. supported is an optional, opt-in routing guard: pass the ISO-639-1 codes you can actually route and detection is re-ranked strictly within that set (the “mind the codes” guard — on a short window whisper may rank a close relative such as Galician gl above the routable Spanish es). Leave it None to let the input speak for itself.

Parameters:

  • pcm (NDArray[np.float32]) – Mono waveform window at the model’s sample rate.

  • model (str, optional) – whisper.cpp model name (default DEFAULT_MODEL).

  • threads (int, optional) – Inference threads (default DEFAULT_THREADS).

  • offset_ms (int, optional) – Offset into pcm handed to whisper’s detector (default 0).

  • supported (tuple[str, ...] or None, optional) – Opt-in routable-language whitelist. None (default) means discover freely — return whisper’s own top language. A tuple restricts the answer to those codes (e.g. DEFAULT_SUPPORTED_LANGS).

Returns:

(iso_639_1_code, probability) — the discovered language and its posterior. When supported is given, the top-ranked code within it.

Return type:

(str, float)

Examples

>>> code, prob = detect_language(pcm)
>>> code
'fr'
vocal_helper.lid.detect_language_regions(pcm, sample_rate=16000, *, model='large-v3-turbo-q5_0', threads=6, window_s=10.0, hop_s=3.0, smooth_s=6.0, min_region_s=8.0, refine_s=4.0, snap_s=1.0, supported=None)[source]

Partition pcm into mono-language regions (posterior-curve method).

A language switch has no sharp acoustic cue, so instead of hunting for a boundary we (1) sample a posterior curve over overlapping windows, (2) Gaussian-smooth it, (3) take the per-frame argmax, (4) place change points where it flips, (5) absorb sub-min_region_s regions, (6) locally refine each change point, and (7) snap it to the nearest silence. Each region’s language is discovered from the audio, never defaulted. The regions must be transcribed before committing to a language, so each is transcribed in its own. supported is an opt-in routing whitelist (None = discover freely, see detect_language()).

Empty or too-short-to-identify audio has no language to discover, so this returns an empty list (no region invented) rather than guessing one.

Parameters:
Return type:

list[LangRegion]

vocal_helper.lid.detect_language_regions_fast(pcm, sample_rate=16000, *, conf_gate=0.5, model='large-v3-turbo-q5_0', threads=6, supported=None)[source]

Partition pcm into language regions, fast path for monolingual audio.

Most real recordings are one language end to end, yet detect_language_regions() pays for a full overlapping-window posterior scan on every file. This wrapper first runs a single cheap global detect_language() over the whole signal: when that detection clears conf_gate on a routable language the file is taken to be monolingual and returned as one region (a single whisper call instead of dozens). Only when the global detection is uncertain — a genuinely code-switched or noisy file — does it fall back to the accurate posterior-curve segmentation of detect_language_regions().

On a corpus of support-call recordings this cut per-file language identification from ~73 s to ~1 s with identical region output on the monolingual majority, while the low-confidence fallback still recovers the switches. It is a drop-in replacement for detect_language_regions() wherever code-switching is the exception rather than the rule.

Parameters:

  • pcm (NDArray[np.float32]) – Mono waveform at sample_rate.

  • sample_rate (int, optional) – Sample rate of pcm in Hz (default DEFAULT_SR).

  • conf_gate (float, optional) – Minimum whole-file detection probability required to accept the single-region fast path (default DEFAULT_FAST_CONF_GATE). Below it, the robust detect_language_regions() scan runs.

  • model (str, optional) – whisper.cpp model name (default DEFAULT_MODEL).

  • threads (int, optional) – Inference threads (default DEFAULT_THREADS).

  • supported (tuple[str, ...] or None, optional) – Opt-in routable-language whitelist. None (default) discovers the language freely (see detect_language()).

Returns:

A single region spanning the file on the fast path, or the full multi-region partition on the fallback path. Always ≥ 1 region.

Return type:

list[LangRegion]

Examples

>>> regions = detect_language_regions_fast(pcm, 16_000)
>>> regions[0].lang
'en'
vocal_helper.lid.detect_language_speechbrain(pcm, *, supported=None)[source]

Identify the language of pcm with SpeechBrain VoxLingua107.

An ECAPA-TDNN spoken-language classifier that shares nothing with whisper but the audio — the independent second opinion. VoxLingua107 labels are "<iso>: <English name>" (e.g. "fr: French"); the ISO-639-1 prefix is returned. By default the classifier’s true label is returned so the second opinion stays genuinely independent — it is never silently remapped to a preferred language. Pass supported only if you want an out-of-whitelist label dropped to the closest routable one.

Parameters:

  • pcm (NDArray[np.float32]) – Mono waveform to classify.

  • supported (tuple[str, ...] or None, optional) – Opt-in routable-language whitelist. None (default) returns the true VoxLingua107 label, even if it is outside any routing set. A tuple re-ranks the classifier’s posterior within that set instead.

Returns:

(iso_639_1_code, probability).

Return type:

(str, float)

vocal_helper.lid.language_posterior_curve(pcm, sample_rate=16000, *, model='large-v3-turbo-q5_0', threads=6, window_s=10.0, hop_s=3.0, supported=None)[source]

Sample a per-window language posterior over time.

Returns (centers[T], langs[L], P[T, L]) where centers are window centre times (s), langs the candidate codes, and each row of P a posterior over langs from whisper’s auto_detect_language on a window_s window centred there. Overlapping windows (hop_s < window_s) give a finely sampled curve.

supported is opt-in: None (default) discovers freely — the posterior axis is whisper’s full language head (adopted from the first usable window) so any language can surface. A tuple restricts (and renormalises over) that routable set instead.

Parameters:
Return type:

tuple[ndarray[tuple[Any, …], dtype[float64]], list[str], ndarray[tuple[Any, …], dtype[float64]]]