speaker_helper.eval.runner module

Run an evaluation over a dataset and gate on versioned thresholds.

Module summary

The runner ties the pieces together: it drives a Speaker (any backend — Voicebox, mock, a future engine) over a list of EvalCase, measures speed and signal integrity for each, optionally re-transcribes the audio to score fidelity, aggregates everything into an EvalReport, and compares that report to committed Thresholds.

Because it targets the Speaker façade, the evaluation is completely engine-agnostic: the same run measures the mock backend in CI or the real Voicebox engine locally, with no code change — only settings.backend differs.

Fidelity (WER/chrF) is a round-trip: it needs a speech-to-text transcriber. That is injected through the Transcriber protocol (e.g. a vocal-helper adapter), so speaker-helper takes no hard STT dependency. Without one, speed and anomaly gates still run.

Usage example

>>> import asyncio
>>> from speaker_helper import Speaker, Settings
>>> from speaker_helper.eval import load_dataset, run_eval
>>> async def demo() -> bool:
...     async with Speaker(Settings.from_mapping({"backend": "mock"})) as spk:
...         report = await run_eval(spk, load_dataset())
...         return report.passed
>>> asyncio.run(demo())
True

Author

Warith HARCHAOUI — https://linkedin.com/in/warith-harchaoui

class speaker_helper.eval.runner.CaseResult(id, language, duration_s, rtf, anomalies=<factory>, wer=None, chrf=None, error=None)[source]

Bases: object

Per-utterance evaluation outcome.

Parameters:

  • id (str) – The originating EvalCase id.

  • language (str) – Language synthesised.

  • duration_s (float) – Audio duration produced.

  • rtf (float) – Real-time factor for this case (below 1.0 is faster than real time).

  • anomalies (list of str) – Anomaly labels (empty means clean); see detect_anomalies().

  • wer (float or None) – Round-trip word error rate, or None when no transcriber ran.

  • chrf (float or None) – Round-trip chrF, or None when no transcriber ran.

  • error (str or None) – Synthesis error message when the case failed outright (counts as an anomaly), else None.

anomalies: list[str]
chrf: float | None = None
duration_s: float
error: str | None = None
id: str
property is_anomalous: bool

True if the case errored or produced any audio anomaly.

language: str
rtf: float
wer: float | None = None
class speaker_helper.eval.runner.EvalReport(backend, n_cases, mean_rtf, p95_rtf, anomaly_rate, mean_wer, mean_chrf, quality, quality_source, passed, failures, cases, thresholds)[source]

Bases: object

Aggregated evaluation results plus the pass/fail verdict.

Parameters:

  • backend (str) – The backend that was evaluated (echoed for provenance).

  • n_cases (int) – Number of cases run.

  • mean_rtf (float) – RTF summary statistics.

  • p95_rtf (float) – RTF summary statistics.

  • anomaly_rate (float) – Fraction of cases that errored or were anomalous.

  • mean_wer (float or None) – Round-trip fidelity means, or None when no transcriber ran.

  • mean_chrf (float or None) – Round-trip fidelity means, or None when no transcriber ran.

  • quality (float) – A quality score in [0, 1] that is always present: the measured mean chrF when a transcriber ran, otherwise the engine’s inherited prior (see speaker_helper.eval.priors). Paired with mean_rtf it places the run on the quality↔RTF plane for pareto_front().

  • quality_source (str) – "measured_chrf" or "prior" — how quality was obtained.

  • passed (bool) – Whether every gated threshold was met.

  • failures (list of str) – Human-readable reasons the run failed (empty when passed).

  • cases (list of CaseResult) – Per-utterance detail.

  • thresholds (dict) – The thresholds applied (echoed for provenance).

anomaly_rate: float
backend: str
cases: list[CaseResult]
failures: list[str]
mean_chrf: float | None
mean_rtf: float
mean_wer: float | None
n_cases: int
p95_rtf: float
passed: bool
quality: float
quality_source: str
thresholds: dict
to_dict()[source]

Return a JSON-serialisable dict of the whole report.

Return type:

dict

class speaker_helper.eval.runner.Transcriber(*args, **kwargs)[source]

Bases: Protocol

Optional speech-to-text used for the fidelity round-trip.

Any object with this coroutine can be injected (e.g. a vocal-helper adapter) to enable WER/chrF scoring. speaker-helper never imports one.

async transcribe(wav_bytes, *, language)[source]

Return the transcript of a WAV payload in language.

Parameters:
Return type:

str

async speaker_helper.eval.runner.run_eval(speaker, cases=None, *, thresholds=None, transcriber=None)[source]

Evaluate speaker over cases and gate on thresholds.

Parameters:

  • speaker (Speaker) – The façade to drive; its backend is whatever settings.backend chose.

  • cases (list of EvalCase or None) – Cases to run; defaults to the built-in French dataset.

  • thresholds (Thresholds or None) – Pass/fail bar; defaults to Thresholds.load() (bundled YAML).

  • transcriber (Transcriber or None) – Optional STT enabling the WER/chrF round-trip. When None, fidelity metrics are skipped (and their thresholds are not gated).

Returns:

The aggregated report and verdict.

Return type:

EvalReport