speaker_helper.eval package

Submodules

Module contents

speaker-helper evaluation layer — measure speed and quality, gate on thresholds.

Module summary

The project’s coding contract forbids “vibe checks”: anything AI must be evaluated against a committed dataset with versioned metrics and thresholds. This package provides speaker-helper’s measurement machinery and makes it a first-class, CI-gating layer.

It is engine-agnostic: everything runs against the Speaker façade, so the exact same evaluation grades the deterministic mock backend in CI or the real Voicebox kokoro engine locally — only settings.backend differs.

Public surface

load_dataset(), EvalCase
Thresholds
run_eval(), EvalReport, CaseResult, Transcriber
metrics: word_error_rate(), chrf(), detect_anomalies(), percentile()

Author

Warith HARCHAOUI — https://linkedin.com/in/warith-harchaoui

class speaker_helper.eval.CaseResult(id, language, duration_s, rtf, anomalies=<factory>, wer=None, chrf=None, error=None)[source]

Bases: object

Per-utterance evaluation outcome.

Parameters:

id (str) – The originating EvalCase id.
language (str) – Language synthesised.
duration_s (float) – Audio duration produced.
rtf (float) – Real-time factor for this case (below 1.0 is faster than real time).
anomalies (list of str) – Anomaly labels (empty means clean); see detect_anomalies().
wer (float or None) – Round-trip word error rate, or None when no transcriber ran.
chrf (float or None) – Round-trip chrF, or None when no transcriber ran.
error (str or None) – Synthesis error message when the case failed outright (counts as an anomaly), else None.

anomalies: list[str]

chrf: float | None = None

duration_s: float

error: str | None = None

id: str

property is_anomalous: bool: True if the case errored or produced any audio anomaly.

language: str

rtf: float

wer: float | None = None

class speaker_helper.eval.EvalCase(id, text, language='fr', reference='')[source]

Bases: object

One reference utterance to synthesise and score.

Parameters:

id (str) – Stable identifier (used in reports and to diff runs).
text (str) – The text to synthesise.
language (str) – ISO-639-1 language code to synthesise in.
reference (str) – Ground-truth transcript to compare a re-transcription against. Defaults to text when a dataset line omits it.

id: str

language: str = 'fr'

reference: str = ''

text: str

class speaker_helper.eval.EvalReport(backend, n_cases, mean_rtf, p95_rtf, anomaly_rate, mean_wer, mean_chrf, quality, quality_source, passed, failures, cases, thresholds)[source]

Bases: object

Aggregated evaluation results plus the pass/fail verdict.

Parameters:

backend (str) – The backend that was evaluated (echoed for provenance).
n_cases (int) – Number of cases run.
mean_rtf (float) – RTF summary statistics.
p95_rtf (float) – RTF summary statistics.
anomaly_rate (float) – Fraction of cases that errored or were anomalous.
mean_wer (float or None) – Round-trip fidelity means, or None when no transcriber ran.
mean_chrf (float or None) – Round-trip fidelity means, or None when no transcriber ran.
quality (float) – A quality score in [0, 1] that is always present: the measured mean chrF when a transcriber ran, otherwise the engine’s inherited prior (see speaker_helper.eval.priors). Paired with mean_rtf it places the run on the quality↔RTF plane for pareto_front().
quality_source (str) – "measured_chrf" or "prior" — how quality was obtained.
passed (bool) – Whether every gated threshold was met.
failures (list of str) – Human-readable reasons the run failed (empty when passed).
cases (list of CaseResult) – Per-utterance detail.
thresholds (dict) – The thresholds applied (echoed for provenance).

anomaly_rate: float

backend: str

cases: list[CaseResult]

failures: list[str]

mean_chrf: float | None

mean_rtf: float

mean_wer: float | None

n_cases: int

p95_rtf: float

passed: bool

quality: float

quality_source: str

thresholds: dict

to_dict()[source]

Return a JSON-serialisable dict of the whole report.

Return type:: dict

class speaker_helper.eval.Thresholds(max_mean_rtf=1.0, max_p95_rtf=1.0, max_anomaly_rate=0.0, max_mean_wer=None, min_mean_chrf=None)[source]

Bases: object

The bar an evaluation run must clear to pass.

Parameters:

max_mean_rtf (float) – Maximum allowed mean real-time factor across cases.
max_p95_rtf (float) – Maximum allowed 95th-percentile RTF (guards tail latency). 1.0 means even the slow cases stay faster than real time.
max_anomaly_rate (float) – Maximum allowed fraction of cases with any audio anomaly (0.0 = none tolerated).
max_mean_wer (float or None) – Maximum allowed mean word error rate of the re-transcription round-trip. None disables the gate (used when no transcriber is provided).
min_mean_chrf (float or None) – Minimum allowed mean chrF of the round-trip. None disables the gate.

classmethod load(path=None)[source]

Load thresholds from a YAML file, falling back to defaults.

Parameters:: path (str or Path or None) – YAML file with a subset of the threshold fields. When None, the bundled thresholds.yaml is used if present; otherwise the dataclass defaults apply. Unknown keys are ignored.
Returns:: The populated thresholds.
Return type:: Thresholds

max_anomaly_rate: float = 0.0

max_mean_rtf: float = 1.0

max_mean_wer: float | None = None

max_p95_rtf: float = 1.0

min_mean_chrf: float | None = None

class speaker_helper.eval.Transcriber(*args, **kwargs)[source]

Bases: Protocol

Optional speech-to-text used for the fidelity round-trip.

Any object with this coroutine can be injected (e.g. a vocal-helper adapter) to enable WER/chrF scoring. speaker-helper never imports one.

async transcribe(wav_bytes, *, language)[source]

Return the transcript of a WAV payload in language.

Parameters:

wav_bytes (bytes)
language (str)

Return type:

str

speaker_helper.eval.available_languages()[source]

Return the sorted language codes that ship a built-in dataset.

Return type:: list[str]

speaker_helper.eval.chrf(reference, hypothesis, *, max_n=6, beta=2.0)[source]

Character n-gram F-score (chrF) between reference and hypothesis.

A robust, tokenisation-free similarity that credits partial-word overlap — well suited to speech round-trips where the transcriber may mis-segment.

Parameters:

reference (str) – Ground-truth text.
hypothesis (str) – Transcript of the synthesised audio.
max_n (int) – Highest character n-gram order to average over (1..``max_n``).
beta (float) – Weight of recall relative to precision (chrF’s conventional beta=2).

Returns:

Score in [0, 1]; 1.0 is identical (after normalisation), 0.0 no shared n-grams. Returns 1.0 when both are empty.

Return type:

float

Examples

>>> round(chrf("bonjour le monde", "bonjour le monde"), 3)
1.0

speaker_helper.eval.detect_anomalies(result, *, min_chars_per_sec=4.0, max_chars_per_sec=40.0, clipping_threshold=0.999)[source]

Return a list of anomaly labels for one synthesised result (empty = clean).

The checks are engine-agnostic and decidable from the audio alone:

empty_audio — zero or near-zero duration.
invalid_sample_rate — non-positive sample rate.
duration_too_short / duration_too_long — the characters-per-second implied by len(text) / duration falls outside the plausible band, signalling truncation or a runaway stretch.
clipping — a sample reaches full scale.

Parameters:

result (AudioResult) – The synthesised audio and its metadata.
min_chars_per_sec (float) – Plausible speaking-rate band (see module constants).
max_chars_per_sec (float) – Plausible speaking-rate band (see module constants).
clipping_threshold (float) – Absolute amplitude at or above which audio is deemed clipped.

Returns:

Sorted anomaly labels; an empty list means no anomaly was found.

Return type:

list of str

speaker_helper.eval.engine_quality_prior(engine)[source]

Return the prior quality score in [0, 1] for a TTS engine.

Parameters:: engine (str) – Engine id (e.g. "kokoro").
Returns:: The inherited prior, or 0.80 for an unknown engine.
Return type:: float

speaker_helper.eval.format_matrix(reports)[source]

Render a per-language report mapping as a compact text matrix.

Parameters:: reports (dict) – Mapping language -> EvalReport (e.g. from run_multilang_eval()).
Returns:: A fixed-width table with one row per language and a PASS/FAIL verdict.
Return type:: str

speaker_helper.eval.load_dataset(path=None, *, language=None)[source]

Load evaluation cases from a JSON Lines file.

Parameters:

path (str or Path or None) – Explicit dataset file. Takes precedence over language.
language (str or None) – When path is None, load the bundled <language>_reference.jsonl set. When both are None, the default (French) set is loaded.

Returns:

The cases in file order.

Return type:

list of EvalCase

Raises:

FileNotFoundError – If the resolved dataset does not exist (the error lists the languages that do ship a dataset).
ValueError – If a line is not valid JSON or lacks the required id/text keys.

speaker_helper.eval.pareto_front(points)[source]

Return the non-dominated points: maximise quality, minimise RTF.

A point A dominates B if A is at least as good on both axes (higher or equal quality, lower or equal mean_rtf) and strictly better on one.

Parameters:: points (sequence) – Candidates, each exposing quality and mean_rtf (e.g. EvalReport).
Returns:: The Pareto-optimal subset, in input order.
Return type:: list

speaker_helper.eval.percentile(values, p)[source]

Return the p-th percentile (0-100) via linear interpolation.

Parameters:

values (sequence of float) – Samples (need not be sorted).
p (float) – Percentile in [0, 100].

Returns:

The interpolated percentile, or 0.0 for an empty input.

Return type:

float

Examples

>>> percentile([1, 2, 3, 4], 50)
2.5
>>> percentile([], 95)
0.0

async speaker_helper.eval.run_eval(speaker, cases=None, *, thresholds=None, transcriber=None)[source]

Evaluate speaker over cases and gate on thresholds.

Parameters:

speaker (Speaker) – The façade to drive; its backend is whatever settings.backend chose.
cases (list of EvalCase or None) – Cases to run; defaults to the built-in French dataset.
thresholds (Thresholds or None) – Pass/fail bar; defaults to Thresholds.load() (bundled YAML).
transcriber (Transcriber or None) – Optional STT enabling the WER/chrF round-trip. When None, fidelity metrics are skipped (and their thresholds are not gated).

Returns:

The aggregated report and verdict.

Return type:

EvalReport

async speaker_helper.eval.run_multilang_eval(base_settings, languages, *, thresholds=None, transcriber=None, warmup=True)[source]

Evaluate an engine across several languages, one report per language.

Parameters:

base_settings (Settings) – Base configuration (backend, engine, connection). Its language and voice_id are overridden per language so the engine auto-picks a voice matching each language.
languages (list of str) – ISO-639-1 codes to measure; each must have a bundled dataset (see available_languages()).
thresholds (Thresholds or None) – Pass/fail bar applied to every language; defaults to the bundled bar.
transcriber (Transcriber or None) – Optional STT enabling the WER/chrF round-trip in each language.
warmup (bool) – When True (default), warm each language’s voice before measuring so the numbers reflect steady state rather than a one-off cold model/voice load. Set False to include cold-start cost.

Returns:

Mapping language -> EvalReport in the requested order.

Return type:

dict

speaker_helper.eval.word_error_rate(reference, hypothesis)[source]

Word error rate between a reference and a hypothesis transcript.

Parameters:

reference (str) – The ground-truth text (what should have been spoken).
hypothesis (str) – The transcript of the synthesised audio.

Returns:

edit_distance(ref_words, hyp_words) / len(ref_words). 0.0 for an exact match; can exceed 1.0 when the hypothesis inserts many words. Returns 0.0 if both are empty and 1.0 if only the reference is.

Return type:

float

Examples

>>> word_error_rate("le chat dort", "le chien dort")
0.3333333333333333