speaker_helper.eval.metrics module

Dependency-free quality and speed metrics for synthesised speech.

Module summary

This is speaker-helper’s measurement machinery, built around its AudioResult. It answers two questions for every utterance — is it fast? and is it good? — with pure-Python functions that need no model and no network:

Speed. percentile() over per-utterance real-time factors (RTF).
Signal anomalies. detect_anomalies() flags empty audio, aberrant duration for the amount of text, clipping, and invalid sample rates — the “TTS anomaly rate” gate. It decodes the WAV with soundfile (imported lazily) but computes everything else in the standard library.
Fidelity (optional round-trip). When a transcriber is available the audio can be re-transcribed and compared to the reference text with word_error_rate() (WER) and chrf() (character n-gram F-score). Both are implemented here rather than pulled from jiwer / sacrebleu so the core evaluation has zero heavy dependencies.

Usage example

>>> from speaker_helper.eval.metrics import word_error_rate, chrf
>>> word_error_rate("le chat dort", "le chat dort")
0.0
>>> round(chrf("bonjour", "bonjour"), 3)
1.0

Author

Warith HARCHAOUI — https://linkedin.com/in/warith-harchaoui

speaker_helper.eval.metrics.chrf(reference, hypothesis, *, max_n=6, beta=2.0)[source]

Character n-gram F-score (chrF) between reference and hypothesis.

A robust, tokenisation-free similarity that credits partial-word overlap — well suited to speech round-trips where the transcriber may mis-segment.

Parameters:

reference (str) – Ground-truth text.
hypothesis (str) – Transcript of the synthesised audio.
max_n (int) – Highest character n-gram order to average over (1..``max_n``).
beta (float) – Weight of recall relative to precision (chrF’s conventional beta=2).

Returns:

Score in [0, 1]; 1.0 is identical (after normalisation), 0.0 no shared n-grams. Returns 1.0 when both are empty.

Return type:

float

Examples

>>> round(chrf("bonjour le monde", "bonjour le monde"), 3)
1.0

speaker_helper.eval.metrics.detect_anomalies(result, *, min_chars_per_sec=4.0, max_chars_per_sec=40.0, clipping_threshold=0.999)[source]

Return a list of anomaly labels for one synthesised result (empty = clean).

The checks are engine-agnostic and decidable from the audio alone:

empty_audio — zero or near-zero duration.
invalid_sample_rate — non-positive sample rate.
duration_too_short / duration_too_long — the characters-per-second implied by len(text) / duration falls outside the plausible band, signalling truncation or a runaway stretch.
clipping — a sample reaches full scale.

Parameters:

result (AudioResult) – The synthesised audio and its metadata.
min_chars_per_sec (float) – Plausible speaking-rate band (see module constants).
max_chars_per_sec (float) – Plausible speaking-rate band (see module constants).
clipping_threshold (float) – Absolute amplitude at or above which audio is deemed clipped.

Returns:

Sorted anomaly labels; an empty list means no anomaly was found.

Return type:

list of str

speaker_helper.eval.metrics.mean(values)[source]

Return the arithmetic mean, or 0.0 for an empty input.

Parameters:: values (Sequence[float])
Return type:: float

speaker_helper.eval.metrics.normalize_text(text)[source]

Lowercase, strip accents/punctuation, and collapse whitespace.

A deliberately aggressive normalisation so WER/chrF measure what was said, not casing, accentuation, or punctuation the TTS/STT pair may drop.

Parameters:: text (str) – Raw text.
Returns:: Normalised text (lowercase ASCII words separated by single spaces).
Return type:: str

Examples

>>> normalize_text("Où est le Café ?")
'ou est le cafe'

speaker_helper.eval.metrics.peak_amplitude(wav_bytes)[source]

Return the peak absolute sample amplitude of a WAV payload (0.0 if empty).

Parameters:: wav_bytes (bytes)
Return type:: float

speaker_helper.eval.metrics.percentile(values, p)[source]

Return the p-th percentile (0-100) via linear interpolation.

Parameters:

values (sequence of float) – Samples (need not be sorted).
p (float) – Percentile in [0, 100].

Returns:

The interpolated percentile, or 0.0 for an empty input.

Return type:

float

Examples

>>> percentile([1, 2, 3, 4], 50)
2.5
>>> percentile([], 95)
0.0

speaker_helper.eval.metrics.word_error_rate(reference, hypothesis)[source]

Word error rate between a reference and a hypothesis transcript.

Parameters:

reference (str) – The ground-truth text (what should have been spoken).
hypothesis (str) – The transcript of the synthesised audio.

Returns:

edit_distance(ref_words, hyp_words) / len(ref_words). 0.0 for an exact match; can exceed 1.0 when the hypothesis inserts many words. Returns 0.0 if both are empty and 1.0 if only the reference is.

Return type:

float

Examples

>>> word_error_rate("le chat dort", "le chien dort")
0.3333333333333333