speaker_helper.eval.deepeval_metrics module

DeepEval metrics wrapping speaker-helper’s speed and integrity measurements.

Module summary

The evaluation contract asks for a dedicated evaluation framework, not just hand-rolled asserts. speaker-helper’s native metrics (RTF, audio anomalies) are audio-specific, so instead of forcing a text-only framework to measure audio we adapt those measurements as DeepEval custom metrics. That plugs speaker-helper into DeepEval’s test-case model, reporting, and thresholds while keeping the actual measurement in this project.

Both metrics are deterministic and offline — they read values a caller stores in test_case.additional_metadata (rtf and anomalies), so no LLM, API key, or network is involved.

deepeval is imported at module import time, so this module is optional: it is only imported by callers that installed the eval extra.

Usage example

>>> # from deepeval.test_case import LLMTestCase
>>> # tc = LLMTestCase(input="Bonjour.", actual_output="<audio>",
>>> #                   additional_metadata={"rtf": 0.3, "anomalies": []})
>>> # RealTimeFactorMetric(threshold=1.0).measure(tc)  # -> 1.0 (passes)

Author

Warith HARCHAOUI — https://linkedin.com/in/warith-harchaoui

class speaker_helper.eval.deepeval_metrics.AudioIntegrityMetric(*args, **kwargs)[source]

Bases: BaseMetric

DeepEval metric: passes when the synthesised audio has no anomaly.

Notes

Reads test_case.additional_metadata['anomalies'] (a list of labels from detect_anomalies()); an empty list passes.

async a_measure(test_case)[source]

Async wrapper around measure() (the work is synchronous).

Parameters:: test_case (Any)
Return type:: float

is_successful()[source]

Return whether the last measure() found clean audio.

Return type:: bool

measure(test_case)[source]

Score 1.0 when there are no audio anomalies, else 0.0.

Parameters:: test_case (Any)
Return type:: float

class speaker_helper.eval.deepeval_metrics.RealTimeFactorMetric(*args, **kwargs)[source]

Bases: BaseMetric

DeepEval metric: passes when synthesis is faster than real time.

Parameters:: threshold (float) – Maximum acceptable real-time factor (compute_s / duration_s). Defaults to 1.0 — the operating point (faster than real time).

Notes

Reads test_case.additional_metadata['rtf']; a missing value scores 0.

async a_measure(test_case)[source]

Async wrapper around measure() (the work is synchronous).

Parameters:: test_case (Any)
Return type:: float

is_successful()[source]

Return whether the last measure() passed the threshold.

Return type:: bool

measure(test_case)[source]

Score 1.0 if the case’s RTF is below the threshold, else 0.0.

Parameters:: test_case (Any)
Return type:: float

class speaker_helper.eval.deepeval_metrics.RoundTripIdempotenceMetric(*args, **kwargs)[source]

Bases: BaseMetric

DeepEval metric: text → speech → text should recover the text.

This formalises the idempotence view of synthesis quality: if you synthesise a sentence and transcribe the audio back (with vocal-helper, the STT sibling), an intelligible voice yields a transcript close to the original. We score that closeness with chrF (character n-gram F-score, robust to the ASR’s tokenisation) and pass when it clears a bar.

What this does and does not measure

It measures intelligibility — a necessary lower bound on quality — not naturalness (a clear-but-robotic voice can still score high). It also couples TTS quality with the ASR’s quality, so absolute chrF is best read as a relative ranking across engines scored by the same ASR (the ASR’s error is largely common-mode and cancels when comparing engines). For a naturalness axis, pair this with a MOS predictor (e.g. UTMOSv2) measured in the speak study.

param threshold:: Minimum acceptable round-trip chrF in [0, 1]. Defaults to 0.75, matching the bundled fidelity bar (thresholds.yaml: min_mean_chrf).
type threshold:: float

Notes

Reads test_case.metadata['chrf'] (the round-trip chrF a caller stored from chrf()); a missing value scores 0.

async a_measure(test_case)[source]

Async wrapper around measure() (the work is synchronous).

Parameters:: test_case (Any)
Return type:: float

is_successful()[source]

Return whether the last measure() cleared the chrF bar.

Return type:: bool

measure(test_case)[source]

Score the round-trip chrF (0 when unmeasured) and gate on the bar.

Parameters:: test_case (Any)
Return type:: float

Parameters:: threshold (float)