speaker_helper.eval.multilang module

Multi-language measurement: run the evaluation across several languages.

Module summary

A single engine (kokoro speaks many languages) does not perform identically in every language. This module runs the evaluation gate once per language — each with its own bundled dataset and a voice auto-picked for that language — and returns one EvalReport per language. That is the raw material for a language × (speed, quality) matrix: the product-side counterpart of choosing an operating point per language.

Because it drives the engine-agnostic Speaker, the same call measures the deterministic mock backend (for tests) or a real engine (for real numbers) — only settings.backend differs.

Usage example

>>> import asyncio
>>> from speaker_helper import Settings
>>> from speaker_helper.eval import run_multilang_eval
>>> reports = asyncio.run(run_multilang_eval(
...     Settings.from_mapping({"backend": "mock"}), ["fr", "en", "es"]))
>>> sorted(reports)
['en', 'es', 'fr']

Author

Warith HARCHAOUI — https://linkedin.com/in/warith-harchaoui

speaker_helper.eval.multilang.format_matrix(reports)[source]

Render a per-language report mapping as a compact text matrix.

Parameters:

reports (dict) – Mapping language -> EvalReport (e.g. from run_multilang_eval()).

Returns:

A fixed-width table with one row per language and a PASS/FAIL verdict.

Return type:

str

async speaker_helper.eval.multilang.run_multilang_eval(base_settings, languages, *, thresholds=None, transcriber=None, warmup=True)[source]

Evaluate an engine across several languages, one report per language.

Parameters:

  • base_settings (Settings) – Base configuration (backend, engine, connection). Its language and voice_id are overridden per language so the engine auto-picks a voice matching each language.

  • languages (list of str) – ISO-639-1 codes to measure; each must have a bundled dataset (see available_languages()).

  • thresholds (Thresholds or None) – Pass/fail bar applied to every language; defaults to the bundled bar.

  • transcriber (Transcriber or None) – Optional STT enabling the WER/chrF round-trip in each language.

  • warmup (bool) – When True (default), warm each language’s voice before measuring so the numbers reflect steady state rather than a one-off cold model/voice load. Set False to include cold-start cost.

Returns:

Mapping language -> EvalReport in the requested order.

Return type:

dict