speaker_helper.eval.dataset module

Evaluation datasets: reference utterances the engine must speak well.

Module summary

An evaluation is only reproducible if the inputs are versioned alongside the code. This module defines EvalCase (one reference utterance) and loads JSON Lines datasets — the built-in French set data/fr_reference.jsonl ships in the package, and callers may point at their own file.

Each line is a JSON object with id, language, text and, optionally, a reference transcript to score a re-transcription round-trip against (defaulting to text itself).

Usage example

>>> from speaker_helper.eval.dataset import load_dataset
>>> cases = load_dataset()          # the built-in French set
>>> cases[0].language
'fr'

Author

Warith HARCHAOUI — https://linkedin.com/in/warith-harchaoui

class speaker_helper.eval.dataset.EvalCase(id, text, language='fr', reference='')

Bases: object

One reference utterance to synthesise and score.

Parameters:

• id (str) – Stable identifier (used in reports and to diff runs).

• text (str) – The text to synthesise.

• language (str) – ISO-639-1 language code to synthesise in.

• reference (str) – Ground-truth transcript to compare a re-transcription against. Defaults to text when a dataset line omits it.

id: str

language: str = 'fr'

reference: str = ''

text: str

speaker_helper.eval.dataset.available_languages()

Return the sorted language codes that ship a built-in dataset.

Return type:

list[str]

speaker_helper.eval.dataset.dataset_path_for_language(language)

Return the bundled dataset path for a language code (e.g. "en").

Parameters:

language (str)

Return type:

Path

speaker_helper.eval.dataset.load_dataset(path=None, *, language=None)

Load evaluation cases from a JSON Lines file.

Parameters:

• path (str or Path or None) – Explicit dataset file. Takes precedence over language.

• language (str or None) – When path is None, load the bundled <language>_reference.jsonl set. When both are None, the default (French) set is loaded.

Returns:

The cases in file order.

Return type:

list of EvalCase

Raises:

• FileNotFoundError – If the resolved dataset does not exist (the error lists the languages that do ship a dataset).

• ValueError – If a line is not valid JSON or lacks the required id/text keys.