vocal_helper.eot_bench module

vocal_helper.eot_bench

End-of-turn detection benchmark utilities — reusable evaluation functions inspired by LiveKit’s eot-bench methodology [@livekitturndetector] (April 2026 whitepaper).

Motivation

Every EOT / turn-taking study we run needs the same primitives :

  • A ground-truth turn boundary (the moment speaker A actually stops holding the floor).

  • A detector’s commit boundary (the moment the detector emits “turn ended, hand over”).

  • A latency band under which we score the detector (e.g. 300 ms, 600 ms, 1200 ms).

  • A pair of failure modes to count : * false-cutoff — the detector fired before the true turn

    end (interrupts the speaker prematurely).

    • hang — the detector waited too long after the true turn end (perceived as agent hesitation).

This module implements both metrics as pure functions of aligned lists of (true_turn_end_s, detector_commit_s) pairs. Every call is deterministic ; no models are loaded here. The consumer brings the ground truth (from RTTM / manual annotation) and the detector output ; this module scores them.

Origin — the LiveKit paper reports :

  • 9.9 % false-cutoff at 300 ms median semantic latency

  • 4.5 % false-cutoff at 600 ms

  • 3.0 % false-cutoff at 1200 ms

on their proprietary conversational corpus. We adopt the same three latency bands as the default reporting protocol.

Public surface

  • EOTPair — one (true_turn_end, detector_commit) datum.

  • false_cutoff_rate() — fraction of pairs where commit < true_end - tolerance.

  • hang_rate() — fraction of pairs where commit > true_end + tolerance.

  • score() — full breakdown at the standard {300, 600, 1200} ms latency bands.

Author

Warith Harchaoui — https://www.linkedin.com/in/warith-harchaoui/

class vocal_helper.eot_bench.EOTPair(true_turn_end_s, detector_commit_s)[source]

Bases: object

One aligned ground-truth / detector datum.

Parameters:

  • true_turn_end_s (float)

  • detector_commit_s (float)

true_turn_end_s

Absolute (mix-time) second at which the speaker actually released the floor. In practice pulled from a hand-labelled turn boundary or a bridged word-level RTTM.

Type:

float

detector_commit_s

Absolute second at which the detector emitted its end-of-turn decision. For a semantic-EOT model this is typically true_turn_end + inference_latency on hits.

Type:

float

detector_commit_s: float
true_turn_end_s: float
vocal_helper.eot_bench.false_cutoff_rate(pairs, *, tolerance_s)[source]

Share of pairs where the detector fired tolerance_s early.

A “false cutoff” is a decision to end the turn while the speaker is still speaking — the perceptual failure LiveKit’s metric targets. Setting tolerance_s = 0 treats any early commit as a false cutoff ; realistic tolerances are 50-150 ms (below human turn-taking noise floor).

Parameters:
Return type:

float

vocal_helper.eot_bench.hang_rate(pairs, *, latency_budget_s)[source]

Share of pairs where the detector waited > latency_budget_s.

The other-side failure : the speaker finished but the agent kept waiting, producing perceived agent hesitation. latency_budget_s is the acceptable upper bound (LiveKit uses 300 / 600 / 1200 ms bands).

Parameters:
Return type:

float

vocal_helper.eot_bench.median_latency_s(pairs)[source]

Median signed latency across the corpus.

Parameters:

pairs (Sequence[EOTPair])

Return type:

float

vocal_helper.eot_bench.score(pairs, *, latency_bands_ms=(300, 600, 1200), tolerance_ms=50)[source]

Full LiveKit-style report on a set of paired detections.

Parameters:

  • pairs (Iterable[EOTPair]) – Iterable of aligned (true_end, detector_commit) pairs.

  • latency_bands_ms (Sequence[int]) – Latency budgets to score the hang rate against. Default is LiveKit’s canonical (300, 600, 1200) ms bands.

  • tolerance_ms (int) – Acceptable early-commit slack before we count a false-cutoff. Default 50 ms sits below human turn-taking noise floor (Heldner & Edlund 2010 median gap 200 ms).

Returns:

{"n": int, "median_latency_ms": float, "false_cutoff_rate": float, "hang_rate_at_ms": {band: rate, ...}}.

Return type:

dict