video_helper.faces.sampling module

video_helper.faces.sampling

The smart-sampling harness: the piece that makes Active-Speaker Detection affordable on a full recording.

Heavy ASD is never run over the whole video. Instead:

  1. Shots: PySceneDetect segments the video into shots (cheap; skipped gracefully if scenedetect is absent → one shot).

  2. Face census: a cheap low-fps YuNet-only pass counts faces and records where (which time regions) faces appear. No ASD here.

  3. Candidate windows: for each audio speaker cluster, short windows are proposed where that cluster speaks and a face is on screen, spread across distinct shots for diversity.

  4. Iterative ASD: heavy ASD runs only on a small batch of windows; per-face speaking scores vote each cluster onto a global face identity (tracks are stitched across windows/shots by face embedding, which also counts the faces along the video). After each round the assignment is checked for certainty (vote margin + coverage); clusters still uncertain get more windows, up to a hard clip budget.

Output: per cluster, the assigned global face, its coverage and certainty margin, and the best crops collected (so the caller can embed the face without decoding the video again).

class video_helper.faces.sampling.SpeakerFaceAssignment(speaker, face_id, coverage, margin, crops=<factory>)[source]

Bases: object

The face assigned to one diarization cluster.

Parameters:
speaker

Diarization cluster label.

Type:

int

face_id

Global face identity (stable across the whole video).

Type:

int

coverage

Fraction of the cluster’s sampled speech during which the assigned face was on screen and scored as speaking. Drives the fusion “face vs voice”.

Type:

float

margin

Vote margin over the runner-up face in [0, 1] — the certainty signal.

Type:

float

crops

Best (frame_bgr, Face) samples of the assigned face, for embedding.

Type:

list[tuple[np.ndarray, Face]]

coverage: float
crops: list[tuple[ndarray, Face]]
face_id: int
margin: float
speaker: int
video_helper.faces.sampling.active_speaker_map(video_path, audio_16k, speaker_turns, *, asd_engine='auto', clip_len=3.0, asd_fps=12.0, census_period=1.0, clip_budget=24, per_round=6, margin_tau=0.35, coverage_floor=0.3, asd_tau=0.4, rescue_budget=None)[source]

Assign each diarization cluster to a global on-screen face via sampled ASD.

Parameters mirror the design knobs in .private/face.md §4/§7: clip_len and asd_fps bound per-window cost; clip_budget caps total heavy work; margin_tau/coverage_floor define per-cluster certainty; clusters below it pull more windows until certain or the budget is spent.

rescue_budget adds a last-chance pass: if the shared clip_budget runs out while some clusters are still uncertain, resume ASD on just those, drawing from their remaining candidate windows, until each is certain, out of windows, or clearly not improving (a no-progress guard drops a genuinely off-screen speaker rather than burning the machine on it). None (the default) lets the rescue run until the finite window pool or the no-progress guard stops it; an int caps the extra windows.

The iterative rounds and rescue pass above chase full certainty (margin and coverage), but the final cut is coverage-only: a cluster is omitted only if it never gathered enough sampled evidence (coverage < coverage_floor), logged and left for the caller’s voiceprint fallback. A cluster that gathered enough evidence but never reached the margin target is still returned, with its true margin reported on the SpeakerFaceAssignment (the field is meant as a caller-facing certainty signal, not an internal admission gate) so the caller can apply their own threshold instead of the evidence being silently discarded.

Parameters:
Return type:

list[SpeakerFaceAssignment]