video_helper.faces.asd module

video_helper.faces.asd

Active-Speaker Detection (ASD): given the on-screen faces during a window and the concurrent audio, score which face is producing the speech.

Two interchangeable engines behind one interface:

  • LipMotionASD — a zero-weight proxy. Scores the temporal variance of mouth openness per track, gated by audio activity in the window. Always available, fully offline, no download. The graceful-degradation default.

  • LightASD — the accurate audio-visual cross-attention model (Light-ASD) run through ONNX Runtime. Downloaded on first use from the user’s mirror; if the weights are not hosted yet, available() is False and the caller falls back to the proxy.

The engine is called per clip window (not per whole video): the smart-sampling harness (video_helper.faces.sampling) picks a small set of windows so this heavy step runs on a fraction of the footage.

class video_helper.faces.asd.ASDEngine[source]

Bases: object

Interface: score each track’s speaking likelihood within one clip window.

available()[source]
Return type:

bool

name = 'base'
score_tracks(frames, tracks, audio_16k, fps)[source]
Parameters:
Return type:

dict[int, float]

class video_helper.faces.asd.LightASD[source]

Bases: ASDEngine

Light-ASD (Junhua-Liao et al., CVPR 2023) via PyTorch — accurate engine.

Loads the pretrained weights (light_asd.pth, research license) into the vendored _lightasd model and scores each face track with the model’s own audio-visual head. No ONNX: a faithful ONNX export of the MFCC front-end is fragile, so we run the real network. Degrades to unavailable (the harness swaps in the lip-motion proxy) if torch or the weights are missing.

Preprocessing matches the original exactly: audio → 13-cepstrum MFCC with fps-adjusted windows; visual → 112x112 grayscale mouth crops fed RAW (the model normalises (x/255 - 0.4161)/0.1688 internally — feeding an already /255 array would double-divide). Audio runs at ~4x the visual frame rate, so we align to 4 * T_visual MFCC frames; the two front-ends reduce to a common length, and we min-clip to be safe.

available()[source]
Return type:

bool

name = 'light-asd'
score_tracks(frames, tracks, audio_16k, fps)[source]
Parameters:
Return type:

dict[int, float]

class video_helper.faces.asd.LipMotionASD[source]

Bases: ASDEngine

Weights-free proxy: lip-motion variance × audio activity.

available()[source]
Return type:

bool

name = 'lip-motion'
score_tracks(frames, tracks, audio_16k, fps)[source]
Parameters:
Return type:

dict[int, float]

video_helper.faces.asd.get_engine(name='auto')[source]

Return an ASD engine by name, degrading to the proxy when needed.

"auto" prefers Light-ASD when its weights are hosted, else the proxy. "light-asd" forces the accurate engine (still falls back if unavailable). "lip-motion" forces the proxy.

Parameters:

name (str)

Return type:

ASDEngine