video_helper.faces.digest module

video_helper.faces.digest

Build a short, dense digest video — a handful of small clips lifted from a long recording and concatenated end-to-end — so a heavy per-frame model (Active Speaker Detection, or anything else that wants “the interesting moments” rather than the whole file) runs once on a compact file instead of many separate seeks into the original.

Two problems this solves at once:

Cost. ASD-class models are expensive per frame. Most of a long recording is redundant for the purpose of anchoring a diarization cluster to a face: once a speaker/face pairing is well-sampled, more footage of the same pairing adds little. A digest built from anchor moments (see below) concentrates the heavy model’s attention on the frames that actually matter.
Seek robustness. Many small independent seeks into one large source file are the exact pattern that has triggered real decoder instability on long recordings (multithreaded PyAV decode contexts closed mid-stream; see video_helper.main.extract_frames()’s _extract_via_pyav notes). Building the digest is still several seeks into the original, but every later stage (ASD, face tracking, re-scoring, human review) reads only the small, uniformly-encoded digest file — one open, no further seeking into the fragile long original.

Anchor-driven window selection

The caller supplies anchor_times: timestamps (seconds) where something diarization-worthy happens — typically the union of raw (unnamed) diarization speaker-change instants and shot-change instants, merged and sorted by the caller. This module is deliberately agnostic about where anchors come from: it knows nothing about diarization or scene detection, only about timestamps.

From the anchors, two complementary window families are built around each fused anchor F[i] (anchors closer than merge_gap seconds collapse to their midpoint first):

Boundary windows [F[i] - window, F[i] + window] — centred on the anchor, where a speaker or shot change is happening and the speaker/face identity most needs (re-)confirming.
Mid-segment windows [mid - window, mid + window], where mid is the midpoint between two consecutive fused anchors (and, at the two ends of the timeline, between the start/end of the video and the nearest anchor) — a calm, representative sample of an already-established segment, away from any cut.

Overlapping windows are merged (min start, max end) before extraction, so dense anchor clusters do not produce redundant, near-duplicate clips.

The digest manifest

build_asd_digest() returns (and optionally writes to disk) a list of DigestSegment, one per clip actually placed in the digest, each carrying both its position in the digest (digest_start/digest_end) and its corresponding position in the original source video (source_start/source_end). Any consumer that runs a per-frame model on the digest and gets a result at digest-time t maps it back to the original recording’s timeline via the manifest — this is the only way the two timelines are ever reconciled, so treat it as required bookkeeping, not an optional extra.

Splice-boundary caveat for consumers

The digest is a concatenation of clips from different, non-contiguous parts of the source video. Anything that tracks continuity frame-to-frame (face tracking by IoU, an ASD model’s temporal context) must be reset at every segment boundary in the manifest — a face that happens to land in a similar position right after a splice is coincidence, not continuity. This module does not run any such tracking itself, so it cannot enforce the reset; it only guarantees the manifest gives every splice point precisely, in digest_start order.

Engine independence

Nothing here is specific to any Active Speaker Detection engine (Light-ASD, LR-ASD, a future replacement, or the zero-weight lip-motion proxy in video_helper.faces.asd). This module only builds a video file and a timestamp mapping; whichever ASDEngine a caller later runs against the digest is an orthogonal choice.

class video_helper.faces.digest.DigestSegment(digest_start, digest_end, source_start, source_end)[source]

Bases: object

One clip placed in the digest, with its two timelines reconciled.

Parameters:

digest_start (float)
digest_end (float)
source_start (float)
source_end (float)

digest_start

Start time (seconds) of this clip within the digest video.

Type:: float

digest_end

End time (seconds) of this clip within the digest video.

Type:: float

source_start

Start time (seconds) of this clip within the original source video.

Type:: float

source_end

End time (seconds) of this clip within the original source video.

Type:: float

digest_end: float

digest_start: float

source_end: float

source_start: float

to_source_time(digest_time)[source]

Map a timestamp inside this segment’s digest span back to source time.

Parameters:: digest_time (float) – A timestamp (seconds), in the digest’s own timeline.
Returns:: The corresponding source-video timestamp, or None when digest_time does not fall inside this segment.
Return type:: float or None

Examples

>>> seg = DigestSegment(digest_start=0.0, digest_end=12.0,
...                     source_start=340.0, source_end=352.0)
>>> seg.to_source_time(3.0)
343.0
>>> seg.to_source_time(99.0) is None
True

video_helper.faces.digest.build_asd_digest(video_path, anchor_times, output_video, *, window=6.0, merge_gap=12.0)[source]

Build a compact digest video from anchor-driven windows of video_path.

See the module docstring for the full design (why a digest, how windows are chosen, the splice-boundary caveat for consumers). In short: fuse nearby anchors to their midpoint, form a boundary window around each fused anchor and a mid-segment window between consecutive anchors (and at the two ends of the timeline), merge overlapping windows, then build the digest in three phases:

Transcode once — to_editing_intermediate() turns the source into an all-keyframe (GOP 1), PCM-audio intermediate. Every timestamp in it is a valid, lossless, frame-accurate cut point.
Cut + concat on the intermediate — each window is lifted with extract_video_chunk() (copy=True, pure stream-copy, no re-encode) and the clips are stitched with concat_videos() (reencode=False), safe because every chunk shares the intermediate’s exact codec/timebase.
Final encode — the stitched intermediate (huge, all-keyframe/PCM) is re-encoded once via video_converter() down to a normal delivery-sized file, audio included.

Doing the heavy per-window work as stream-copy and re-encoding only once, at the end, keeps the digest audio-safe and avoids paying a re-encode cost per window (see concat_videos() for why re-encoding a concat directly, reencode=True, used to silently drop audio).

Parameters:

video_path (str) – Path to the source video.
anchor_times (list of float) – Timestamps (seconds) where something diarization-worthy happens — the caller’s merged, sorted union of e.g. raw diarization speaker-change instants and shot-change instants. This function does not know or care where they came from.
output_video (str) – Path to write the concatenated digest video (with audio).
window (float, optional) – Half-width in seconds of every extracted window (default 6.0 — a full window is then 2 * window = 12s, comfortably covering the 1-6s duration range LR-ASD’s own evaluation methodology uses for reliability).
merge_gap (float, optional) – Anchors closer than this (seconds) fuse to one midpoint before windows are formed (default 12.0).

Returns:

One entry per clip actually placed in the digest, in digest-time order. Also written alongside output_video as <output_video>.manifest.json (a JSON array of {digest_start, digest_end, source_start, source_end}).

Return type:

list of DigestSegment

Raises:

AssertionError – If video_path is not a valid video, or no windows could be formed.

Notes

Splice boundaries (every digest_start/digest_end pair in the returned manifest) are real discontinuities: any frame-to-frame continuity assumption (face tracking, an ASD model’s own temporal context) must be reset there by the caller. This function only builds the video and the mapping; it runs no tracking or detection itself, so it has no opinion on which engine consumes the digest — Light-ASD, LR-ASD, or anything else conforming to video_helper.faces.asd.ASDEngine.

Examples

>>> segs = build_asd_digest(
...     "meeting.mp4", anchor_times=[12.0, 340.0, 341.0, 900.0],
...     output_video="digest.mp4",
... )
>>> segs[0].source_start
0.0