best_engine_ai_helper.validate_vlm module

validate_vlm — Ralph Eyeball Loop gate for VLM validation.

Validates that the selected vision-language model (VLM) can correctly identify concrete visual defects in a reference fixture. The test is intentionally simple and fast: a small PNG with an obvious problem is enough to distinguish a working VLM from one that is broken, quantized below threshold, or not yet warmed up.

The fixture contains two seeded defects: 1. A bar with near-zero contrast against the background (accessibility fail). 2. A truncated x-axis label (layout fail).

A VLM that misses both defects fails the gate. A VLM that identifies at least one is considered functional for the sprezzature visual-critique workflow.

Author

Warith Harchaoui <warith.harchaoui@deraison.ai>

best_engine_ai_helper.validate_vlm.validate(llm_chat)[source]

Run the VLM gate against the reference fixture.

Uses the fixture from _make_fixture_png(). Sends the PNG to the VLM with the critique prompt, then asks a text call to produce a pass/fail verdict. The VLM passes if the verdict dict has "pass": true.

Parameters:: llm_chat (callable) – The chat function from llm.py, or a compatible mock. This is injected so tests can patch it without touching global state.
Returns:: True if the VLM identified at least one seeded defect; False otherwise.
Return type:: bool

Examples

>>> def mock_chat(p, **kw):
...     if kw.get("images"):
...         return "I see a low-contrast bar and a clipped label."
...     return {"pass": True, "reason": "Critique mentions contrast issue."}
>>> validate(mock_chat)
True