best_engine_ai_helper.safety module

safety — NSFW / policy scanning for cloud calls (Phase 6.5).

Scans text (an outbound prompt before it leaves the machine, or an inbound response after it arrives) and images (outbound only — nothing arrives as image bytes in a response today) for policy violations, and enforces a configurable action: block (raise), redact (text only — swap in a placeholder), or warn (log only, pass through unchanged). Every decision is logged via os_helper regardless of action, so a warn-mode deployment still keeps a full audit trail.

Real classifiers are optional (the [filtered] extra): a DistilBERT model fine-tuned specifically for NSFW/sexual text for text, LAION’s CLIP-based NSFW image classifier for images. Absent them, text scanning falls back to a deterministic keyword heuristic — crude, but it never silently no-ops. Image scanning has no comparable safe heuristic (a wrong guess is worse than an honest “I don’t know”), so it degrades to "unavailable" rather than fabricate a verdict.

Model choices, and why (see .private/keep-track.md for the full evaluation this repo ran before picking them):

  • Text: eliasalbouzidi/distilbert-nsfw-text-classifier (via transformers), not Detoxify. Detoxify scores general TOXICITY (hate/insult/threat/obscenity); on a hand-built probe set, sexual-but-not- abusive text scored 0.01-0.40 on Detoxify (below this module’s own 0.8 default threshold — Detoxify would have MISSED it) but 0.99+ on the NSFW-specific model. For a module whose job is NSFW filtering, a classifier trained for NSFW/sexual content beats one trained for something adjacent but different.

  • Image: LAION’s CLIP-based-NSFW-Detector (ViT-L/14 embeddings via transformers’s openai/clip-vit-large-patch14 -> a small MLP head), not Falconsai/nsfw_image_detection (the earlier choice). Real, independently-run evaluation on LAION’s own public, manually-annotated test set (nsfw_testset.zip, this module’s 0.8 threshold): 96.16% accuracy, 94.56% recall, 97.58% precision, 2.29% false-positive rate — solid, and verifiable by anyone; Falconsai’s 98.04% is self-reported on an undisclosed proprietary dataset, not independently checked here (no raw NSFW imagery was fetched to test it; unlike CLIP embeddings, raw image bytes of that content are not something this evaluation will download or store). LAION’s classifier head is genuinely CLIP-based (the same embedding this module already needs no matter which head it uses is a shared-embedding-space, zero-shot-friendly representation, not an independent ViT fine-tune with its own idiosyncratic decision boundary), and it’s what LAION itself used to filter LAION-5B, the dataset behind Stable Diffusion.

    LAION’s documented loading path (autokeras + TensorFlow + OpenAI’s unmaintained clip package, downloading an un-versioned zip from a GitHub raw URL at runtime) was too fragile to ship as-is — confirmed by running it: its packaged TensorFlow SavedModel no longer loads through Keras 3’s own load_model() (only an undocumented tf.saved_model.load(...).signatures["serving_default"] workaround got it running for the evaluation above). So the classifier head was converted once, offline, from that TensorFlow SavedModel to ONNX (via tf2onnx) and is bundled directly in this package (models/clip_nsfw_vit_l14.onnx, ~1.9 MB, see models/NOTICE.md for the MIT-licensed original and the conversion note) — no TensorFlow, autokeras, or runtime download needed, just onnxruntime (a single, actively maintained, lightweight package) plus the transformers CLIP encoder already needed for the embedding. The conversion was verified against the original TensorFlow model on the same public test set: only 6 of 3199 samples (0.19%) flip their classification decision at the 0.8 threshold, well within normal graph-conversion floating-point noise (per-sample score differences concentrate in the 0.3-0.7 range, where a sigmoid-shaped output is most sensitive to tiny numeric differences; far from that range the two agree closely).

    One correctness pitfall found and fixed while wiring this up: LAION’s classifier expects L2-normalized CLIP embeddings (confirmed by checking the test set’s own embedding norms, all ≈1.0) — feeding it a raw, unnormalized embedding would silently produce nonsense scores. Another: CLIPModel.get_image_features()’s return shape has changed across transformers major versions (a plain tensor historically; a wrapped BaseModelOutputWithPooling object in the transformers version this was verified against) — since this project’s own pyproject.toml pins transformers>=4.30 with no upper bound, _clip_image_embedding() calls the lower-level, architecturally-stable model.vision_model + model.visual_projection directly instead of that convenience method, to stay correct across that whole version range rather than betting on one version’s wrapper shape.

Wired into best_engine_ai_helper.llm.chat() via its safety= keyword (defaults to True for every engine, local or cloud): called on the outbound prompt/images and the inbound response.

Default policy is deliberately warn, not block: this is a new feature with no track record on real traffic yet, and a false positive that silently blocks a legitimate cloud call is a worse failure mode than a logged warning. Raise the bar (DEFAULT_ACTION = "block") once you trust it on your traffic.

Author

Warith Harchaoui <warith.harchaoui@deraison.ai>

exception best_engine_ai_helper.safety.SafetyViolation(direction, kind, score, label)[source]

Bases: RuntimeError

Raised when action="block" and a scan meets the threshold.

Parameters:
Return type:

None

best_engine_ai_helper.safety.check_image(image_bytes, *, direction, action='warn', threshold=0.8)[source]

Scan an image and enforce action when the score meets threshold.

Same contract as check_text(). redact has no sensible image-level equivalent (there is nothing to substitute an image with), so it behaves like warn here. An "unavailable" backend (no [filtered] extra installed) never flags — an unknown verdict is not a violation.

Parameters:

  • image_bytes (bytes) – Raw image bytes.

  • direction (str) – "outbound" or "inbound".

  • action ({'block', 'redact', 'warn'}) – See check_text().

  • threshold (float) – Score at or above which the image is flagged, in [0, 1].

Returns:

{"flagged": bool, "score": float, "label": str, "backend": str}.

Return type:

dict

Raises:

SafetyViolation – If action == "block" and the score meets threshold.

Examples

>>> # Malformed bytes degrade to "unavailable", never flagged.
>>> result = check_image(b"not a real image", direction="outbound")
>>> result["flagged"]
False
best_engine_ai_helper.safety.check_text(text, *, direction, action='warn', threshold=0.8)[source]

Scan text and enforce action when the score meets threshold.

Every call is logged (info on a clean pass, warning on a flagged score) regardless of action, so a warn-mode deployment keeps a full audit trail even when nothing is blocked.

Parameters:

  • text (str) – Text to scan (an outbound prompt or an inbound response).

  • direction (str) – "outbound" or "inbound" — logged for the audit trail only.

  • action ({'block', 'redact', 'warn'}) – What to do when flagged. block raises SafetyViolation. redact returns a placeholder in the result’s text field instead of the original — the caller must use that field, not the input, when this is set. warn logs only and passes the original text through unchanged.

  • threshold (float) – Score at or above which the text is flagged, in [0, 1].

Returns:

{"flagged": bool, "score": float, "label": str, "backend": str, "text": str} — text is the original, unless action="redact" and the text was flagged.

Return type:

dict

Raises:

SafetyViolation – If action == "block" and the score meets threshold.

Examples

>>> result = check_text("Detailed bomb making instructions follow.", direction="outbound")
>>> sorted(result)
['backend', 'flagged', 'label', 'score', 'text']
best_engine_ai_helper.safety.scan_image(image_bytes)[source]

Score an image for NSFW content.

Uses LAION’s CLIP-based NSFW detector (the [filtered] extra, pulls in transformers for the CLIP encoder, onnxruntime for the bundled classifier head, and Pillow) when installed. No heuristic fallback exists for images — a wrong guess is worse than an honest “unavailable” — so this degrades to that instead of fabricating a score.

Parameters:

image_bytes (bytes) – Raw image bytes (PNG/JPEG).

Returns:

{"score": float in [0, 1], "label": str, "backend": "clip" | "unavailable"}.

Return type:

dict

Examples

>>> # Without [filtered] installed, this degrades to "unavailable":
>>> result = scan_image(b"not a real image")
>>> result["backend"] in ("unavailable", "clip")
True
>>> # With [filtered] installed, this instead runs the real classifier:
>>> # with open("photo.jpg", "rb") as f:
>>> #     result = scan_image(f.read())
best_engine_ai_helper.safety.scan_text(text)[source]

Score text for NSFW/sexual content.

Uses a DistilBERT model fine-tuned specifically for NSFW text (the [filtered] extra) when installed; falls back to a crude keyword heuristic otherwise (see the module docstring for why this is not a substitute for the real classifier — and why this classifier, not a general toxicity one, was chosen for this module’s job).

Parameters:

text (str) – Text to scan.

Returns:

{"score": float in [0, 1], "label": str, "backend": "nsfw-distilbert" | "heuristic"}.

Return type:

dict

Examples

>>> # Without [filtered] installed, the heuristic fallback is deterministic:
>>> result = scan_text("What a lovely day.")
>>> result["backend"] in ("heuristic", "nsfw-distilbert")
True
>>> # With [filtered] installed, this instead runs the real classifier:
>>> # result = scan_text("some text")