best_engine_ai_helper.llm module

llm — pluggable local AND cloud model backend for best-engine-ai-helper.

Provides two public functions, chat and embed, that route requests to the backend named by a resolved engine descriptor (preferred) or the SPREZZATURE_LLM_BACKEND environment variable (legacy path). Callers use only these two functions; the transport details (Ollama JSON API vs OpenAI-compatible REST vs Anthropic/Gemini’s own formats vs LangChain) stay invisible to them.

Supported backends

ollama

Default. POSTs to {SPREZZATURE_LLM_BASE_URL}/api/generate. Works offline once the model is pulled.

openai

Any OpenAI-compatible server: vLLM, llama.cpp, LM Studio, Text Generation Inference, OpenAI itself, Mistral, OpenRouter, Together, Azure OpenAI. POSTs to {base_url}/v1/chat/completions.

anthropic

Claude’s own Messages API (POST {base_url}/v1/messages).

gemini

Google’s own generateContent API (POST {base_url}/v1beta/models/{model}:generateContent).

langchain

Thin wrapper over ChatOllama or ChatOpenAI from LangChain. Only useful if you need LangChain retrievers or agent abstractions.

A cloud engine descriptor (engine.resolve with mode: cloud) carries an embedded local fallback; chat() tries the cloud primary first and falls over to the local fallback on failure (paid -> local, the safe direction) — see engine= below and best_engine_ai_helper.engine.

Environment variables

SPREZZATURE_LLM_BACKEND

ollama | openai | anthropic | gemini | langchain. Defaults to ollama. Only consulted on the legacy no-engine path.

SPREZZATURE_LLM_BASE_URL

Base URL of the server. Defaults to http://localhost:11434.

BEST_LLM_TEXT (legacy alias: SPREZZATURE_LLM_TEXT)

Model tag for text-only prompts. When unset, falls back to the selection persisted by pull in ~/.best-engine-ai-helper/config.json, then to the qwen3:8b default. Resolved by config.text_model().

BEST_LLM_VISION (legacy alias: SPREZZATURE_LLM_VISION)

Model tag for prompts that include images. Same precedence as the text model; resolved by config.vision_model().

SPREZZATURE_LLM_API_KEY

API key for servers that require one on the legacy path. Empty string by default (most local servers do not require authentication). On the engine= path, the key comes from the env var NAMED in the engine’s api_key_env field instead (never the key value itself, never persisted) — see _cloud_api_key().

Observability

Every chat() call fans a small event dict out to any observer registered via add_observer() (backend, model, kind, char counts, real token counts when the provider reports them, latency, success/error). No observer is registered by default. best_engine_ai_helper.observe provides a SQLite-backed sink (call observe.enable()) that turns this into a queryable local activity/cost ledger, surfaced by the activity CLI command and the /api/activity endpoint.

Privacy and safety

chat(..., pseudonymize=True) scrubs personal data from the prompt with a local LLM before it reaches a cloud provider, and restores it in the response — see best_engine_ai_helper.privacy. Cloud-only (a no-op on a local engine — there is nowhere for personal data to leak to).

chat(..., safety=...) scans the prompt/images before sending and the response after receiving for NSFW/policy violations — see best_engine_ai_helper.safety. On by default for EVERY engine, local or cloud (content policy is independent of who is billed); pass safety=False to opt out.

Author

Warith Harchaoui <warith.harchaoui@deraison.ai>

best_engine_ai_helper.llm.add_observer(fn)[source]

Register a per-call observer; it receives the event dict chat() emits.

Parameters:

fn (callable) – Called with one event dict after every chat() call, success or failure. Must not raise — an exception is caught and logged, never propagated, so a broken observer can’t take down inference.

Return type:

None

best_engine_ai_helper.llm.chat(prompt, *, system=None, images=None, json_schema=None, model=None, temperature=0.2, engine=None, kind=None, cache=False, retries=0, pseudonymize=False, safety=None)[source]

Send a prompt to the configured model (local or cloud) and return the response.

The backend and model come from one of two sources. When engine is given (the suite’s preferred path), it is read from a resolved engine descriptor — the gitignored llm.engine.yaml a repo gets from best-engine-ai-helper resolve. A cloud engine’s embedded local fallback is tried after the cloud primary on failure (paid -> local); pass a list of descriptors to define your own failover order instead. Otherwise the legacy env path applies: the backend is SPREZZATURE_LLM_BACKEND and the model resolves via env / persisted config.

Parameters:

  • prompt (str) – User-facing prompt text.

  • system (str or None) – System-level instructions sent before the user prompt. Use for persona, output format constraints, or house style rules.

  • images (list[bytes] or None) – Raw image bytes (PNG or JPEG). When provided, the vision model is used unless model is specified explicitly.

  • json_schema (dict or None) – When provided, the response is constrained to this JSON Schema where the backend supports grammar-constrained output (Ollama, Gemini); other backends (OpenAI-compatible via response_format, Anthropic via a prompt instruction) parse best-effort, same as every backend’s fallback when the model still returns non-JSON.

  • model (str or None) – Override the model tag/id. Wins over both the engine descriptor and the env default. When absent with no engine, defaults to the vision model when images are present, else the text model.

  • temperature (float) – Sampling temperature. Lower values are more deterministic. Defaults to 0.2 because structured extraction tasks benefit from low variance.

  • engine (dict | str | list | None) – A resolved engine descriptor (dict from engine.resolve() / engine.ensure(), or a path to llm.engine.yaml), or an explicit list of descriptors to try in order. When given, its backend / base_url and the per-kind model drive the request.

  • kind ({'llm', 'vlm'} or None) – Which model to use from engine. Defaults to vlm when images are present, else llm. Ignored when engine is None.

  • cache (bool) – Memoize identical calls (same backend/model/prompt/schema/images) via wallet-helper (the [cloud] extra) so a repeated call never pays for the same cloud request twice. A no-op with a warning if the extra is not installed. Ignored on the legacy env path (no engine to key on).

  • retries (int) – Retry a transient transport failure this many times (exponential backoff via tenacity when installed, immediate retry otherwise) before moving to the next engine in the failover chain.

  • pseudonymize (bool) – Scrub personal data from prompt with a local LLM before it reaches a cloud engine, and restore it in the response — see best_engine_ai_helper.privacy. No-op on a local engine, or when the cloud engine has no local fallback to do the scrubbing with (warns in that case rather than silently skipping).

  • safety (bool or None) – Scan the prompt/images before sending and the response after receiving for policy violations — see best_engine_ai_helper.safety. None (the default) resolves to True for every engine, local or cloud: NSFW/policy content is a content-policy concern independent of who is billed, not a cloud-only risk. Pass False to opt out explicitly.

Returns:

When json_schema is provided and the model returns valid JSON, the result is parsed and returned as a dict. Otherwise a plain string.

Return type:

str or dict

Raises:

  • RuntimeError – If every engine in the failover chain fails, or (with no engine) the single legacy-path backend is unreachable or returns a malformed response.

  • ValueError – If a backend name (env path) or transport (engine path) is unrecognised.

Examples

>>> # Text prompt (no model running needed for this docstring to parse)
>>> # result = chat("Summarise this paper in one sentence.")
>>> # Vision prompt
>>> # with open("chart.png", "rb") as f:
>>> #     result = chat("Describe the chart.", images=[f.read()])
best_engine_ai_helper.llm.clear_observers()[source]

Remove all registered observers (chiefly for tests, or to disable).

Return type:

None

best_engine_ai_helper.llm.embed(text)[source]

Return an embedding vector for the given text.

Only the Ollama backend is supported for embeddings. The OpenAI-compatible embedding endpoint (/v1/embeddings) is not yet implemented because the retrieval use case is not yet in scope.

Parameters:

text (str) – Input text to embed.

Returns:

Dense embedding vector from the Ollama /api/embeddings endpoint.

Return type:

list[float]

Raises:

Examples

>>> # vec = embed("hello world")  # requires Ollama running
>>> # len(vec) > 0
>>> True
True