best_engine_ai_helper.cloud_catalog module

cloud_catalog — pick the best PAID model for a task, the cloud counterpart of best_engine_ai_helper.score.

Local mode (score, recommend) auto-picks the best model from a catalog by weighing benchmark quality against memory fit — free, since the hardware is already paid for. Cloud mode auto-picks the best model from a different catalog by weighing benchmark quality against price: every candidate fits on any machine (the provider owns the hardware), so cost is the resource being budgeted instead of memory.

The catalog is pricing.yaml at the package root: the same table best_engine_ai_helper.observe already reads to price a completed call, extended with provider, kind (llm/vlm), structured_output, and benchmarks per entry. One file, two consumers — pricing after the call, ranking before it — so cost and quality can never drift apart into two different numbers for the same model.

Author

Warith Harchaoui <warith.harchaoui@deraison.ai>

best_engine_ai_helper.cloud_catalog.blended_price_per_1m(entry)[source]

A single $/1M-token comparability figure for one catalog entry.

A real workload’s input/output ratio varies by task (a long-document summary is input-heavy; an open-ended generation is output-heavy), so this is a simple average of the two list prices, not a workload-specific estimate — the same “rough estimate, not a bill” honesty pricing.yaml already documents for the ledger side.

Parameters:

entry (dict[str, Any]) – A catalog entry with input_per_1m/output_per_1m (USD).

Returns:

(input_per_1m + output_per_1m) / 2. 0.0 when both are absent.

Return type:

float

Examples

>>> blended_price_per_1m({"input_per_1m": 2.0, "output_per_1m": 10.0})
6.0
best_engine_ai_helper.cloud_catalog.load_cloud_catalog(path=None)[source]

Load the bundled paid-model catalog (pricing.yaml).

Parameters:

path (Path or None) – Override the catalog file (tests use a fixture). Defaults to the bundled pricing.yaml next to pyproject.toml.

Returns:

One dict per model, each carrying its YAML key as id plus provider, kind, structured_output, benchmarks, input_per_1m, output_per_1m.

Return type:

list[dict[str, Any]]

Raises:

FileNotFoundError – If path is given explicitly and does not exist.

Examples

>>> entries = load_cloud_catalog()
>>> all({"id", "provider", "kind"} <= e.keys() for e in entries)
True
best_engine_ai_helper.cloud_catalog.pick_cloud(catalog, kind, application=None, quality_vs_cost=0.7, provider=None)[source]

Return the single best paid candidate, or None if nothing matches.

Thin wrapper over rank_cloud() — rank_cloud(...)[0], or None on an empty ranking, so callers do not need an IndexError guard.

Parameters:
Returns:

The top-ranked entry, or None when no candidate matches kind (and provider, if given).

Return type:

dict[str, Any] or None

Examples

>>> catalog = [{"id": "m", "provider": "p", "kind": "llm",
...             "structured_output": True, "benchmarks": {"general": 80},
...             "input_per_1m": 1.0, "output_per_1m": 2.0}]
>>> pick_cloud(catalog, "llm")["id"]
'm'
>>> pick_cloud(catalog, "vlm") is None
True
best_engine_ai_helper.cloud_catalog.rank_cloud(catalog, kind, application=None, quality_vs_cost=0.7, provider=None)[source]

Rank paid catalog candidates by a quality/price trade-off.

Mirrors score.rank()’s shape (structured-output capability first, then a combined score), so the local and cloud pickers read the same way. Each candidate gets score (raw benchmark), price_per_1m (blended_price_per_1m()), and combined — a 0..1 blend of min-max-normalized quality and (inverted) price across the candidate pool.

Parameters:

  • catalog (list[dict[str, Any]]) – Entries from load_cloud_catalog().

  • kind ({'llm', 'vlm'}) – vlm keeps only vision-capable entries; llm keeps both llm and vlm entries (a VLM answers text-only prompts too — same rule as score.rank()).

  • application (str or None) – Benchmark axis keyword ("code", "math", "ocr", "vision", "chat", "generalist"), forwarded to score._benchmark_score(). None uses the default kind-based rule (vision axis for a VLM, general for an LLM).

  • quality_vs_cost (float) – 0..1 weight on quality vs. price; clamped into range. 1.0 picks purely on benchmark score (price ignored); 0.0 picks purely on price (quality ignored, subject to the structured-output gate). Defaults to DEFAULT_QUALITY_VS_COST.

  • provider (str or None) – Restrict candidates to one provider (e.g. "openai"), case- insensitive. None ranks across every provider in the catalog.

Returns:

Candidates sorted best-first. Empty when nothing matches kind (and provider, if given).

Return type:

list[dict[str, Any]]

Examples

>>> catalog = [
...     {"id": "cheap", "provider": "p", "kind": "llm", "structured_output": True,
...      "benchmarks": {"general": 70}, "input_per_1m": 0.1, "output_per_1m": 0.1},
...     {"id": "strong", "provider": "p", "kind": "llm", "structured_output": True,
...      "benchmarks": {"general": 90}, "input_per_1m": 5.0, "output_per_1m": 15.0},
... ]
>>> rank_cloud(catalog, "llm", quality_vs_cost=1.0)[0]["id"]  # quality only
'strong'
>>> rank_cloud(catalog, "llm", quality_vs_cost=0.0)[0]["id"]  # price only
'cheap'