best_engine_ai_helper.score module

score — select the best model for the current hardware.

Given the detected memory pool and the merged catalog, this module picks the highest-scoring model that fits within a safety headroom. The algorithm is intentionally simple: filter, then sort by benchmark score, then take the max.

Memory priority mirrors what Ollama uses at runtime:

unified_gb (Apple Silicon) > vram_gb (discrete GPU) > ram_gb * 0.5 (CPU)

The 0.5 factor for CPU RAM is conservative; a model loader competes with the OS, background daemons, and the inference server itself for RAM.

Author

Warith Harchaoui <warith.harchaoui@deraison.ai>

best_engine_ai_helper.score.effective_budget(hw, headroom=0.5, load=None)[source]

Compute the memory budget (GB) a model may occupy at run time.

The budget is the accelerator’s usable memory pool, scaled by an extra headroom margin left for the operating system, your own application, and KV-cache growth as context fills. On Apple Silicon the usable pool is not the whole unified memory: Metal caps GPU allocations at about 66% of the pool at or below 36 GB and about 75% above it, beyond which inference spills to CPU. Compare a catalog entry’s ram_gb (already a peak-inference estimate, weights plus a moderate KV cache) against this budget.

Parameters:

  • hw (dict[str, float | None]) – Output of detect.available_memory(). Expected keys: unified_gb, vram_gb, ram_gb.

  • headroom (float) – Extra safety fraction applied on top of the accelerator cap, reserving room for the OS, the caller’s workload, and KV growth. Defaults to MAX_HEADROOM (0.5) and is clamped down to it — a larger value is never honoured, to keep picks realistic.

  • load (dict or None) – Live server state from detect.server_load() (available_ram_gb, cpu_percent, disk_free_gb, …). When given, the theoretical accelerator budget is additionally capped at what is ACTUALLY free right now (another process, or an already-running engine, holds memory the static hardware totals in hw know nothing about), and further derated when the CPU is already saturated or the disk is nearly full. None (the default) reproduces the pre-existing, load-blind behaviour exactly.

Returns:

Effective memory budget in GB.

Return type:

float

Examples

>>> effective_budget({'unified_gb': 96.0, 'vram_gb': None, 'ram_gb': 96.0})
36.0
>>> effective_budget({'unified_gb': None, 'vram_gb': 24.0, 'ram_gb': 64.0})
11.04
>>> # headroom above the 0.5 ceiling is clamped, not honoured
>>> effective_budget({'unified_gb': 96.0, 'vram_gb': None, 'ram_gb': 96.0}, headroom=0.85)
36.0
>>> # a busy machine gets a smaller budget than an idle one with the same hardware
>>> hw = {'unified_gb': 96.0, 'vram_gb': None, 'ram_gb': 96.0}
>>> effective_budget(hw, load={'available_ram_gb': 10.0})
10.0
best_engine_ai_helper.score.estimated_tokens_per_second(entry, bandwidth_gbs, backend='ollama')[source]

Estimate local decode throughput (tokens/s) for a model on this machine.

Token generation is memory-bandwidth bound: each new token requires reading the model’s active weights from memory once, so the ceiling is bandwidth / model_bytes. The estimate derates that ceiling by a backend-specific decode efficiency (see _DECODE_EFFICIENCY_BY_BACKEND) to reflect KV-cache reads, kernel overhead, and sampling. It describes steady-state generation, not the compute-bound prefill of a long prompt.

Parameters:

  • entry (dict[str, Any]) – Catalog entry; its model_footprint_gb() is the active-model size.

  • bandwidth_gbs (float or None) – Memory bandwidth in GB/s from detect.compute_profile(). When None (unknown hardware, e.g. an unrecognised discrete-GPU model) the estimate is not computable and None is returned.

  • backend ({'ollama', 'vllm'}) – Serving backend. Affects both the size proxy (heavier FP16 weights under vLLM decode more slowly than the Q4 Ollama figure) AND the decode efficiency (vLLM’s PagedAttention + CUDA-graph decode tracks closer to the bandwidth ceiling than llama.cpp’s — confirmed to differ on identical Ubuntu + discrete-GPU hardware, not just a cross-platform artifact).

Returns:

Estimated tokens/s, rounded to one decimal, or None when bandwidth or model size is unavailable.

Return type:

float or None

best_engine_ai_helper.score.model_footprint_gb(entry, backend='ollama')[source]

Estimate a model’s peak inference memory (GB) on a given serving backend.

The catalog’s ram_gb is an Ollama Q4 estimate. vLLM instead loads the full FP16/BF16 HuggingFace weights (~2 bytes/param) plus KV cache and runtime buffers, so the same model is markedly heavier there. Sizing a vLLM pick against ram_gb would over-promise and pick a model that will not actually fit — the “unrealistic recommendation” this guards against.

Parameters:

  • entry (dict[str, Any]) – Catalog entry; uses ram_gb (Ollama) or size_b (vLLM).

  • backend ({'ollama', 'vllm'}) – Serving backend the footprint is estimated for. Unknown values are treated as 'ollama' (the conservative catalog figure).

Returns:

Estimated peak memory in GB.

Return type:

float

best_engine_ai_helper.score.rank(hw, catalog, kind, headroom=0.5, application=None, backend='ollama', load=None)[source]

Return all candidates sorted by benchmark score, annotated with fit status.

Each entry in the result gets a _fits key (bool) indicating whether it fits within the effective budget. The top entry is identical to what select() would return.

Parameters:

  • hw (dict[str, float | None]) – Output of detect.available_memory().

  • catalog (list[dict[str, Any]]) – Merged model entries from catalog.load_catalog().

  • kind ({'llm', 'vlm'}) – The inference kind to filter and rank by.

  • headroom (float) – Extra safety on top of the accelerator cap. Clamped to MAX_HEADROOM (0.5); default 0.5.

  • application (str or None) – Optional use-case keyword ("code", "math", "ocr", "vision", "chat", "generalist"). Drives which benchmark column is used for ranking.

  • backend ({'ollama', 'vllm'}) – Serving backend, so _fits reflects the footprint that actually loads (FP16 for vLLM, Q4 ram_gb for Ollama).

  • load (dict or None) – Live server state from detect.server_load(), forwarded to effective_budget(). None reproduces the load-blind behaviour.

Returns:

Candidates sorted descending by benchmark score, each with _fits.

Return type:

list[dict[str, Any]]

Examples

>>> catalog = [{'id': 'v', 'kind': 'vlm', 'ram_gb': 9.0,
...             'benchmarks': {'vision': 80}}]
>>> rank({'unified_gb': 96.0, 'vram_gb': None, 'ram_gb': 96.0}, catalog, 'vlm')[0]['_fits']
True
best_engine_ai_helper.score.select(hw, catalog, kind, headroom=0.5, application=None, backend='ollama', load=None)[source]

Pick the best-scoring model that fits in available memory.

Candidates for a ‘vlm’ selection include both VLMs and LLMs with vision capability (kind == ‘vlm’). Candidates for a ‘llm’ selection include text-only LLMs and VLMs (since a VLM handles text-only prompts equally).

Selection order: 1. Filter: keep entries whose ram_gb fits within the effective budget. 2. Rank: structured-output capability first (a model that cannot honour

Ollama structured output is never chosen over one that can), then benchmark score (application-specific if given, else vision for VLM or general for LLM). This matches rank(), so rank(...)[0] and select(...) agree.

  1. Last resort: if nothing fits, return the smallest model in the catalog rather than raising; the caller decides whether to warn the user.

Parameters:

  • hw (dict[str, float | None]) – Output of detect.available_memory().

  • catalog (list[dict[str, Any]]) – Merged model entries from catalog.load_catalog().

  • kind ({'llm', 'vlm'}) – The type of model to select.

  • headroom (float) – Extra safety on top of the accelerator cap. Clamped to MAX_HEADROOM (0.5); default 0.5.

  • application (str or None) – Optional use-case keyword ("code", "math", "ocr", "vision", "chat", "generalist"). Selects the benchmark axis used for scoring. None uses the default kind-based rule.

  • backend ({'ollama', 'vllm'}) – Serving backend, so memory fit is checked against the footprint that actually loads (FP16 for vLLM, Q4 ram_gb for Ollama).

  • load (dict or None) – Live server state from detect.server_load(), forwarded to effective_budget(). None reproduces the load-blind behaviour.

Returns:

The selected catalog entry.

Return type:

dict[str, Any]

Raises:

ValueError – If the catalog is empty.

Examples

>>> hw = {'unified_gb': 96.0, 'vram_gb': None, 'ram_gb': 96.0}
>>> catalog = [{'id': 'v', 'kind': 'vlm', 'ram_gb': 9.0,
...             'benchmarks': {'vision': 80}}]
>>> select(hw, catalog, kind='vlm')['id']
'v'