best_engine_ai_helper package

Subpackages

Submodules

Module contents

best_engine_ai_helper — pick the best local LLM/VLM for the current hardware.

Public API (importable without invoking the CLI):

from best_engine_ai_helper.detect import platform_name, chip_vendor, available_memory from best_engine_ai_helper.catalog import load_catalog, estimate_ram from best_engine_ai_helper.hardware import load_hardware, lookup_chip from best_engine_ai_helper.score import select, rank, effective_budget from best_engine_ai_helper import text_model, vision_model # cheap tag resolvers

The CLI entry point is best-engine-ai-helper (see cli.py). Importing this package does not trigger any CLI parsing or subprocess calls.

Downstream suite packages that just need “which model tag do I use?” should call text_model() / vision_model() — they resolve an env override, then the selection persisted by pull, then a safe default, without ever probing hardware, so they are cheap and deterministic (CI-safe).

Author

Warith Harchaoui <warith.harchaoui@deraison.ai>

best_engine_ai_helper.available_memory()[source]

Detect available memory for model inference.

Probes in priority order: 1. Apple Silicon unified memory (macOS with Apple chip) 2. NVIDIA VRAM via nvidia-smi 3. AMD VRAM via rocm-smi 4. System RAM via psutil (always populated)

Returns:

A dict with the following keys:

unified_gbfloat or None

Apple Silicon unified memory pool, in GB.

vram_gbfloat or None

Discrete GPU VRAM (sum of all visible GPUs), in GB.

ram_gbfloat

Total system RAM in GB. Always a positive float.

Return type:

dict[str, float | None]

Examples

>>> mem = available_memory()
>>> set(mem.keys()) == {'unified_gb', 'vram_gb', 'ram_gb'}
True
>>> mem['ram_gb'] > 0
True
best_engine_ai_helper.chip_name()[source]

Return the Apple Silicon chip name (e.g. "Apple M2 Max"), or None.

Read from system_profiler on macOS; None on other platforms or when the chip line is absent.

Return type:

str | None

best_engine_ai_helper.chip_vendor()[source]

Identify the primary compute vendor for model inference.

Checks in order: Apple Silicon, NVIDIA (nvidia-smi), AMD (rocm-smi), then falls back to ‘cpu’.

Returns:

One of: ‘apple’, ‘nvidia’, ‘amd’, ‘intel’, ‘cpu’.

Return type:

str

Examples

>>> chip_vendor() in ('apple', 'nvidia', 'amd', 'intel', 'cpu')
True
best_engine_ai_helper.compute_profile()[source]

Describe the machine’s inference accelerator and memory bandwidth.

Returns a dict with:

  • accelerator: "gpu-metal" (Apple Silicon), "gpu-cuda" (NVIDIA), "gpu-rocm" (AMD), or "cpu" (no discrete accelerator detected).

  • chip: the chip / GPU name when known, else None.

  • bandwidth_gbs: memory bandwidth in GB/s when known, else None. This is the ceiling on decode throughput; token generation reads the whole active model from memory once per token, so tokens/s scales with it.

Bandwidth is only tabulated for Apple Silicon here (published specs); discrete-GPU bandwidth is left None because VRAM size, not bandwidth, is the binding constraint the catalog already models, and the figure varies by exact board. Callers treat a None bandwidth as “throughput not estimated”.

Return type:

dict[str, Any]

best_engine_ai_helper.effective_budget(hw, headroom=0.85)[source]

Compute the memory budget (GB) a model may occupy at run time.

The budget is the accelerator’s usable memory pool, scaled by an extra headroom margin left for the operating system, your own application, and KV-cache growth as context fills. On Apple Silicon the usable pool is not the whole unified memory: Metal caps GPU allocations at about 66% of the pool at or below 36 GB and about 75% above it, beyond which inference spills to CPU. Compare a catalog entry’s ram_gb (already a peak-inference estimate, weights plus a moderate KV cache) against this budget.

Parameters:

  • hw (dict[str, float | None]) – Output of detect.available_memory(). Expected keys: unified_gb, vram_gb, ram_gb.

  • headroom (float) – Extra safety fraction applied on top of the accelerator cap, reserving room for the OS, the caller’s workload, and KV growth. Default 0.85.

Returns:

Effective memory budget in GB.

Return type:

float

Examples

>>> effective_budget({'unified_gb': 96.0, 'vram_gb': None, 'ram_gb': 96.0})
61.2
>>> effective_budget({'unified_gb': None, 'vram_gb': 24.0, 'ram_gb': 64.0})
18.768
best_engine_ai_helper.estimate_ram(disk_gb, quant)[source]

Estimate peak inference RAM from on-disk model size.

The estimate covers the model weights plus KV cache at the default context length (4K tokens). For models with very large context windows (256K+), actual RAM may exceed this estimate significantly; treat it as a lower bound.

Parameters:

  • disk_gb (float) – On-disk footprint of the model in GB.

  • quant (str) – Quantization identifier, e.g. ‘Q4_K_M’, ‘Q8_0’, ‘FP16’, ‘Q2_K’.

Returns:

Estimated peak RAM in GB.

Return type:

float

Examples

>>> estimate_ram(6.1, 'Q4_K_M')
6.832
>>> estimate_ram(10.0, 'FP16')
10.5
best_engine_ai_helper.estimated_tokens_per_second(entry, bandwidth_gbs)[source]

Estimate local decode throughput (tokens/s) for a model on this machine.

Token generation is memory-bandwidth bound: each new token requires reading the model’s active weights from memory once, so the ceiling is bandwidth / model_bytes. The estimate derates that ceiling by _DECODE_EFFICIENCY to reflect KV-cache reads, kernel overhead, and sampling. It describes steady-state generation, not the compute-bound prefill of a long prompt.

Parameters:

  • entry (dict[str, Any]) – Catalog entry; uses ram_gb as the active-model size proxy.

  • bandwidth_gbs (float or None) – Memory bandwidth in GB/s from detect.compute_profile(). When None (unknown hardware) the estimate is not computable and None is returned.

Returns:

Estimated tokens/s, rounded to one decimal, or None when bandwidth or model size is unavailable.

Return type:

float or None

best_engine_ai_helper.load_catalog(catalog_path=None)[source]

Load the bundled seed catalog merged with the user’s local cache.

Cache entries whose id matches a seed entry overwrite the seed entry. New cache entries (no matching seed id) are appended. The seed is never modified on disk.

Parameters:

catalog_path (Path or None) – Path to the seed models.yaml. Defaults to the bundled file next to pyproject.toml. Pass an explicit path in tests to use a fixture.

Returns:

Merged model entries. Each entry is guaranteed to have at minimum: id, kind, ram_gb, benchmarks.

Return type:

list[dict[str, Any]]

Raises:

FileNotFoundError – If catalog_path is given explicitly and does not exist.

Examples

>>> entries = load_catalog()
>>> len(entries) > 0
True
>>> all('id' in e for e in entries)
True
best_engine_ai_helper.load_config()[source]

Return the persisted selection dict, or an empty dict if absent/unreadable.

Reads ~/.best-engine-ai-helper/config.json (written by pull.write_env()). A missing file is the normal “never ran detection” case, so it maps to {} rather than an error; a corrupt file is treated the same way so a bad write can never break a downstream caller.

Returns:

The parsed config, or {} when the file does not exist or cannot be parsed as a JSON object.

Return type:

dict

best_engine_ai_helper.load_hardware(hardware_path=None)[source]

Load the bundled hardware chip table merged with the user’s local cache.

Cache entries whose (chip, memory_gb) pair matches a seed entry overwrite it. New entries are appended. The seed file is never modified.

Parameters:

hardware_path (Path or None) – Path to the seed hardware.yaml. Defaults to the bundled file. Pass an explicit path in tests to use a fixture.

Returns:

Merged hardware entries. Each entry has at minimum: chip, vendor, memory_gb, ollama_usable_gb.

Return type:

list[dict[str, Any]]

Examples

>>> entries = load_hardware()
>>> len(entries) > 0
True
>>> all('chip' in e for e in entries)
True
best_engine_ai_helper.lookup_chip(chip_name, hardware)[source]

Find a hardware entry by a case-insensitive substring match on the chip name.

When multiple entries share the same chip name (for example, an Apple M2 Max at 32 GB and at 96 GB), this returns the first match in the list order. The caller should supply the most specific chip string available to avoid ambiguity.

Parameters:

  • chip_name (str) – Chip name or substring to search for, e.g. 'Apple M2 Max'.

  • hardware (list[dict[str, Any]]) – Hardware entries as returned by load_hardware().

Returns:

The first matching entry, or None if no entry contains chip_name as a case-insensitive substring.

Return type:

dict[str, Any] or None

Examples

>>> hw = load_hardware()
>>> entry = lookup_chip('Apple M2 Max', hw)
>>> entry is not None
True
>>> entry['vendor']
'apple'
best_engine_ai_helper.parse_task(task)[source]

Turn a vague task phrase into the model kinds and benchmark axis it implies.

Returns a dict with kinds (subset of ["llm", "vlm"] in pull order), application (the benchmark axis for the text model), and matched (the keywords that fired, for the report’s justification). A task that mentions nothing visual still gets an LLM on the generalist axis; any vision keyword adds a VLM.

Parameters:

task (str | None)

Return type:

dict[str, Any]

best_engine_ai_helper.platform_name()[source]

Return the current OS as a short lowercase string.

Returns:

One of: ‘darwin’, ‘linux’, ‘windows’.

Return type:

str

Examples

>>> platform_name() in ('darwin', 'linux', 'windows')
True
best_engine_ai_helper.rank(hw, catalog, kind, headroom=0.85, application=None)[source]

Return all candidates sorted by benchmark score, annotated with fit status.

Each entry in the result gets a _fits key (bool) indicating whether it fits within the effective budget. The top entry is identical to what select() would return.

Parameters:

  • hw (dict[str, float | None]) – Output of detect.available_memory().

  • catalog (list[dict[str, Any]]) – Merged model entries from catalog.load_catalog().

  • kind ({'llm', 'vlm'}) – The inference kind to filter and rank by.

  • headroom (float) – Extra safety on top of the accelerator cap. Default 0.85.

  • application (str or None) – Optional use-case keyword ("code", "math", "ocr", "vision", "chat", "generalist"). Drives which benchmark column is used for ranking.

Returns:

Candidates sorted descending by benchmark score, each with _fits.

Return type:

list[dict[str, Any]]

Examples

>>> catalog = [{'id': 'v', 'kind': 'vlm', 'ram_gb': 9.0,
...             'benchmarks': {'vision': 80}}]
>>> rank({'unified_gb': 96.0, 'vram_gb': None, 'ram_gb': 96.0}, catalog, 'vlm')[0]['_fits']
True
best_engine_ai_helper.recommend_engines(hw, catalog, task=None, *, headroom=0.85, compute=None)

Recommend the best engine per needed kind for this hardware and task.

Parameters:
Returns:

A JSON-ready report: the parsed task, hardware, memory budget, and for each needed kind the chosen model plus the full ranked candidate table, with per-model fit and estimated throughput.

Return type:

dict

best_engine_ai_helper.resolved_models()[source]

Return both resolved tags in one call, reading the config file at most once.

Returns:

{"text": <tag>, "vision": <tag>} — the same values text_model() and vision_model() would return.

Return type:

dict

best_engine_ai_helper.select(hw, catalog, kind, headroom=0.85, application=None)[source]

Pick the best-scoring model that fits in available memory.

Candidates for a ‘vlm’ selection include both VLMs and LLMs with vision capability (kind == ‘vlm’). Candidates for a ‘llm’ selection include text-only LLMs and VLMs (since a VLM handles text-only prompts equally).

Selection order: 1. Filter: keep entries whose ram_gb fits within the effective budget. 2. Rank: structured-output capability first (a model that cannot honour

Ollama structured output is never chosen over one that can), then benchmark score (application-specific if given, else vision for VLM or general for LLM). This matches rank(), so rank(...)[0] and select(...) agree.

  1. Last resort: if nothing fits, return the smallest model in the catalog rather than raising; the caller decides whether to warn the user.

Parameters:

  • hw (dict[str, float | None]) – Output of detect.available_memory().

  • catalog (list[dict[str, Any]]) – Merged model entries from catalog.load_catalog().

  • kind ({'llm', 'vlm'}) – The type of model to select.

  • headroom (float) – Extra safety on top of the accelerator cap. Default 0.85.

  • application (str or None) – Optional use-case keyword ("code", "math", "ocr", "vision", "chat", "generalist"). Selects the benchmark axis used for scoring. None uses the default kind-based rule.

Returns:

The selected catalog entry.

Return type:

dict[str, Any]

Raises:

ValueError – If the catalog is empty.

Examples

>>> hw = {'unified_gb': 96.0, 'vram_gb': None, 'ram_gb': 96.0}
>>> catalog = [{'id': 'v', 'kind': 'vlm', 'ram_gb': 9.0,
...             'benchmarks': {'vision': 80}}]
>>> select(hw, catalog, kind='vlm')['id']
'v'
best_engine_ai_helper.text_model()[source]

Return the model tag to use for text-only prompts (lint, summaries, …).

Precedence: BEST_LLM_TEXT env (or legacy SPREZZATURE_LLM_TEXT) -> persisted config.json -> DEFAULT_TEXT_MODEL. Never probes hardware, never raises.

Return type:

str

best_engine_ai_helper.to_markdown(report)[source]

Render a recommend() report as a Markdown document.

Parameters:

report (dict[str, Any])

Return type:

str

best_engine_ai_helper.vision_model()[source]

Return the model tag to use for prompts that include images (alt-text, OCR).

Precedence: BEST_LLM_VISION env (or legacy SPREZZATURE_LLM_VISION) -> persisted config.json -> DEFAULT_VISION_MODEL. Never probes hardware, never raises.

Return type:

str

best_engine_ai_helper.write_report(report, stem)[source]

Write both a JSON and a Markdown rendering; return (md_path, json_path).

Parameters:
Return type:

tuple[Path, Path]