best_engine_ai_helper.detect module

detect — turn raw hardware facts into AI-inference throughput estimates.

Raw probing (subprocess calls to nvidia-smi / rocm-smi / system_profiler / lspci, CPU/RAM introspection) is NOT done here: it lives in os_helper.hardware_utils, a generic cross-platform hardware probe shared by every repo in the AI Helpers suite. This module is the thin, AI-domain layer on top of it — it takes the vendor + chip/GPU name + memory size that os_helper reports and turns them into what a local model picker actually needs: an accelerator kind, a memory-bandwidth estimate, and the memory pool available for inference.

Memory bandwidth is the ceiling on decode throughput (token generation reads the whole active model from memory once per token), so it — not raw core count — is what compute_profile() reports and what score.estimated_tokens_per_second keys its tokens/s estimate on. Apple Silicon publishes per-chip bandwidth; NVIDIA/AMD publish it per GPU model. Both are tabulated below by substring match on the chip/GPU name os_helper reports; an unrecognised model degrades gracefully to “throughput not estimated” rather than a wrong number.

Author

Warith Harchaoui <warith.harchaoui@deraison.ai>

best_engine_ai_helper.detect.available_memory()[source]

Detect available memory for model inference.

Reads three pools from os_helper’s hardware facts:

Apple Silicon unified memory (macOS with Apple chip)
NVIDIA/AMD VRAM (summed across all visible GPUs)
System RAM (always populated)

Returns:

A dict with the following keys:

unified_gbfloat or None: Apple Silicon unified memory pool, in GB.
vram_gbfloat or None: Discrete GPU VRAM (sum of all visible GPUs), in GB.
ram_gbfloat: Total system RAM in GB. Always a positive float.

Return type:

dict[str, float | None]

Examples

>>> mem = available_memory()
>>> set(mem.keys()) == {'unified_gb', 'vram_gb', 'ram_gb'}
True
>>> mem['ram_gb'] > 0
True

best_engine_ai_helper.detect.chip_name()[source]

Return the Apple Silicon chip name (e.g. "Apple M2 Max"), or None.

Delegates to os_helper.apple_chip_name(); None on non-macOS platforms or when the chip line is absent (old Intel Macs).

Return type:: str | None

best_engine_ai_helper.detect.chip_vendor()[source]

Identify the primary compute vendor for model inference.

Returns:: One of: ‘apple’, ‘nvidia’, ‘amd’, ‘intel’, ‘cpu’. Delegates to os_helper.gpu_vendor().
Return type:: str

Examples

>>> chip_vendor() in ('apple', 'nvidia', 'amd', 'intel', 'cpu')
True

best_engine_ai_helper.detect.compute_profile()[source]

Describe the machine’s inference accelerator and memory bandwidth.

Returns a dict with:

accelerator: "gpu-metal" (Apple Silicon), "gpu-cuda" (NVIDIA), "gpu-rocm" (AMD), or "cpu" (no discrete accelerator detected).
chip: the chip / GPU name when known, else None.
bandwidth_gbs: memory bandwidth in GB/s when the chip/GPU matches a known model in _APPLE_BANDWIDTH_GBS / _NVIDIA_BANDWIDTH_GBS / _AMD_BANDWIDTH_GBS, else None. This is the ceiling on decode throughput; token generation reads the whole active model from memory once per token, so tokens/s scales with it.

Bandwidth is tabulated for Apple Silicon (per-chip) and for discrete NVIDIA/AMD GPUs (per-board, matched on the model name os_helper reports). An unrecognised GPU model — a new SKU not yet in the table, or a multi-GPU box where the name string is ambiguous — degrades to bandwidth_gbs: None rather than a fabricated number; callers treat that as “throughput not estimated”, never as zero.

Return type:: dict[str, Any]

best_engine_ai_helper.detect.platform_name()[source]

Return the current OS as a short lowercase string.

Returns:: One of: ‘darwin’, ‘linux’, ‘windows’. Delegates to os_helper.platform_name().
Return type:: str

Examples

>>> platform_name() in ('darwin', 'linux', 'windows')
True

best_engine_ai_helper.detect.server_load()[source]

Snapshot the machine’s CURRENT load — as opposed to its static capacity.

Requires an os-helper release whose hardware_info() includes the live fields cpu.percent / available_ram_gb / disk / gpu_utilization_percent (added alongside this function; not yet in a published os-helper release as of this writing — bump the os-helper pin in pyproject.toml once one ships, or this raises KeyError on a fresh install).

Feeds score.effective_budget()’s optional load parameter, so a recommendation reflects what else is happening on this machine right now: another process (or an already-running engine) can hold memory the static hardware totals from available_memory() know nothing about.

Returns:

available_ram_gbfloat: Free system RAM right now.
cpu_percentfloat: Instantaneous CPU utilization, 0-100.
gpu_percentfloat or None: Live discrete-GPU utilization, 0-100; None on Apple Silicon, CPU-only machines, or when the vendor CLI is unavailable.
disk_free_gbfloat: Free space on the disk holding the home directory (where model caches live).
disk_percent_usedfloat: 0-100.
running_enginesint: Best-effort count of already-loaded Ollama models plus running vLLM server processes.

Return type:

dict[str, Any]

Examples

>>> load = server_load()
>>> load["available_ram_gb"] >= 0
True
>>> 0 <= load["cpu_percent"] <= 100
True
>>> load["running_engines"] >= 0
True