How the comparison works

Quail is measured against the other ways to run models on a Mac by a harness in its repository, the same way every time, so a gap is Quail's and not the test's.

The same work for every engine

Two lanes, never mixed. The same .gguf file runs in Quail, llama.cpp's own server and Ollama. The same MLX folder runs in Quail, oMLX, Rapid-MLX and Ollama's MLX runner.
The same settings: 8 slots of 8,192 tokens, a full-precision KV cache, temperature 0, one seed, and thinking off.
One server at a time, on a quiet Mac. The run refuses to start on battery, in Low Power Mode, during a backup, or with another model server running. It pauses Spotlight and the photo-analysis daemons while it runs.
Cool before every measurement. Each level waits until the Mac's thermal state is Nominal, and a level that ends hotter than Moderate is run again. Two rounds alternate which engine goes first.

What's measured

Speed, with GuideLLM: 512-token prompts with 256-token replies at 1, 2, 4 and 8 requests at once, and 4,096-token prompts with 128-token replies at one. That gives throughput, time to the first token, time between tokens, peak memory and power.
The engine's own ceiling, with no server in the way (llama.cpp's llama-batched-bench, mlx_lm.benchmark). A server isn't blamed for its engine.
Quality, with lm-eval: 150 GSM8K problems and 112 MMLU-Pro questions, compared item by item (McNemar's test), so two engines are judged on the same questions.
Tool calling, with BFCL: 40 cases in each of five categories.

What counts as a gap

The report lists every level where another engine did better than Quail by more than 5%, and by more than the spread between rounds. Each gap becomes an issue, and is fixed only once its cause is known.

Finding the cause

Trace it. QUAIL_MLX_TRACE records how long the MLX serving loop spends on each step, prompt slice and join. It showed that 8 requests' decoding steps took most of the time.
Check against the engine's own reference on the same Mac, such as mlx-lm or llama-server with the same settings. It shows whether the gap is Quail's code or the engine underneath.
Change one thing at a time, with alternating builds on a cooled Mac, and check that the replies at temperature 0 are unchanged.

The harness and the reports

It's all in the repository: the harness, its design (ADR-0063), and every report, with its raw results, under docs/benchmarks. task bench:compare runs it on any Apple silicon Mac.