QuailBenchmark
Measure a model on your own Mac, the same way every time, so runs can be compared — across models, formats, settings and Macs.
Run one
Start the server, open Benchmark (in the menu or the Quail window), choose a model and click Run Benchmark. A small model takes about a minute. Whatever was loaded before is loaded again afterwards. From the terminal: quail bench <model>.
To see how Quail compares with Ollama, oMLX and Rapid-MLX, measured on the same Mac with the same weights, see Compare.
What it measures
The fixed quail-bench-1 suite: one warm-up, then three runs of each, reporting the median with the lowest and highest.
| Prompt 512 / 4K | How fast it reads a prompt of 512 and 4,096 tokens (tokens a second). |
|---|---|
| Generate | How fast it writes 256 tokens (tokens a second). |
| 1st token | Time to the first token of a reply to the 512-token prompt. |
| Next turn | Time to the first token when a conversation continues, using what's already cached. |
| 4 at once | Total tokens a second with four requests at the same time. |
| Load | Time to load the model from disk. |
Each result also records the Mac, the model file, the engine and its settings, and the conditions: thermal state, Low Power Mode, battery, and other models loaded.
Another server
quail bench --url http://127.0.0.1:11434 [model] times any server that speaks OpenAI's chat completions (Ollama, LM Studio, llama-server, oMLX, or Quail on another Mac) from the terminal, without the app. Add --api-key if the server needs one. With no model named, it uses the first one the server lists.
It runs the same tests except Load, as a suite of its own, quail-bench-url-1, because it can't see inside the server:
- The prompts are text. They are sized from the server's own token counts, so a "512" prompt may come to 509 or 517; the result says which.
- Times are taken in the terminal, so prompt speed includes the trip to the server and the first token.
- A server that ignores the request to write every token asked for (Ollama, oMLX, Rapid-MLX) may stop early; the result says so.
Its numbers aren't compared with quail-bench-1's, and nothing is saved. --json prints the full result.
Compare
Select two results to compare them: the older is the baseline, and each figure reads “1.23× faster” or “19% slower”. Right-click to set a different baseline. Copy as Markdown and Export JSON… share results. After a benchmark, the Models page shows the measured speed next to each model.
Fair numbers
- Plug a laptop in, and quit busy apps.
- Macs slow down as they heat up. Let the Mac cool between big runs, and alternate the order when comparing two things.
- Compare like with like: the same context size and KV cache, and nothing else loaded.