QuailHow Quail works
What Quail is for, how it's built, and how we measure it against the other ways to run models on a Mac, and act on what we find.
What it's for
Quail runs AI models on your Mac and gives your tools an API they already speak: OpenAI's Chat Completions, Responses and Embeddings, and Anthropic's Messages. Its aims:
- One app for both model formats. GGUF and MLX run from one server and one model folder.
- Safe by default. It answers only this Mac, needs a key from the start, refuses other web pages, and sends no usage data.
- Nothing to install around it. It's a signed Mac app, with no Python.
- As fast as the best engine for each format, measured in the open, including where it isn't yet.
How it's built
quail commandpull, chat, launch, bench, ps, logsThe design and every decision behind it are in the repository: the architecture and the decision records.
How we measure and improve speed
- Measure against the others on the same Mac, with the same weights, prompts and settings, one server at a time: Quail against llama.cpp's server and Ollama for GGUF, and against oMLX and Rapid-MLX for MLX. How the comparison works.
- List every gap larger than 5% and larger than the spread between rounds. Each becomes an issue.
- Find the cause before changing anything. We trace where Quail spends its time, and check it against the engine's own reference (mlx-lm, llama-server) on the same Mac. Then we try one change at a time, alternating builds on a cooled Mac.
- Fix it in its own change, with before-and-after numbers and a test that the replies are unchanged.
- Run the whole comparison again and publish it, losses included.
What the first comparison changed
The first comparison found Quail level with llama.cpp on GGUF, except on long prompts, and behind oMLX and Rapid-MLX on MLX in four ways. What changed as a result, measured in the second on the same M1 Max:
Several requests at once on MLX
Gemma 4 now batches: with 8 requests at once, its first token takes 1.6 s instead of 41 s. MLX 0.32 made every batched step about a quarter faster.
Qwen3.5 and 3.6 on MLX
Quail's own model code and a fused GPU kernel: 13.1 ms a token instead of 15.8 ms, and a 4,096-token prompt's first token in 5.2 s instead of 8.0 s.
MLX memory
Freed GPU memory is no longer kept without limit: Qwen3 8B's peak fell from 32.5 GB to 10.4 GB.
Long prompts on GGUF
Earlier requests' caches move out of the way: a 4,096-token prompt now reaches its first token as fast as on llama.cpp's own server.
What's next
- Throughput with several MLX requests at once. Rapid-MLX still makes 10–20% more tokens a second (#139).
- One request on MLX. Rapid-MLX compiles its whole step and fuses more of the model; it's still 7–10% faster alone on Qwen3.6 and Gemma 4 (#141).
- Gemma 4 on GGUF. Its MMLU-Pro score is a few points under llama-server's. The prompts are identical, and Quail's replies match llama-server's with the same cache settings; a larger sample will say whether it's more than chance (#152). The MLX gap the comparison found was a prompt-cache bug, fixed in 0.63.1.