Several requests at once on MLX

The first comparison found that more requests at once added little on Quail's MLX engine, and nothing at all on Gemma 4. Three changes followed.

Gemma 4 now batches

Found: with 8 requests at once, Gemma 4 26B-A4B took 41 s to the first token, against 3.1 s for oMLX. Its requests were served one after another.

Why: the library Quail builds on had no text-only version of this model with its mixture-of-experts layers. So it loaded through the vision model, whose attention takes one position for the whole batch.

Changed: Quail now keeps its own copy of Gemma 4's language model. Each sequence reads its own position from the cache, so sequences of different lengths decode together. A new cache handles the model's sliding-window layers in a batch. A request gives the same reply alone as before.

Result: with 8 at once, the first token in 1.6 s instead of 41 s, and 68.1 tokens a second instead of 44.9. That's the fastest first token of the three engines. (#140, #145)

Prompts read in bigger slices

Found: a lone long prompt was read 512 tokens at a time, where mlx-lm reads 2,048.

Changed: a prompt read with nothing else running now goes in 2,048-token slices, up to 8,192 tokens. A prompt read while others decode stays at 512, so it holds them up less at a time.

Result: on Qwen3.6, a 4,096-token prompt's first token in 7.0 s instead of 8.3 s, and the model changes in Qwen3.5 and 3.6 on MLX then took it to 5.2 s. (#139, #146)

MLX 0.32

Found: a trace of the serving loop showed that the steps decoding 8 requests together took 61–65% of the time. On a Mac mini (M4 Pro), each took about 80 ms, where mlx-lm took 57 ms.

Why: four causes in Quail were ruled out one at a time: memory handling, how each row is sampled, reading tokens back, and building the graph. Then mlx-lm itself was run on the MLX version Quail used, 0.31.1. It was just as slow, 75–85 ms. The difference was MLX, not Quail.

Changed: Quail moved to MLX 0.32.2. No release of mlx-swift-lm, the Swift library Quail builds on, takes that version yet, so Quail uses a pinned commit of it until one does. A weekly check flags that release when it comes (#149).

Result: a step of 8 fell from about 80 ms to 61 ms on the mini. On the comparison's M1 Max, 8 requests at once made 49.8 tokens a second on Qwen3 8B instead of 43.8, and 88.0 on Qwen3.6 instead of 76.7. (#150)

Still to do

With 2 to 8 requests at once, Rapid-MLX still makes 10–20% more tokens a second. #139 stays open for it.