Qwen3.5 and 3.6 on MLX

Alone, Qwen3.6 35B-A3B took 15.8 ms a token on Quail against 12.2 ms on Rapid-MLX. On Qwen3 8B the engines were level, so the gap was in this model family's code.

Quail's own model code

Why: Python's mlx-lm compiles several small calculations in this model into single GPU kernels, and the Swift library ran them step by step. mlx-lm also kept one value at a lower precision.

Changed: Quail now keeps its own copy of the model code (MIT, adapted from mlx-swift-lm) and compiles the same calculations. Weights and downloads are unchanged. Replies at temperature 0 were byte-identical to before. ADR-0066 sets the rules for keeping such copies.

Result: 14.2 ms a token instead of 15.6 on the M1 Max, the same as mlx-lm itself. (#141, #147)

One kernel for each GatedDeltaNet layer

Why: for each new token, three of every four layers in these models ran about a dozen small GPU kernels between their input and output: a convolution, two normalisations, a gate, the recurrence and a final normalisation.

Changed: they now run as one kernel, adapted from Rapid-MLX (Apache-2.0). The kernel repeats the same steps in the same order. The first time a model loads, Quail runs a random layer both ways and uses the kernel only if its output matches byte for byte. The server's log says which way it went.

Result: 10.79 ms a token instead of 11.22 on a Mac mini (M4 Pro), with identical replies. (#148)

Together

On the comparison's M1 Max, Qwen3.6 now takes 13.1 ms a token instead of 15.8. A 4,096-token prompt's first token takes 5.2 s instead of 8.0 s, the fastest of the three engines.

Tried and not kept

A special kernel for reading long prompts, from oMLX and Rapid-MLX. It saved 1–2% at these lengths.
The Swift library's newer version of this model, which fuses and compiles other parts. It measured 10.99 ms a token against Quail's 10.79 ms.

Still to do

Rapid-MLX compiles its whole single-request step and fuses more of the model's work. It's still about 7% faster alone on Qwen3.6. #141 stays open for it.