A 2013 Mac Pro runs a 35B model: 66 tok/s reading (llama-bench pp512), ~11 tok/s writing

Contents

1. Whose code we measured#

Earlier versions of this article called the tree we measured "stock" and "upstream llama.cpp". It was neither. It is ai-janitor/llama-metal at commit 72ba023: a separate repository by one author, not a GitHub fork, created on 16 February 2026, and at 72ba023 its history was a single root commit with no parent. It describes itself as "Metal GPU backend for llama.cpp — AMD/Intel non-Apple-Silicon support". Commit 72ba023 does not exist in ggml-org/llama.cpp. We had never checked whose code we were measuring.

That repository rewrote the Metal code; among the changes its root commit lists is runtime SIMD-width detection for 16-, 32- and 64-wide hardware. Mainline ggml-org/llama.cpp has no ggml-metal/matmul/ directory and no chunks_per_thread in its Metal code. So the four defects — the wave-64 reduction bug, the q6_K loader bug, the vector-path crash from pp320 and the build error from a missing #include <string> — all belong to ai-janitor/llama-metal, not to llama.cpp. Below, "base state" means ai-janitor/llama-metal 72ba023 plus only what is named.

Credit where it is due: ai-janitor/llama-metal is the only path we found that makes Metal compute on these GPUs at all. Everything about Metal in this article stands on it.

2. Mainline llama.cpp on this hardware#

We then measured mainline ggml-org/llama.cpp at commit 7cf1c54. Provenance: a shallow clone containing only that commit; its local tree hash, 860d4eb…, equals the tree hash GitHub's API returns for 7cf1c54 in ggml-org/llama.cpp; our local modifications are all uncommitted and were excluded.

It builds and sees both cards. With Metal on macOS 12 the build exits with code 0, with no #include <string> added. It enumerates MTL0 and MTL1, "AMD Radeon HD - FirePro D700", 6144 MiB each, family Common3, and reports simdgroup reduction = false.

It does not run matrix multiplication on these GPUs. test-backend-ops on card 0: all 1649 MUL_MAT cases and all 904 MUL_MAT_ID cases are "not supported". The cause is in the code: ggml-metal-device.m lines 1119–1120 enable SIMD-group reductions only for Apple7 or Metal3 devices, and lines 1834–1841 allow MUL_MAT only with them. So mainline hands matrix multiplication on these cards to the CPU. We found no public issue, pull request, discussion or document that discusses this gate; the only written statement of scope is the README, which lists Metal as Apple Silicon. Issue #19563 shows that some discrete AMD GPUs report Metal3, pass this gate, and produce wrong output.

With -ngl 99 it aborts: ggml-metal-device.m:2327: GGML_ASSERT(buf_src) in ggml_metal_buffer_set_tensor. Apple's requirement: makeBuffer(bytesNoCopy:…) needs a page-aligned pointer and a length giving a page-aligned region. GGUF tensors are tightly packed, and on the D700 the call returns nil; which condition is violated we did not establish. This crash is already known. Issue #26949 of 12 August 2026 describes exactly this mechanism on an Intel Mac with a discrete AMD GPU and gives the cause and a two-line fix; it was closed by the stale bot as not planned on 10 October 2026, the same day we reproduced it. We did not find it first; we confirmed it on another machine.

Earlier efforts for this hardware class, none merged: #15228 (opened 11 August 2025, closed by the stale bot as not planned on 3 May 2026), #19527 (a pull request opened 11 February 2026 by ai-janitor, the same author as llama-metal; still open and unmerged), #19563 (opened 12 February 2026, closed by the stale bot as not planned on 7 May 2026). These are the facts; we draw no conclusion about anyone's motives.

3. One table, one set of conditions#

This is the article's central comparison. All rows: Qwen2.5-Coder-7B-Instruct Q4_K_M, llama-bench -p 512 -n 128 -t 2 -fa 0 -r 3; the GPU rows on two cards, -sm layer -ts 1/1. These are exactly the blog's conditions except its context setting, which llama-bench cannot express.

Row	prompt reading pp512	generation tg128	Correctness, same binary
CPU, mainline 7cf1c54, -ngl 0	20.58 tok/s (20.60 / 20.53 / 20.58)	3.42 tok/s (3.43 / 3.40 / 3.42)	CPU
Metal, ai-janitor/llama-metal + our fixes	52.36 tok/s (52.37 / 52.36 / 52.31)	11.73 tok/s (11.75 / 11.73 / 11.66)	941/941, 616/616
Vulkan/MoltenVK, unmodified mainline 7cf1c54	75.46 tok/s (74.97 / 75.78 / 75.46)	14.61 tok/s (14.57 / 14.63 / 14.61)	1017/1128, 817/921 — failures only in i-quants
Linux / Mesa RADV, third-party blog, same machine	—	21.39 tok/s	not measured by us

Caveat, and it belongs right here: -t 2 is the blog's setting, and it cripples the CPU. In our runs Metal generation stayed near 11–12 tok/s both at -t 8 on one card and at -t 2 on two cards; we did not isolate the thread effect. So the GPU-against-CPU generation ratios of this table overstate what the GPU gains. With matched threads, one card, -t 8 on both sides, Metal generated 11.15 tok/s against the CPU's 9.89 tok/s (+12.7%); that CPU run also gave pp512 20.46 tok/s. On macOS Metal's generation gain over a fully threaded CPU is modest; the large gain is prompt reading. We have no -t 8 run for MoltenVK, so we give no MoltenVK-against-best-CPU ratio.

Generation ratios, valid only within this table and only at -t 2: Metal/CPU ×3.43 and MoltenVK/CPU ×4.27 — both inflated by the handicapped CPU, see above; the GPU-against-GPU ratio is unaffected: MoltenVK/Metal ×1.25. Prompt-reading ratios, same table: MoltenVK/Metal ×1.44; Metal/CPU ×2.54. The Linux/RADV row is context only: a different OS, driver and binary, not a run we controlled. The Metal row is the first of two repeats; the second gave 52.32 / 11.70 tok/s.

The Vulkan build is unmodified mainline: tree hash 860d4eb, zero changes of ours in ggml-vulkan. Its correctness, same build, card Vulkan0: MUL_MAT 1017 of 1128 passed, MUL_MAT_ID 817 of 921 passed. All failures are in i-quant types (iq2_xxs, iq2_xs, iq2_s, iq1_s, iq1_m, iq3_s, iq3_xxs); f16, f32, bf16, q4_0, q4_1, q4_K, q5_K, q6_K and q8_0 have zero failures, so the Q4_K_M row is correct. We report the i-quant failures as observed; we did not check whether they are already known in llama.cpp.

A reading, not a measurement: with long inputs and short outputs, prompt reading is the dominant cost, and that is where these cards help most on macOS.

4. ai-janitor/llama-metal 72ba023 on macOS 12: build error and launch failure#

Does not build. Clean ai-janitor/llama-metal at commit 72ba023 does not compile on macOS 12: ggml-backend.cpp:1183, "implicit instantiation of undefined template std::basic_string". One #include <string> is missing; mainline 7cf1c54 builds without it (section 2). Our tree's difference from clean 72ba023 in that file is exactly that one line. Everything called "base state" below is 72ba023 plus that line; otherwise there would be nothing to measure.

Does not run. Even once built, the base state dies at ggml-metal-context.m:415: GGML_ASSERT(buf_dst) failed during llama-bench: Metal returns nil where the code expects a buffer. What makes it run a model is a separate change of ours — a no-copy fallback: if Metal returned nil, the buffer is created and the data copied by hand. It does not change correctness, but without it neither the base state nor "base state + the wave-64 fix" reaches a first token. It is the same class of problem as mainline's buf_src crash in section 2.

Hence: the base state of ai-janitor/llama-metal has no speed on this hardware. There is no "before" to compare against until that fallback is added. It is not slow and not falsely fast — it does not start.

Once the no-copy fallback is added, every vector-path state measured ~9.95 tok/s at pp256 (per-patch series in logs/bench_series.log: 9.95): the base state, the base state with the wave-64 fix, and our full state with tiling disabled (a separate run gave 9.94). ~9.95 is the vector path's figure, not a property of any one state.

5. The correctness ladder#

Every rung was built from scratch from a clean checkout of ai-janitor/llama-metal 72ba023 plus the named change only. The exception is the 929/941 rung: it was built from whole files, and its wave-64 variant is from patch 0006, not 0003 (see UNVERIFIED.md in the repository). Metal, card 0, test-backend-ops test. The tool prints "N/M tests passed"; earlier versions of this article read the first number as failures.

Rung	MUL_MAT	MUL_MAT_ID
base state + no-copy fallback	433 of 869 passed, 436 failed	162 of 616 passed, 454 failed
+ the wave-64 fix	869/869 passed	616/616 passed
+ the tiled dispatch	869/869 passed (unchanged)	616/616 passed (unchanged)
+ our 72 test cases (m = 128)	929 of 941 passed, 12 failures — all q6_K	616/616 passed
+ the q6_K alignment fix	941/941 passed	616/616 passed

Three things from that table.

First, the single wave-64 fix closes every failure of the base state at once — all 436 MUL_MAT failures and all 454 MUL_MAT_ID failures go together. We previously quoted an intermediate rung of 534/869; it did not reproduce on fresh builds, so we withdraw it and replace it with nothing.

Second, the tiled dispatch does not affect correctness at all. It is purely a speed change. The old text gave a different impression — correcting that: 869/869 before it and 869/869 after.

Third, the denominators differ, and they must not be compared silently. 941 = the 869 MUL_MAT cases of the base state plus 72 of ours: 12 each on f16, q4_0, q8_0, q4_K, q5_K and q6_K, all at m = 128 rows. So 433/869 and 941/941 are different test sets, and they cannot be put into one ratio. The MUL_MAT_ID tallies contain none of our cases.

This is not an exotic quantisation edge case. In its base state, 192 of 262 f16 and 143 of 198 f32 MUL_MAT cases fail, and so do the quantised types (failed of total: q4_0 14 of 43, q4_1 16 of 43, q8_0 16 of 43, q4_K 16 of 41, q5_K and q6_K 4 of 11 each, iq4_xs and q3_K 11 of 11 each). That is, ai-janitor/llama-metal at 72ba023 multiplies plain f16 and f32 matrices incorrectly on this GPU class. In the full state there are zero failures in every type.

Bug 1: wave 64 (ai-janitor/llama-metal). Several of its mat-vec kernels compute chunks_per_thread = 32 / NW. With NW = 64 that is 0, so the kernels skip all work. The patch touches 13 annotated sites: 10 actual fixes and 3 optimisations.

Bug 2: the tiled q6_K loader (ai-janitor/llama-metal). It reads two values from one byte, swaps nibbles and subtracts 32 from an unsigned value. Long prompts produced @@@@. The existing tests use m = 16 rows, while the tiled kernel needs m ≥ 64 (a full tile) and n ≥ 16, so they never reach it. That is exactly why the added test set was needed: across the 869 existing cases the bug is invisible, while our 72 find 12 failures, and all 12 are q6_K. q6_K blocks are 210 bytes, so every second block is only 2-byte aligned, and our reading of the observed failures, not a sourced fact, is that GCN before Vega returns bytes from the aligned address on an unaligned 4-byte load.

Separately, and not in a footnote: in the tree where the mat-vec kernels were developed, test-backend-ops test -o MUL_MAT shows one failure — q2_K, m = 16, n = 8, k = 4096. It is identical before and after that rewrite, i.e. unrelated to it; no q2_K fix was carried into that tree.

Lesson: a short query plus a benchmark is not a correctness test. Run a long real prompt, and run the tests beside it.

6. The broken state benchmarks faster than the correct one#

This is the lesson worth keeping from the Metal work, and it takes two rows to measure. ai-janitor/llama-metal 72ba023, the per-patch series, one card, Qwen2.5-Coder-7B-Instruct Q4_K_M, -ngl 99 -fa 0 -t 4 -r 3, each length as its own run, builds from scratch (https://github.com/SynenergyAI/mac-pro-2013-llama-metal, logs/bench_series.log lines 7 and 10; correctness logs/series_run.log lines 2 and 3):

State	Correctness, MUL_MAT	generation tg128
base state + no-copy fallback (patch 0002)	433 of 869 passed, 436 failed	67.03 tok/s (66.77 / 67.03 / 67.11) — WRONG
+ the wave-64 fix (patch 0003)	869/869 passed	7.45 tok/s (7.49 / 7.41 / 7.45)

The broken state is about 9× faster because its wave-64 mat-vec kernels skip their work: they compute zero chunks per thread and exit, while 433 of 869 cases still pass. The benchmark cannot see that — it measures time, not output.

The anomaly is reproduced from a clean build, with the correctness table beside it. The moment wave 64 is fixed, 67.03 turns into 7.45. An earlier run on a different tree gave 67.12 for the same broken state; the series above replaces it.

So: a benchmark figure without a correctness check is worth nothing. Here it was wrong by a factor of about nine, and no error appeared anywhere.

7. The honest before and after, inside ai-janitor/llama-metal#

This pair is internal to ai-janitor/llama-metal, and it is legitimate only because both of its sides pass the correctness check. "Before" is the first correct state of the per-patch series, patch 0003 (869/869 and 616/616). "After" is the full series, patch 0008 (941/941 and 616/616). One card, Qwen2.5-Coder-7B-Instruct Q4_K_M, -ngl 99 -fa 0 -t 4 -r 3 (https://github.com/SynenergyAI/mac-pro-2013-llama-metal, logs/bench_series.log lines 8–10 and 23–25):

	before, 0003 (869/869)	after, 0008 (941/941)
prompt reading pp256	9.95 tok/s (9.96 / 9.95 / 9.92)	49.23 tok/s (49.11 / 49.26 / 49.23) — ×4.95
generation tg128	7.45 tok/s (7.49 / 7.41 / 7.45)	12.89 tok/s (12.92 / 12.89 / 12.89)
prompt reading pp512	crashes (see section 9)	52.58 tok/s (52.66 / 52.58 / 52.08)

In short: the patches make prompt processing ~5x faster (×4.95). At pp512 there is no pair: the "before" state crashes there, so there is nothing to compare with. These one-card, -t 4 figures are not to be divided by the two-card, -t 2 figures of section 3.

What is no longer here. We withdraw the pair "9.9 → 52.33, ×5.29": it combined figures from two different prompt lengths (pp256 and pp512) and presented the vector-path figure as the base state's speed. We also replaced an earlier before/after measured on a different tree with the per-patch series above.

8. Other public measurements of this machine#

A public record for this machine exists. We previously wrote that only one public measurement of this hardware exists. That too was wrong. There are at least three sources:

• a blog post of 26 May 2026 on the same machine with a D700 pair, but under Ubuntu with Mesa RADV and Vulkan, i.e. with macOS removed: Qwen2.5-Coder-7B-Instruct Q4_K_M — 21.39 tok/s decode, Qwen3.6-35B-A3B Q4_K_M — 7.04;
• llama.cpp issue #19615 (https://github.com/ggml-org/llama.cpp/issues/19615; February 2026, closed as a duplicate; third-party figures, not checked by us): a D700 pair with DeepSeek-R1-Distill-Qwen-14B Q4_K_M, ~11 t/s generation and ~70 t/s prompt processing with RADV workarounds — a different model, conditions not matched to ours;
• llama.cpp issue #29596 (https://github.com/ggml-org/llama.cpp/issues/29596; 28 September 2026, open): the same machine, the GPUs SLOWER than the CPU — third-party figures; the ticket does not make clear which are pp and which tg.

What is actually defensible: we found no public figures for this hardware via Metal, and the two kernel bugs we fixed are in ai-janitor/llama-metal; the patches are being prepared for its author (https://github.com/SynenergyAI/mac-pro-2013-llama-metal). Both statements are narrow by exactly that much.

For context, the blog's Linux/RADV run generated 21.39 tok/s on the 7B model; they had Linux with an open driver and more freedom, we stayed on the original macOS 12, so the two are not a controlled comparison. A reading rather than a measurement: generation is memory-bandwidth bound, and by our calculation (not a measurement) 11.7 tok/s generation through Metal is about 25% of the 221 GB/s we measured a Metal kernel reading, or ~21% of the 264 GB/s spec. On macOS, unmodified mainline over MoltenVK already generates ×1.25 faster than our Metal path.

For public cross-hardware tables there is one more rung: Llama-2-7B Q4_0 (the file from TheBloke/Llama-2-7B-GGUF — the model is confirmed), -ngl 99 -fa 0 -p 512 -n 128 -r 3, ai-janitor/llama-metal with our fixes, default thread count:

	pp512	tg128
one card	54.69 tok/s (54.69 / 55.33 / 54.65)	7.92 tok/s (7.92 / 7.91 / 7.93)
two cards (-sm layer -ts 1/1)	55.04 tok/s (55.04 / 55.04 / 55.08)	8.48 tok/s (8.48 / 8.48 / 8.47)

9. Crash boundaries in ai-janitor/llama-metal#

We first attributed the crash at ggml-metal-context.m:319 to our own build. The measured boundaries say otherwise: the crash reproduces in ai-janitor/llama-metal's vector path, in its base state too; the cause of the crash at :319 is not established. Mainline cannot be compared here: it does not run MUL_MAT on these cards via Metal (section 2).

• The vector path on one card crashes from pp320 upward: pp320, pp384 and pp448 all give exit code 134 and :319, while pp256 passes. The same in both rungs built on the base state — pp256 works, pp512 crashes.
• Our full state on two cards is clean at pp320, pp384, pp448 and pp512, and with splits -ts 1/1, -ts 4/6 and -ts 6/4 — about 51–52 tok/s throughout, and nothing depends on the split between the cards.
• Our full state on one card at pp512 works: 52.48.

The tiled path routes around that vector-path crash — and that is a real part of what our change buys, separately from speed.

One crash of our full state on two cards was seen earlier, and it did not reproduce: the same command passed 2 times out of 2. It had been queued immediately after two other processes that had crashed. We record it as one unreproduced crash and build no claim on it.

10. SIMD reductions do work on GCN1 at width 64#

Mainline llama.cpp enables SIMD-group reductions only for Apple7 or Metal3 devices (section 2). The D700 is neither, so mainline sends matrix multiplication on it to the CPU. Apple's Feature Set Tables list SIMD-scoped reductions and permutes for the Mac2 family, so the hardware has them; we found no public discussion of why the gate is drawn where it is. ai-janitor/llama-metal rewrites the reductions to be width-agnostic, and our work started from it. A micro-benchmark on card 0 (4096 threadgroups × 64 threads, 256 reductions per thread):

Reduction	Correct	Time
simd_sum over 64 lanes	yes	993 µs
simd_shuffle_xor within 32 lanes + LDS	yes	1663 µs
quad_sum + LDS tree	yes	2059 µs
LDS tree	yes	2064 µs
sequential loop in LDS	yes	5628 µs
simd_shuffle_xor across lane 32	yes	11898 µs (emulated)

Each of the six variants was measured five times; the table states the medians. For the best variant the five runs were 993, 993, 1025, 1010, 985 µs — median 993, best single run 985 (log: logs/simd-microbench.log).

Our earlier conclusion that forced SIMD reductions "produce garbage on this GPU" was wrong. The garbage came from kernels that assumed a SIMD width of 32. Rewritten to be width-agnostic, the reductions work — and the fastest variant is precisely the one mainline's gate does not allow on this GPU.

11. The tiled path#

In ai-janitor/llama-metal, on GPUs without matrix hardware every quantized matmul went to the vector kernel at any batch size, although tiled kernels for q4_K and q6_K existed. A threshold (GGML_METAL_TILED_MIN_BATCH, default 16) is compared against the batch width n; separately, the tiled path needs m ≥ 64 rows of the weight matrix.

Tiling in isolation is one and the same binary of our full state, with the threshold raised and without it, so the difference is exactly the tiled path against the vector path, and both sides are 941/941 correct. One card, pp256: 9.94 → 49.02 tok/s (×4.93) (a separate run, not the per-patch series). At pp512 the vector side crashes, so there is no pair there (section 9).

Separate kernel measurements and one further pair, both sides our own tree, vector path against tiled (here m is rows of the weight matrix and n the batch width):

• q4_K matmul at n = 512: 123.0 → 668.3 GFLOPS;
• q6_K matmul at n = 512: 137.7 → 704.2 GFLOPS;
• 14B split over both cards (pp256): 4.81 → 24.78 tok/s.

Correctness, to repeat, is not affected by the tiled dispatch at all: 869/869 before it and 869/869 after. What it buys is speed, and a way around the vector path's defect.

12. Rewriting the mat-vec kernels#

The q4_K mat-vec kernel moves 33 MB of weights in 449 µs, i.e. ~73.5 GB/s out of the 264 spec — about 28% of bandwidth. Neither the weight traffic nor the arithmetic alone explains the gap:

• removing all nibble unpacking cut the kernel time by 30% (not logged);
• reading weights from cache instead of memory cut the time by 11% (not logged);
• more waves made it slower: one row per wave took 557 / 551 / 555 µs, two rows (the baseline) 444 / 446 / 443 µs; three runs each (log: logs/ablation-q4K-rows.log).

This pair was re-measured with other load present, so a small disturbance is possible. The load-bearing part here is the ratio of 1.25, not the absolute values; 444 µs agrees with the "before" figure of 449 µs.

Every wave re-reads its slice of the input vector. For m = 4096, k = 14336 that is ~117 MB of cache traffic against 33 MB of weights. The new layout uses 8 threads per super-block, one 16-byte load for quants and one for the block header, and 8 rows per wave so the vector slice is reused.

Measurement conditions for this section. The µs figures come from test-backend-ops perf -o MUL_MAT on a build of ai-janitor/llama-metal at commit 72ba023; "before" is its mat-vec kernel as found, "after" is our rewrite. The generation figures are the per-patch series: one card, Qwen2.5-Coder-7B-Instruct Q4_K_M, -ngl 99 -fa 0 -t 4 -r 3, tg128 (https://github.com/SynenergyAI/mac-pro-2013-llama-metal, logs/bench_series.log lines 16, 19, 22, 25). Patch 0006 uses all 64 lanes, 0007 rewrites the q4_K kernel, 0008 rewrites the q6_K kernel.

Kernel (m × k; m is rows of the weight matrix, n the batch width)	Before	After
q4_K 4096 × 14336	449 µs	357 µs (−20%)
q4_K 14336 × 4096	528 µs	407 µs (−23%)
q6_K 4096 × 14336	883 µs	596 µs (−33%)

Model, generation tg128 (Metal)	0005	+ 0006 (64 lanes)	+ 0007 (q4_K)	+ 0008 (q6_K)
Qwen2.5-Coder-7B Q4_K_M	7.42	11.20	12.21	12.89 tok/s

The 883 → 596 µs improvement cannot be attributed to alignment alone: the same rewrite changed three things at once — the aligned read as pairs of 16-bit values, 8 rows per wave instead of 2, and the load width (one 16-byte load per super-block). The effect is confounded.

After the rewrite the same q4_K kernel moves those same 33 MB in 357 µs, i.e. ~92.4 GB/s, about 35% of the 264 spec. Those are one kernel's figures. Whole-model token generation is a different quantity: by our calculation (not a measurement) even after this work it uses about a quarter of memory bandwidth. That is the headroom our Metal path still has.

13. Vulkan/MoltenVK: the earlier pair and tuning#

MoltenVK from Vulkan SDK 1.4.328.1, installed in the home directory without admin rights. We did not install SDK 1.4.357: in our observation (not logged) its installer needs macOS 13.

Every output from 5 individual requests (non-public long Ukrainian texts) was compared with the Metal output on the same inputs — an informal, non-blind check — and showed the same facts and numbers. The Vulkan correctness figures are those of the unmodified mainline build in section 3.

The Metal↔Vulkan pair was measured as a true pair: one command, one session, one tree and two builds, one per backend (the Metal build carrying our mat-vec rewrite of section 12), card 0, -p 512 -n 64 -r 3 -fa 0. gemma-4-E4B runs on both backends with -ot per_layer_token_embd.weight=CPU — never stated before, and it matters. The measured ratios: gemma-4-E4B ×1.44 on prompt reading and ×1.36 on generation; Qwen2.5-7B ×1.41 and ×1.08. These are one-card, -n 64 ratios from a different build, not to be combined with the ratios of section 3.

	Metal (tuned)	Vulkan
gemma-4-E4B, prompt 512 / generation	76.63 / 11.77 tok/s	110.28 / 15.99 tok/s (×1.44 / ×1.36)
Qwen2.5-7B, prompt 512 / generation	52.47 / 12.89 tok/s	74.23 / 13.98 tok/s (×1.41 / ×1.08)

Tuning mattered less than expected. Only -ub 1024 -b 2048 helped: +4.6% and +3.6% on prompt reading at two prompt lengths (one card, the small model). Of the eight other knobs (MVK_CONFIG_*, GGML_VK_*, plus the llama.cpp batch-size, fusion and f16 flags), five had no effect (±0.3%) and three made things worse: -ub 2048 cost 8.7% of generation, disabling fusion cost 8.8% of generation, and disabling f16 cost 1.7% of prompt reading. Parallel slots (-np 4) raised total throughput — reading and generation together — by 5.8%, while generation itself rose 24% (14.62 → 18.14 tok/s) (log: logs/vulkan-tuning-e4b.log).

14. Three MoltenVK pitfalls#

Prompt cache in RAM (Vulkan/MoltenVK). After every request the server copies the KV cache off the GPU. Through MoltenVK this takes 21.3 and 23.5 s for Qwen2.5-7B on one card, 56.4 and 138.8 s for Qwen3-8B on two cards, and in the E4B setup we saw one pause of 13.2 s; the log is empty throughout. The ceiling here is 138.8 s, not the 85 s we wrote earlier: this is a case of our own number having been too low. In the cases we checked — the configurations in the table below — --cache-ram 0 removes the pause and --no-cache-prompt does not (log: logs/cache-pause-qwen25-7b.log).

Qwen2.5-7B-Instruct Q4_K_M on Vulkan/MoltenVK, one visible device, context 8192, 4 threads, flash attention off, 3 requests in a row (wall / compute, s)	1	2	3
default	34 / 34	64 / 42	86 / 62
--no-cache-prompt	34 / 34	71 / 50	93 / 69
--cache-ram 0	34 / 34	42 / 42	62 / 62

-ub 1024 with a large context (Vulkan/MoltenVK, Qwen3-8B over both cards). Qwen3-8B with a 12k context window, on prompts of 1.8–2.8 thousand tokens, generated at 2.8 tok/s and then hit a GPU Timeout (DeviceLost) (log: logs/qwen3-8b-two-cards.log). With -ub 512: 5 requests in 348 s, no errors, on prompts of 1.8–10 thousand tokens and generating at 12.6–8.2 tok/s. llama-bench with a small context does not reproduce it. Our guess, not confirmed, is the attention buffers without flash attention. -ub appears in this article with four values, and all four are real, each in its own context: -ub 1024 on Vulkan with E4B on one card, -ub 512 on Vulkan over two cards, where -ub 1024 aborted on Qwen3-8B, -ub 256 in the configuration we settled on, gemma-4-12B over two cards, and -ub 2048 as a tuning knob on one card (section 13), where it cost 8.7% of generation. Those are four measurements in four contexts, not one recommendation.

--swa-full without flash attention (gemma-4-12B). gemma-4-12B has layers with different V sizes. Without flash attention the server pads the V cache to 2048 per layer, both cards fill up, and generation drops to 0.45 tok/s — measured on Vulkan, gemma-4-12B, two cards, with --swa-full and flash attention off. Without --swa-full and with a 17k context: 7.0–7.7 tok/s from short to long requests, median 7.5 (Vulkan, gemma-4-12B over two cards, -ub 256, context 17408 — the configuration we settled on). Flash attention was slower on Vulkan here. Measured in a single session on the same machine (Vulkan, gemma-4-12B QAT q4_0, two cards, -ts 4,6 -ub 512, context 17408, a prompt of 16,030 tokens): with flash attention off, prompt reading 66.2 tok/s and generation 5.9 tok/s; with it on, 17.9 and 2.2 tok/s — roughly ×3.7 on prompt reading and ×2.7 on generation (log: logs/flash-attention-gemma12b.log). Free VRAM was actually higher with flash attention on (card 1: 1.27 GB against 0.47 GB), so a free-VRAM shortage does not explain this — flash attention was slower on Vulkan here; the mechanism is not isolated. That is why the configuration we settled on carries -fa off. The chronology matters: Metal came first, then Vulkan — and we had --swa-full enabled on both backends. That is a fact about what was configured, not evidence that the flag was necessary: the 0.45 tok/s cost was measured on a different model and configuration (Vulkan, gemma-4-12B, two cards). So the flag is harmless or necessary in one configuration and expensive in another, and we did not measure what it does on the E4B setup.

15. Qwen3.6-35B-A3B (mixture of experts) on this machine#

Prompt processing 66 tok/s (llama-bench pp512); generation ~11 tok/s (tg128 10.99 at -t 8, 9.01 at -t 2) with 10 expert layers on the GPUs. The best published figure we found for this model on this hardware is 7.04 tok/s (blog, Linux/RADV, -t 2); the blog has no prompt-processing figure for this model. They had Linux with an open driver and more freedom; we stayed on the original macOS 12. Reproduce or beat ours.

On real prompts via llama-server, prompt reading is lower than llama-bench pp512: 42.7 tok/s (10 expert layers on the GPUs, prompts ~2.5k tokens(log: logs/server-35b-real-prompts.log), 39.0 tok/s (all experts on the CPU, 65k context), 34.3 tok/s on a 37.5k-token prompt and 31.3 tok/s at 61.7k.

Conditions: mainline ggml-org/llama.cpp 7cf1c54, Vulkan via MoltenVK, two cards, -sm layer -ts 1/1 -fa 0 -p 512 -n 128 -r 3. File: unsloth/Qwen3.6-35B-A3B-MTP-GGUF, Qwen3.6-35B-A3B-UD-Q4_K_M.gguf. Expert layers placed on the GPUs are split evenly between the two cards with -ot; all other expert tensors stay on the CPU.

Expert placement	Threads	pp512 tok/s	tg128 tok/s	Log
all expert layers on CPU (-ncmoe 41)	2	60.77	7.95	https://github.com/SynenergyAI/mac-pro-2013-llama-metal, logs/moltenvk_35b.log
10 expert layers on the GPUs	2	66.14	9.01	https://github.com/SynenergyAI/mac-pro-2013-llama-metal, logs/moltenvk_35b_experts_on_gpu.log
all expert layers on CPU (-ncmoe 41)	8	60.94	10.13	https://github.com/SynenergyAI/mac-pro-2013-llama-metal, logs/moltenvk_35b.log
10 expert layers on the GPUs	8	66.39	10.99	https://github.com/SynenergyAI/mac-pro-2013-llama-metal, logs/moltenvk_35b_experts_on_gpu.log
Blog, Linux/RADV	2	—	7.04	third-party figure, 26 May 2026

Caveat: the blog's expert placement is unknown, and the thread counts differ between rows.

16. What we got wrong#

This article is worth what its corrections are worth, so they stand here rather than in footnotes.

• "Four defects belong to upstream llama.cpp." Wrong. The wave-64 bug, the q6_K loader bug, the vector-path crash from pp320 and the missing #include <string> belong to ai-janitor/llama-metal at 72ba023, a separate repository. We had never checked whose code we were measuring (sections 1 and 4).
• "433 of 869 MUL_MAT cases failed." Wrong polarity: the tool prints passes. 433 passed and 436 failed; for MUL_MAT_ID 162 of 616 passed and 454 failed (section 5).
• "Mainline's GPU-family gate is a deliberate, cautious design choice." Unestablished: we found no public discussion of it at all; the only written statement of scope is the README listing Metal as Apple Silicon (section 2).
• "Mainline's crash with GPU offload is our first finding in llama.cpp." Wrong: it was reported in issue #26949 two months earlier. We confirmed it on another machine (section 2).
• "Vulkan extracts value where Metal does not." Overstated: on macOS the generation gap is ×1.25 (MoltenVK against our Metal path) (section 3).
• "Forced SIMD reductions produce garbage on this GPU." Wrong. The garbage came from kernels that assumed a SIMD width of 32; the reductions themselves work over 64 lanes and are the fastest variant (section 10).
• "The prompt-cache pause reaches 85 s." Our own number was too low: the ceiling is 138.8 s (section 14).
• Earlier before/after and kernel-rewrite generation figures. They were measured on a different tree; sections 6, 7 and 12 now use the per-patch series in https://github.com/SynenergyAI/mac-pro-2013-llama-metal, logs/bench_series.log.
• "Prompt reading 9.9 → 52.33, ×5.29." Withdrawn. The ratio combined two different prompt lengths (pp256 and pp512) and presented the vector-path figure as the base state's speed. The honest ratio is ×4.95 at pp256 (sections 4 and 7).
• "After a partial wave-64 fix — MUL_MAT 534/869." Withdrawn with nothing in its place: on fresh builds of a clean checkout that intermediate rung did not reproduce, and the full fix gives 869/869 at once (section 5).
• "The crash on two cards is a defect of our build." Withdrawn: the same command passed 2 times out of 2. We record it as one unreproduced crash. The real crash boundary lies in ai-janitor/llama-metal's vector path, its base state included (section 9).
• "Only one public measurement of this hardware exists." Wrong: there are at least three public sources. The narrow, correct statement is that we found no public figures via Metal (section 8).

17. Limits#

• One machine.
• Through Metal, generation still uses roughly a quarter of the memory bandwidth; unmodified mainline over MoltenVK generates ×1.25 faster than our Metal path (section 3). The Linux/RADV figures are context from a different OS, driver and binary, not a comparison we controlled.
• The blog's context setting could not be matched: llama-bench has no such parameter, so the standard -p 512 -n 128 rung was used.
• We did not check whether the MoltenVK i-quant failures of section 3 are already known in llama.cpp.
• We did not measure power draw ourselves. Apple's own published document (https://support.apple.com/102839) for the dual-FirePro-D700 configuration states 44 W at idle and 270 W maximum; that row is for a specific configuration (12-core, 64 GB), and Apple measured its own workload, not LLM inference. Separately, Apple's technical specifications (a third-party figure) state a "maximum continuous power" of 450 W — that is the machine's electrical rating, NOT a consumption figure, and the two are widely conflated. We have no figure for our own draw under LLM load, so the article makes no claim about running cost.
• There was no blind quality grading anywhere in this article.

The configuration we settled on#

• Vulkan/MoltenVK, gemma-4-12B QAT q4_0 across both cards, context 17408
• Metal with our fixes, as a fallback: the only Metal state we know of that computes matrix multiplication correctly on these cards

The conclusion again. On a 13-year-old Mac Pro on its original macOS 12, Qwen3.6-35B reads 66 tok/s (llama-bench pp512; 31–43 tok/s on real prompts via llama-server) and writes ~11 tok/s (section 15); the best published figure we found for this model on this hardware is 7.04 tok/s. For a 7B model, unmodified mainline llama.cpp over Vulkan/MoltenVK needs none of our patches (pp512 75.46 against 52.36 tok/s, tg128 14.61 against 11.73 tok/s for our Metal path, section 3). Our Metal work's value is correctness: ai-janitor/llama-metal computed wrong in its base state (436 of 869 MUL_MAT cases failed), with our fixes it passes 941/941 and 616/616, and the patches are being prepared for its author.

What stays open: how far this machine goes with other expert placements and thread counts. Reproduce or beat ours: https://github.com/SynenergyAI/mac-pro-2013-llama-metal.

API notes for gemma 4 and Qwen3.6 in llama.cpp: the chat templates accept only enable_thinking, so reasoning_effort levels have no effect. A per-request budget is thinking_budget_tokens. Do not send a budget of 0 together with enable_thinking: false; in our observation (not logged) this build breaks the response.

Reproduce#

# Vulkan SDK without admin rights (1.4.357 needs macOS 13)
./vulkansdk-macOS-1.4.328.1.app/Contents/MacOS/vulkansdk-macOS-1.4.328.1 \
  --root "$HOME/VulkanSDK/1.4.328.1" --accept-licenses --default-answer --confirm-command install
source ~/VulkanSDK/1.4.328.1/setup-env.sh
# llama.cpp = unmodified mainline ggml-org/llama.cpp; we measured commit 7cf1c54
cmake -S llama.cpp -B build-vk -DGGML_METAL=OFF -DGGML_VULKAN=ON -DCMAKE_BUILD_TYPE=Release
cmake --build build-vk -j6 --target llama-server llama-bench test-backend-ops
build-vk/bin/test-backend-ops test -b Vulkan0 -o MUL_MAT
build-vk/bin/llama-bench -m model.gguf -ngl 99 -sm layer -ts 1/1 -fa 0 -t 2 -p 512 -n 128 -r 3

The Metal correctness check for ai-janitor/llama-metal with our patches — on the same binary you benchmark with, and before the benchmark, not after:

build/bin/test-backend-ops test -o MUL_MAT
build/bin/test-backend-ops test -o MUL_MAT_ID

In the repository (https://github.com/SynenergyAI/mac-pro-2013-llama-metal): the Metal patches for ai-janitor/llama-metal, the logs, the scripts, the SIMD micro-benchmark and the added test cases.