Final evaluation report
Qwen3.5-35B-A3B Metal-kernel capability evaluation
One frozen model was asked to write correct and faster Metal Shading Language kernels for Apple Silicon. This page is the final report of that campaign, restated from the immutable evidence in the metal-sci-kernels repository. No training, no weight updates, and no other model was called.
Protocol qwen35-metal-eval-v1, frozen, SHA-256 437435375fb760851237cd42fb02392908ee478ef67cc01447a17851d315ff0c. Experiment identity qwen35-metal-20260817T120558Z. Model Qwen/Qwen3.5-35B-A3B, revision 59d61f3ce65a6d9863b86d2e96597125219dc754, BF16, no quantization. Evaluator machine: Apple M4 Pro.
Recommendation: direct private-task RL pilot, conditional on three fixes landing first
Both preregistered decision criteria are met, and the second is met by a wide margin. Reward signal is not the bottleneck. Generation reliability and the measurement apparatus are.
1. The decision
The preregistered decision rule asks for two things. First, at least four nontrivial tasks with fully-correct improving candidates. Second, a group size that yields mixed rewards in at least half of sampled groups. Both are met.
| Preregistered criterion | Observed | Verdict |
|---|---|---|
| At least 4 nontrivial tasks with correct improving candidates | 5 tasks: heat2d, ising, lbm, morton, nbody | Met |
| Improvement not concentrated in one task | 3 tasks clear 1.05x: heat2d, lbm, morton | Met |
| Mixed-reward groups in at least 50% of samples | 99.9% at G=16, 97.1% at G=8 | Met, decisively |
The pooled visible-correct rate is 19/53 = 35.9% (Wilson 95% interval 24.3% to 49.3%). The plan's feared failure scenario was a success rate near 1%, where only about 15% of groups carry any gradient. At 35.9% essentially every group of 8 or more contains both successes and failures. Even the pessimistic end of the confidence interval keeps group informativeness above 94% at G=16.
Three fixes must land before training compute is spent.
- The accounting defect (section 8). A non-finite value in a candidate result crashes the harness before the terminal row is written. A model request is consumed and vanishes from the ledger. This is a blocker because it silently shrinks denominators.
- The non-termination pathology (section 5.3). 45% of requests produced no extractable source, almost all of them degenerate reasoning loops that consumed the entire 32,768-token budget. In a grouped method these are full-cost rollouts returning zero signal.
- The confirmation and held-out evidence (sections 6 and 10). The
headline
mortonresult is measured but not yet formally confirmed, and no held-out evaluation has run.
Bootstrap-first is not indicated, because full-correct improvement is neither sparse nor confined to a single task. Redesign or stop is not indicated, because extraction and compilation succeed often, and the observed failures are mechanical dialect errors rather than conceptual ones.
2. What was actually run
| Gate | Status | Evidence |
|---|---|---|
| 0, Freeze | Sealed | Protocol frozen with 215/35 KernelBench task IDs embedded (gate0_manifest.json) |
| 1, Host | Passed | 1000/1000 stress, 1344/1344 soak, 100/100 tunnel verify |
| 2, Harden | Passed | Failure-class fixtures, hang plus recovery, end-to-end pipeline validation |
| 3, Calibrate | Sealed | 12/12 seeds visible twice and held-out, per-task noise floors measured |
| 4, Metal-Sci | Smoke frozen, one-shot partial | 12 smoke samples frozen, 53 one-shot records |
| 5, KernelBench | Validated, not scored | 3-task slice, canonical 215-task run not started |
Conditions C (five-turn repair) and D (ten-turn optimization) were not run. The one-shot stage reached 53 of 384 planned requests (13.8%) before the session time budget expired, so it is partial and was never frozen. Every number below is labelled accordingly.
3. Gate 1: hosting
The originally supplied B200 endpoint was destroyed when provider credit lapsed. The operator supplied a replacement NVIDIA H200 NVL, which the execution plan names as the documented fallback GPU. This is recorded machine-readably as required_gpu_satisfied = false.
| Phase | Requests | Succeeded | Generated tok/s | p50 / p95 latency |
|---|---|---|---|---|
| Stress | 1000 | 1000 | 1511 | 51.9s / 87.3s |
| Soak (3628s) | 1344 | 1344 | 1512 | 51.6s / 88.8s |
| Tunnel verify | 100 | 100 | none recorded | none recorded |
Zero OOM events, zero engine restarts, zero alias mismatches, and 5,484,787 tokens generated in the soak. Serving config: BF16 weights and KV cache, 131,072-token context, gpu-memory-utilization 0.92, max-num-seqs 32, tensor-parallel 1, qwen3 reasoning parser, seed 1234, bound to remote loopback and reached over an SSH tunnel.
Output-budget finding. Both load phases used the probe's 8192-token ceiling. 81 of 1000 stress responses hit finish_reason=length at exactly 8192 tokens. Among responses that ended naturally, 913 of 919 (99.3%) contained a complete Metal fence. This is probe truncation, not instability, and it is the empirical basis for the frozen 32,768-token scored ceiling.
4. Gate 3: calibration, and why it changes every result
This is the most consequential methodological result of the campaign. Timing the identical seed kernel against itself must yield 1.0. Measuring how far it actually deviates gives each task a credible speedup floor. Below that value, a measured speedup is indistinguishable from noise on this machine.
| Task | Sealed self-comparison | Credible floor | Sessions run | Sealed invalid |
|---|---|---|---|---|
hmc | 0.9999 | 1.0005 | 3 | 1 |
nbody | 1.0011 | 1.0028 | 4 | 3 |
heat2d | 0.9993 | 1.0050 | 4 | 3 |
morton | 1.0024 | 1.0067 | 2 | 0 |
lj | 1.0008 | 1.0069 | 2 | 1 |
adi3d | 1.0019 | 1.0070 | 2 | 0 |
lbm | 0.9969 | 1.0071 | 6 | 4 |
ising | 1.0025 | 1.0075 | 2 | 0 |
wave3d | 0.9989 | 1.0081 | 3 | 2 |
gradshaf | 0.9952 | 1.0103 | 3 | 2 |
saxpy | 1.0066 | 1.0199 | 3 | 2 |
fft3d | 1.0099 | 1.0448 | 2 | 0 |
36 calibration sessions were run, and 18 were sealed invalid. Seven small-kernel tasks needed warmup and measurement counts raised from 5/30 to as high as 200/400 before drift fell inside the 0.03 limit. Kernels under 100 microseconds are dominated by dispatch overhead at the default counts.
Two consequences that materially change interpretation.
lbm initially beat itself by 1.06% with a clean drift score. Had that session
been sealed, any genuine 1% to 2% lbm result would have been meaningless. It
took six sessions to obtain a valid one. And fft3d's floor is
1.0448, so the protocol's own 1.05x practical-win threshold sits barely
above noise for that task. Any borderline fft3d win must be treated as
unresolved.
5. Gate 4: Metal-Sci results
5.1 Format smoke: complete and frozen
12 independent samples, 3 tasks with 4 samples each, at the 8192-token ceiling. Stage freeze 6fb2a4978c817a17….
| Task | Correct | Speedups | Nominal above 1.0 | Floor | Credible |
|---|---|---|---|---|---|
saxpy | 4/4 | 1.0153, 0.9974, 0.9961, 0.9955 | 1 | 1.0199 | 0 |
heat2d | 4/4 | 1.0001, 0.9678, 0.9475, 0.9406 | 1 | 1.0050 | 0 |
fft3d | 0/4 | none | 0 | 1.0448 | 0 |
Two rows carry the valid_improvement status, so a naive reading
reports "2 of 12 improvements". Both sit inside their task's own noise band
(saxpy 1.0153 against a 1.0199 floor, heat2d 1.0001 against
1.0050). The frozen smoke stage contains zero credible speedups. This is
exactly why Gate 3 had to pass before any scoring.
5.2 Seeded one-shot: partial, not frozen
54 model requests issued, 53 terminal records written, 1 lost to the harness defect in section 8. Rates below use the 53 records. The lost request is disclosed, not quietly dropped.
| Regime | Task | n | Extract | Compile | Correct | Speedups | Floor | Credible |
|---|---|---|---|---|---|---|---|---|
| Smoke / control | saxpy | 5 | 4 | 4 | 4 | 1.0066, 1.0065, 1.0017, 0.9961 | 1.0199 | 0 |
| R1 regular stencils | heat2d | 5 | 4 | 4 | 4 | 1.0580, 1.0434, 0.9914, 0.9354 | 1.0050 | 2 |
| R1 regular stencils | wave3d | 5 | 3 | 0 | 0 | none | 1.0081 | 0 |
| R2 compute-bound | hmc | 5 | 1 | 0 | 0 | none | 1.0005 | 0 |
| R2 compute-bound | nbody | 5 | 4 | 4 | 3 | 1.0210, 1.0044, 0.9962 | 1.0028 | 2 |
| R3 multi-field | ising | 4 | 3 | 3 | 3 | 1.0127, 1.0113, 0.9996 | 1.0075 | 2 |
| R3 multi-field | lbm | 4 | 1 | 1 | 1 | 1.0676 | 1.0071 | 1 |
| R4 irregular / atomics | lj | 4 | 2 | 0 | 0 | none | 1.0069 | 0 |
| R5 multi-kernel | gradshaf | 4 | 3 | 3 | 3 | 1.0094, 1.0037, 0.9885 | 1.0103 | 0 |
| R6 butterfly | fft3d | 4 | 1 | 0 | 0 | none | 1.0448 | 0 |
| R7 wavefront | adi3d | 4 | 1 | 0 | 0 | none | 1.0070 | 0 |
| R8 bit-permutation | morton | 4 | 2 | 2 | 1 | 3.2362 | 1.0067 | 1 |
Regimes are non-ordinal. R8 is not "harder" than R1, and regimes must
never be ranked or aggregated as if ordered. saxpy is excluded from the primary
11-task macro-average as a bandwidth-bound control.
Five of twelve tasks produced no correct candidate at all:
adi3d, fft3d, hmc, lj, wave3d.
For adi3d, fft3d and hmc, only 1 of 4 or 5 responses
even yielded extractable source, so the failure is upstream of kernel writing.
pass@k is reported only where the independent-unit count supports it. With n = 4 or 5 per task that means pass@1 and pass@4 only. pass@8 and pass@16 are null, never extrapolated.
5.3 The dominant failure mode: non-termination
24 of 53 one-shot responses (45%) produced no extractable source. 23 of
them (43% of all requests) returned no final answer at all, every one consuming the full
32,768-token budget. Inspection of one saxpy trace found 1253
substantial reasoning lines but only 68 unique, with a single sentence repeated
183 times. The model concluded the seed was already near-optimal and then failed to
stop.
This is not an inadequate budget. saxpy scored 4/4 at the
8192-token smoke ceiling and then looped to the wall at 32,768. More
headroom bought more repetition.
Confound, stated plainly. The smoke and one-shot conditions differ in two ways: the output ceiling (8192 versus 32768) and whether the prompt shows the incumbent seed to improve. This data cannot separate them. The clean ablation is not in the frozen protocol and was not added mid-run. Reported as a measured association, not a cause.
No sampling parameter was changed in response. repetition_penalty is frozen at 1.0 per Qwen's own precise-coding recipe, and altering it now would be a post-hoc change invalidating the run. A repetition penalty and a reasoning-length budget are recommended for a future run only.
6. The headline result: morton
On the R8 bit-permutation task the model produced a candidate measuring 3.236x against the seed in the scored stage, passing all three visible sizes with max-abs errors of 1.8e-07 to 3.0e-07 (float32 rounding).
The mechanism is a genuine algorithmic improvement, not a measurement artifact. The seed decodes each Morton code into (x, y, z) coordinates and re-encodes its neighbours. The candidate stays in Morton bit-space. It derives per-axis masks from logN, extracts each axis's interleaved bits with a single AND, and detects the Dirichlet boundary by testing those bits against zero and against the axis mask, removing the decode and encode round trip entirely. The candidate is smaller than the seed, at 72 lines against 98.
| Confirmation session | Point speedup | Bootstrap 95% CI | Drift | Verdict |
|---|---|---|---|---|
confirm01 | 3.4669 | [3.4247, 3.5107] | 0.0916 | sealed invalid |
confirm02 | 3.1958 | [3.1713, 3.2237] | 0.1464 | sealed invalid |
confirm03 | 3.4118 | [3.3941, 3.4278] | 0.0661 | sealed invalid |
confirm04 | 3.4057 | [3.3788, 3.4317] | 0.3466 | sealed invalid |
Reproduced four times, formally unconfirmed. Four independent fresh sessions measured 3.4669, 3.1958, 3.4118 and 3.4057 (mean 3.370, standard deviation 0.119, 8.0% spread). Every bootstrap interval sits far above 1.0 and every interval is narrower than 0.09. Yet all four sessions were sealed invalid by the thermal-drift stability gate, so under the frozen protocol no formal confirmation exists.
The gate itself appears miscalibrated for this regime, and the evidence is specific. Cooling the machine for 25 minutes before the fourth session produced the worst drift of the four (0.3466), which falsifies the thermal explanation. The same task's seed-versus-seed calibration on the same machine drifted 0.0017, three orders of magnitude less. Drift is computed on the geometric mean of all four samples in a cycle, mixing the seed with a candidate roughly 3.4x faster. That mixture's absolute scale is sensitive to GPU clock state, while the ratio it exists to protect is not. The gate was designed and validated on seed-versus-seed sessions, where both sides have identical duration, and it had never been exercised on a large-speedup pairing.
The rule was not overridden, waived, or recomputed to obtain a pass. The recommended amendment, for a new run identity, is to compute the stability statistic on the per-cycle ratio series, or per side separately, rather than on the mixed per-cycle absolute scale. A gate meant to detect an unstable machine should be invariant to how different the two sides' durations are. It must then be re-validated on both seed-versus-seed and large-speedup pairings.
Held-out evaluation has still not run, so generalization to the sealed configuration is unknown.
7. Gate 5: KernelBench-to-Metal (validation only)
The canonical 215-task run was not started. A 3-task validation slice ran to prove the pipeline, producing 7 attempts before being capped to protect the primary track.
| Task | Attempt 1 | Attempt 2 | Attempt 3 | Outcome |
|---|---|---|---|---|
level1/2 | correct | not reached | not reached | correct, slower than eager |
level1/1 | policy rejection | compile failure | correct | recovered by repair |
level2/1 | policy rejection | compile failure | compile failure | capped |
The ABI is satisfiable and the repair loop works. 2 of 3 tasks reached full correctness against PyTorch eager MPS on 100 random inputs at atol=rtol=0.01, and correct-but-slower candidates were retained rather than floored to baseline.
Blocking finding for the canonical run. The harness rejects submissions whose kernel [[buffer(N)]] indexes are not contiguous and positionally matched to the dispatch plan (plan.py:338), but that rule appears nowhere in the 7664-character prompt or the JSON schema. "buffer" occurs once in the prompt and zero times in the schema. Attempt 1 is the single-turn metric and receives no feedback, while the repair feedback does state the rule. Two of three attempt-1 submissions died on it and both continuing tasks then progressed. This would systematically depress the single-turn score for a harness reason and bias the very single-turn-versus-multi-turn comparison the track exists to report. The prompt must state the contract before the canonical run, which requires a new run identity. That is legitimate, since no canonical scored generation has begun.
Honest qualification: in this instance the model also erred independently, declaring four kernel buffers against five plan arguments. The under-specification stands on its own, because a correctly-sized but differently-ordered argument set would fail for an undocumented reason.
Reference preflight, partial. 38 of 250 tasks executed before being
paused: 35 ok, 1 reference timeout, and 2 host-capacity MPS out-of-memory
failures (level1/35 and level1/38, one allocating 21 GiB
on a 24 GiB machine). These are hardware-capacity exclusions, categorically different from
the reconstruction's MPS-unsupported exclusions, and must be reported separately. Gimlet
used an M4 Max, and this machine is an M4 Pro.
Timing noise floor. Ten self-paired calibration sessions gave a geomean self-ratio of 0.9876 where truth is exactly 1.0, with session geomeans spanning plus or minus 4.2%. Speedup claims below roughly 1.05x on this track are not separable from noise.
8. Defects and deviations
8.1 Accounting defect (blocker)
A non-finite float (NaN or Inf) in a candidate result, expected from a diverging kernel, makes canonical_json raise inside the harness's artifact write, before the terminal row is appended. The request is consumed and lost. Observed once in 54 one-shot units (oneshot-lbm-1, traceback captured). The defect file recorded a second suspect mid-run, but that unit was still generating at the time and later completed normally, so the final reconciliation counts exactly one loss. The protocol requires non-finite output to be recorded as a failed candidate, not lost.
Not patched during the run: Gates 0 and 3 are sealed and both bind the evaluator code hash, so editing the evaluator mid-stage would invalidate those seals. The fix plus its regression test is queued as a blocker requiring a new run identity.
8.2 Protocol deviations (13 recorded)
Principal ones: the required B300 was unavailable and the plan's documented H200 fallback was used (required_gpu_satisfied = false). vLLM runs under the platform's supervisor rather than the runbook's nested Docker. A second 2xH100 instance contributed hosting load evidence only, with no generation attributed to it. Small-kernel calibration counts were raised per the runbook's provision. One labelled pre-freeze pipeline-validation trajectory was run and is excluded from every scored denominator. The Gimlet 31+4 manifest is a transparent reconstruction, never an exact reproduction.
8.3 Limitations
Partial one-shot stage (13.8% of planned requests), never frozen. No repair or optimization condition. No held-out evaluation. No formally valid confirmation. The evaluator is an M4 Pro, not Gimlet's M4 Max, so runtimes are not comparable. Results pin to this machine and software stack and must not be generalized to all Apple GPUs.
9. Reproducing this
All commands run against the metal-sci-kernels repository and its sibling kernelbench-to-metal-eval. This section is command listings only.
cd /Users/shamane/Documents/metal-sci-kernels
.venv/bin/python -m pytest tests/ -q # 70 tests
.venv/bin/python -m unittest discover -s evaluation_analysis/tests # 30 tests
.venv/bin/python -m evaluation_analysis protocol \
evaluation_protocol/qwen35_metal_eval_v1.toml # ok=true, no blockers
RUN=runs/qwen35-metal-20260817T120558Z
.venv/bin/python $RUN/gate23_driver.py seed-visible # 12 seeds x2, fresh workers
.venv/bin/python $RUN/gate23_driver.py seed-heldout # 12 seeds, held-out sizes
.venv/bin/python $RUN/gate23_driver.py self-calibration --sessions 1
.venv/bin/python $RUN/gate23_driver.py gate2-fixtures # bad kernels + hang recovery
.venv/bin/python $RUN/collect_report.py # rebuild the report dataset
.venv/bin/python $RUN/confirm_winner.py --task morton \
--candidate <winner>.metal --session-suffix confirmNN # fresh-session confirmation
cd /Users/shamane/Documents/kernelbench-to-metal-eval
PYTHONPATH=src python3 -m unittest discover -s tests # 107 tests
PYTHONPATH=src python3 -m kernelbench_metal.cli reference-preflight \
--canonical-checkout /Users/shamane/Documents/KernelBench \
--overlay-checkout /Users/shamane/Documents/KernelBench-gimlet \
--output-root results/reference_preflight_full --tasks "$(cat /tmp/preflight_remaining.txt)"10. Work the report leaves queued, in priority order
- Fix the non-finite accounting defect and add its regression test (blocker, new run identity).
- Amend and re-validate the confirmation stability gate (section 6), then formally
confirm
mortonand run held-out evaluation on it. Four sessions already reproduce the effect above 3.19x. - State the buffer-index contract in the KernelBench prompt before the canonical 215-task run.
- Complete the one-shot stage to 32 samples per task, then run the repair and optimization conditions.
- Finish the 250-task reference preflight (212 remaining) on an otherwise idle Mac.
- Investigate the non-termination pathology with a proper ablation separating the output ceiling from the seed-in-prompt framing.
The H200 host that served the campaign is the canonical endpoint and stays up for the confirmation and held-out work. The 2xH100 instance contributed only hosting load evidence and is safe to shut down.