Benchmark walkthrough: how the two Metal kernel evaluation tracks work

1. The big picture: one campaign, two tracks

This is a single capability evaluation campaign of one frozen model, Qwen/Qwen3.5-35B-A3B (pinned revision 59d61f3ce65a6d9863b86d2e96597125219dc754), asking one question. Can it write correct and faster raw Metal Shading Language (MSL) kernels for Apple Silicon? There is no training anywhere in this campaign. No weight updates, no fine-tuning, and no other models in evaluation sessions. Every prompt, generation, and result artifact carries the marker evaluation-only/do-not-train (the DATA_POLICY constant in metal_kernels/records.py), and the runtimes refuse unmarked artifacts.

The campaign is split across two machines.

• A rented NVIDIA GPU host serves the policy model with vLLM behind an OpenAI-compatible Chat Completions API. The port is published only on the remote host's loopback interface, and this Mac reaches it through an SSH local port-forward, so the endpoint seen locally is loopback port 8080. The protocol requires an NVIDIA B300 (see section 4).
• This Mac (Apple M4 Pro, 16 GPU cores, macOS 26.5) does all the Metal work. It compiles every generated kernel at runtime, dispatches it on the local GPU, checks correctness, and times it. The model never touches the Mac, and the Mac never touches the model weights.

And into two evaluation tracks.

2. Track A: Metal-Sci (primary, 12 scientific tasks)

Metal-Sci is a 12-task scientific-compute benchmark (DL4C 2026 paper) hardened for this Qwen-only evaluation. Each task ships a seed kernel in seeds/<task>.metal, a Python task module in metal_kernels/tasks/<task>.py (input generation, dispatch geometry, CPU reference, tolerance, roofline ceiling), three visible problem sizes, and one held-out size. The registry is asserted to be exactly these 12 tasks. tasks.assert_exact_task_registry() runs at import time and at the top of run_benchmark.py.

2.1 Tasks and regimes

Tasks are grouped into regimes, families of structurally distinct GPU and memory bottlenecks. The regimes are non-ordinal. R1 through R8 are categories, not difficulty levels. Never rank, order, or aggregate across regimes as if they formed a scale.

2.2 Visible versus held-out configurations

• Visible split: three in-distribution problem sizes per task (for example heat2d uses 256², 512², 1024²). These are printed in the prompt, and all compiler, correctness, and performance feedback the model ever sees comes from them.
• Held-out split: one auxiliary configuration per task, recorded in the task modules of the code repo. The specific sizes are deliberately not repeated on this public page, so that the held-out property stays intact for later campaigns that reuse them. The held-out configuration is never shown to the model. The prompt explicitly states that "other evaluation configurations are intentionally undisclosed". Held-out evaluation runs only after search, on frozen selected winners.

This is enforced in code, not just in policy. metal_kernels/prompts.py raises (_require_visible) if a held-out CandidateResult would ever enter a prompt or feedback packet, and Task.evaluate_candidate in metal_kernels/task.py rejects any size that belongs to the other split. The held-out gate exists because in-distribution scores alone license confidently-wrong kernels, for example a hard-coded size enumeration that is 10.6x faster on the visible sizes and silently wrong (or O(N²)-slow) at the unseen one.

Scoring: a candidate's visible score is the geometric mean of achieved over roofline-ceiling across the three visible sizes, hard-gated on correctness. Any tolerance failure at any size forces the score to None, with no partial credit.

2.3 The four evaluation conditions

The frozen plan (metal_kernels/metal_sci_plan.py, mirrored in the protocol TOML) defines four staged conditions, run in order and frozen stage by stage.

Selection: one visible winner per independent trajectory (a visible-correct one-shot sample counts as its own one-turn trajectory), chosen by highest interleaved visible task-point speedup, with ties broken by the lexicographically smallest SHA-256 of the normalized source. If no candidate is visible-correct, there is no winner. The checked-in seed is never substituted. Duplicate sources (exact or whitespace-normalized) are never re-executed. A shared source registry reuses the owner's immutable outcome, and a repeat within the same trajectory ends it with policy_stop / repeated_identical_output.

2.4 Paired ABBA interleaved timing

The scored speedup for every candidate is a same-session paired seed and candidate statistic (metal_kernels/paired_timing.py plus metal_sci_runtime.py). The seed and the candidate are compiled together in one fresh worker process and their dispatches are interleaved on one GPU queue, so thermal and scheduler drift hit both sides equally.

1. Schedule: per visible shape, 5 warmups per side, then 15 ABBA cycles (30 retained measurements per side). Each cycle contains two seed and two candidate dispatch samples in a randomized assignment, either seed-candidate-candidate-seed or candidate-seed-seed-candidate. The order is deterministic, derived from the SHA-256 of the canonical JSON of the protocol ID, task, candidate ID, session ID, and an ABBA-order label.
2. Cycle ratio: geometric mean of the two seed GPU durations divided by the geometric mean of the two candidate durations.
3. Per-shape point speedup: geometric mean of the 15 cycle ratios.
4. Task-point speedup: geometric mean of the per-shape speedups. This is the primary metric.
5. Practical win threshold: a selected winner counts as a practical win only at 1.05x or above. Exactly 1.0 is a tie.
6. Confirmation: every held-out-correct selected winner with initial held-out speedup at or above 1.05 is re-timed in a completely new session (new OS worker, Metal device and queue, session ID, and ABBA schedule) and must pass a 10,000-resample stratified paired percentile bootstrap (resampling ABBA cycles within each shape) with the 95% CI lower bound at or above 1.0.
7. Thermal guard: per session, the first-versus-last-third drift of cycle time scales must stay within 3%. An unstable session is sealed as invalid and repeated after cooling, under a new session ID.

Compilation and pipeline construction never sit inside the timed region. A compile-only preflight worker (120 s hard deadline) runs first, and the paired session re-compiles both sides during setup before any timed dispatch. Visible candidate work has a 300 s deadline, and held-out work has 1,200 s. The parent's wait for the Metal lock is excluded from those deadlines. After any timeout or launch failure, a recovery probe (the checked-in saxpy seed at its smallest visible shape, in a fresh worker under the lock) must pass before another model request is issued. If it fails, the run halts as incomplete rather than scoring further candidates.

2.5 The machine-wide Metal lock

2.6 Append-only, hash-chained records

Each run directory under results/ contains three layers of record.

• artifacts/: a content-addressed, write-once store (records.ArtifactStore) holding the system prompt, user prompts, raw generations, reasoning, extracted sources, evaluation results, and paired timing payloads. Files are named by SHA-256, and rewriting an existing hash with different bytes raises.
• terminal_records.jsonl: an append-only JSONL ledger (records.AppendOnlyJSONL) where every attempted model request ends in exactly one terminal row. Each row stores sequence, previous_record_sha256, and a record_sha256 computed over the canonical JSON of the row. The genesis previous-hash is 64 zero hex characters. The chain is re-verified on every append and at the end of every run. Corrections are new appended records, never edits.
• Human-readable copies: numbered prompt, response, reasoning, candidate, and result files, plus history.json, summary.json, model_best.metal, and seed_inclusive_incumbent.metal.

2.7 How to run Track A

All commands run from the metal-sci-kernels checkout. The .venv (Python 3.11) already exists, so recreate it only if needed.

Environment and tests (no model calls, safe any time)

cd /Users/shamane/Documents/metal-sci-kernels
python3.11 -m venv .venv && .venv/bin/pip install -r requirements.txt
.venv/bin/python -m pytest tests/ -q

Seed-only evaluation (visible split, fresh locked worker, executes Metal)

Verifies the checked-in seed kernel compiles, passes correctness, and times on this Mac. The task name is one of the 12 registry names.

.venv/bin/python run_benchmark.py --task heat2d --evaluate-seed-only

Held-out evaluation of an explicit candidate (operator-only, never during search)

Runs an explicit .metal source on the task's held-out split (default deadline 1,200 s). Omit --candidate-source to run the seed itself.

.venv/bin/python run_benchmark.py --task heat2d --evaluate-held-out \
  --candidate-source results/<run-dir>/model_best.metal

Scored generation against the vLLM Qwen server

Requires the SSH tunnel and server from section 4. The transport is explicit: --provider openai-chat is the vLLM path. A seeded one-shot sample is one independent one-turn trajectory, so --iterations 1 is mandatory for that condition (the harness rejects anything else). Run it once per sample with a distinct --trajectory index.

.venv/bin/python run_benchmark.py --task nbody --model qwen35-metal \
  --provider openai-chat --base-url http://127.0.0.1:8080/v1 \
  --condition one_shot --iterations 1 --trajectory 0

# 5-turn repair trajectory
.venv/bin/python run_benchmark.py --task lbm --model qwen35-metal \
  --provider openai-chat --base-url http://127.0.0.1:8080/v1 \
  --condition repair --iterations 5 --trajectory 0

# 10-turn optimization trajectory
.venv/bin/python run_benchmark.py --task fft3d --model qwen35-metal \
  --provider openai-chat --base-url http://127.0.0.1:8080/v1 \
  --condition optimization --iterations 10 --trajectory 0

Defaults already match the frozen protocol: --temperature 0.6, --top-p 0.95, --top-k 20, --thinking-enabled, --max-tokens 32768, --request-timeout 1800 (one HTTP request, zero transport retries), --warmup 5 --measure 30 (equal to 15 ABBA cycles), --evaluation-timeout 300 s visible and 1,200 s held-out, --lock-timeout 300, and --compile-timeout 120. Per-request sampling seeds are derived deterministically from the protocol ID, suite, task, condition, trajectory, and attempt. Results land in results/<task>_<model>_<timestamp>_<run-id>/.

3. Track B: KernelBench-to-Metal (Gimlet-style reconstruction)

Track B asks the same model to replace PyTorch modules with raw Metal kernels, following the methodology Gimlet Labs described in their "AI-generated Metal kernels" blog post, as a transparent reconstruction, because Gimlet published neither its evaluator nor its exact task manifest. The adapter lives in the sibling repo kernelbench-to-metal-eval and executes generated MSL through the public torch.mps.compile_shader API (pinned PyTorch 2.12.0), comparing correct candidates against PyTorch eager on MPS.

3.1 Where the 250 tasks come from, and the frozen 215-task manifest

• Canonical source: ScalingIntelligence/KernelBench v0.1, pinned commit 129232ea3c4cfa05c2dd5473bdb935d6246dc84b (MIT): 100 Level 1 plus 100 Level 2 plus 50 Level 3 tasks, 250 in total. Level 4 is not evaluated. Canonical CUDA KernelBench evaluation is explicitly out of scope.
• Compatibility overlay: one recorded change from Gimlet's fork (gimletlabs/KernelBench at 67f3a48…). Level 1 task 97 loses its hard-coded CUDA and fp16 input construction. It is applied as an overlay with explicit provenance, never by mutating the canonical source identity.
• Exclusions, 31 plus 4 equals 35: 31 MPS-unsupported tasks whose sources contain historically unsupported PyTorch MPS 3D operations (ConvTranspose3d, MaxPool3d, AvgPool3d), identified by static source inspection. And 4 trivial Level 2 tasks (IDs 9, 23, 80, 83) whose outputs are constant or numerically indistinguishable from zero under the evaluation tolerance. For example task 80 takes the max over dim 1 minus its own mean, which is exactly zero, and GELU(0)=0. Task 83 uniquely matches Gimlet's published semantic example.
• Result: a frozen, hash-pinned manifest of 215 tasks, split 91 / 74 / 50 by level, at manifests/gimlet_style_215_reconstruction.json (SHA-256 9369baa8…e0a9fd26), listing every included and excluded task with reason code, evidence class, and per-task source hash.

3.2 The declarative submission ABI and anti-cheat checks

A submission is one JSON object (schema: schemas/submission.schema.json). There is no candidate-authored Python, wrapper class, callback, or eval anywhere. It contains the exact evaluation-only/do-not-train marker, harness-owned identity fields the model must copy verbatim (task ID, expected submission ID, task source SHA-256, input count), one raw metal_source string (which must itself contain the marker comment), harness-owned output and scratch allocation specs, one or more named dispatches with argument bindings, grid, and threadgroup dimensions, and the output names to return. Dimensions and scalars use a small expression tree (const, input_dim, parameter_dim, add, sub, mul, floordiv, ceildiv, min, max). Nothing Turing-complete.

The harness owns everything executable. It compiles the library with torch.mps.compile_shader during prepare, resolves every declared kernel before any candidate execution, allocates all tensors, binds read-only model parameters, dispatches, synchronizes, checks correctness, and times. Compilation and pipeline construction can never occur inside a timed region. Anti-cheat, in layers:

• No CPU or framework fallback: PYTORCH_ENABLE_MPS_FALLBACK=0 is set before PyTorch is imported in the worker. CPU, PyTorch and MLX compute fallback, and oracle or reference calls are static rejections.
• No cached or precomputed outputs: initialized constant tables larger than 64 entries are rejected as precomputed-output risk (64 entries or fewer is allowed for legitimate small FFT and index coefficient tables). 100 unseen randomized inputs per task are the runtime backstop.
• No input mutation: inputs must be bound const device. Chunked SHA-256 digests (bounded transient memory) verify inputs on every correctness call and the full parameter set on a designated case, so a kernel that casts constness away is caught at runtime.
• No output aliasing: returning an input (or aliasing storage into one) is rejected. Only fresh harness allocations are returned.
• Canary pairs: for every correctness input, the harness initializes all outputs and scratch with two distinct dtype-safe canary patterns, dispatches twice, and rejects any output whose digest depends on the canary. This catches unwritten or partially-written outputs and nondeterminism. Canary runs are excluded from timing.
• At least one real Metal dispatch: successful custom-Metal dispatch calls are counted and zero-dispatch candidates are rejected, so "do nothing and hope" cannot pass.

This is defense in depth for generated, non-adversarial benchmark submissions, not a security boundary against hostile native code. Raw GPU code can still hang or fault the device, which is why every candidate runs in a fresh spawned worker with hard deadlines under the shared Metal lock, with post-timeout recovery probes.

3.3 The 1-trajectory, 5-attempt repair loop

• One trajectory per task, at most 5 total attempts. Attempt 1 gets no feedback.
• Sanitized feedback only: later attempts may see only a sanitized extraction or format error, a sanitized compiler error, or a generic correctness-failure notice. No performance numbers, no profiling, no hidden output values, and no CUDA reference, ever.
• Stop on first correct. A fully correct candidate ends the trajectory (stop_on_first_correct). There is no optimization condition in this track.
• API failures are terminal for that logical attempt: zero transparent HTTP retries, and no revision is requested because no extraction, compile, or correctness feedback exists. Every trajectory ends in a required terminal record with stop reason success, budget_exhausted, or infrastructure_abort.

3.4 Correctness and timing

• Correctness is 100 random inputs per candidate, compared elementwise against PyTorch eager on MPS with atol = rtol = 0.01. The rule is that the absolute difference must be at most atol + rtol * abs(reference), and integral or boolean outputs use exact equality. Input seeds are derived deterministically per protocol ID, task ID, and input index. The reference model is initialized with seed 42 in eval mode, and pristine model state plus RNG state are restored before each side of every input.
• Timing is 10 warmups plus 100 paired samples. Wall-clock (time.perf_counter_ns) around explicitly synchronized MPS work, 100 paired eager and candidate samples with a deterministic randomized side order. torch.mps.empty_cache() and synchronization happen outside each timed side. The point statistic is the geometric mean of the 100 paired eager over candidate time ratios.
• Slower-but-correct candidates are kept and reported as-is (retain_slower_correct_candidates: true), never floored to 1.0x, unlike some historical aggregates.

3.5 How to run Track B

Non-Metal validation (compiles, executes, and times nothing on the GPU, safe any time)

cd /Users/shamane/Documents/kernelbench-to-metal-eval
PYTHONPATH=src python3 -m unittest discover -s tests -v
PYTHONPATH=src python3 -m kernelbench_metal.cli validate-submission fixtures/correct.json
PYTHONPATH=src python3 -m kernelbench_metal.cli verify-reconstruction-leads manifests/gimlet_reconstruction_leads.json
PYTHONPATH=src python3 -m kernelbench_metal.cli verify-frozen-reconstruction manifests/gimlet_style_215_reconstruction.json \
  --canonical-checkout /Users/shamane/Documents/KernelBench \
  --overlay-checkout /Users/shamane/Documents/KernelBench-gimlet

Metal execution drivers (these DO run on-device MPS and Metal work)

Three CLI subcommands execute real GPU work in fresh spawn workers under the machine-wide lock with the frozen config/protocol.json deadlines. Only the coordinator that serializes device execution may invoke them.

cd /Users/shamane/Documents/kernelbench-to-metal-eval
# 250-task original-shape eager MPS reference preflight (eligibility evidence)
PYTHONPATH=src python3 -m kernelbench_metal.cli reference-preflight \
  --canonical-checkout /Users/shamane/Documents/KernelBench \
  --overlay-checkout /Users/shamane/Documents/KernelBench-gimlet \
  --output-root results/reference_preflight

# Handwritten raw-MSL fixture execution vs fixtures/expectations.json
PYTHONPATH=src python3 -m kernelbench_metal.cli run-metal-fixtures --output-root results/fixture_runs

# Mac timing-noise and drift calibration (self-paired sessions of the known-good fixture)
PYTHONPATH=src python3 -m kernelbench_metal.cli timing-calibration --output-root results/timing_calibration

4. Hosting the policy model (vLLM plus SSH tunnel)

The policy model is served remotely by vLLM inside Docker on a rented NVIDIA GPU. The protocol requires a B300 with 288 GiB HBM. The fail-closed runbook is hosting/README.md, driven by hosting/gate1_host.py, which refuses wrong GPU classes, tag-only images, busy ports, and unacknowledged launches. Key serving facts, frozen in the protocol:

• vllm serve Qwen/Qwen3.5-35B-A3B at the pinned revision, BF16, no weight quantization, BF16 KV cache, --tensor-parallel-size 1, text-only (--language-model-only), 131,072-token context, --reasoning-parser qwen3, server seed 1234, and initial --gpu-memory-utilization 0.92.
• Served alias: qwen35-metal (--served-model-name qwen35-metal). This is the --model value the Track A CLI sends.
• Publication is loopback-only on the remote host (-p 127.0.0.1:8080:8080, and vLLM's 0.0.0.0 bind exists only inside the container's network namespace). The inference API is never publicly exposed.
• This Mac reaches it via an SSH local port-forward, also loopback-only.

ssh -N -o ServerAliveInterval=30 -o ExitOnForwardFailure=yes \
  -L 8080:127.0.0.1:8080 "${POLICY_SSH_TARGET}"

Smoke-test the tunnel and alias from a second Mac terminal by listing the served models:

curl -fsS http://127.0.0.1:8080/v1/models

A chat-completions smoke request against the qwen35-metal alias should use the protocol sampling settings: temperature 0.6, top_p 0.95, top_k 20, max tokens 4096 for the probe, and thinking enabled through the chat_template_kwargs field (enable_thinking: true). A suitable probe prompt asks for one fenced metal code block containing a valid add-one kernel.

The harness reads the final code only from the first choice's message content. Qwen's parsed thinking arrives separately (vLLM 0.27.x exposes it as message.reasoning, with reasoning_content as a compatibility fallback) and is preserved as an immutable diagnostic artifact. It is never compiled and never fed back into a later model turn. The structured probes in hosting/ (gate1_probe.py smoke, context, sweep, stress, and soak phases, gate1_telemetry.py, gate1_stop.py) produce the hashed Gate-1 evidence.

5. Order of gates and hard safety rules

The campaign advances through gates in strict order. Every gate must pass, and be independently re-checkable from its artifacts, before the next begins.

Hard safety rules that hold in every session:

• One Metal timer at a time. All GPU work in both repos goes through /tmp/qwen35-metal-evaluation.lock. Never bypass, rename, or duplicate it, and never time Metal from two processes concurrently.
• No held-out leakage. Held-out shapes, outputs, and timings never appear in prompts or feedback packets. Held-out evaluation runs only on frozen selected winners, after search.
• No training on any of this data. Every prompt, generation, and result carries evaluation-only/do-not-train. Do not strip the marker, and do not reuse evaluation outputs beyond the protocol. This campaign updates no weights.
• Label results honestly. Track B results are always the "Gimlet-style KernelBench-to-Metal reconstruction", never an exact Gimlet reproduction, never an official CUDA KernelBench score, and never a direct comparison with Gimlet's M4 Max numbers.
• No fake 1.0x, append-only records, and no secrets in git (see section 2.6, and note that secrets.env and *.secrets.env are gitignored). Never commit or push without an explicit user request.

6. FAQ

Why can't an M4 Pro result be compared with Gimlet's M4 Max numbers?

They are different GPUs. This Mac is an M4 Pro with 16 GPU cores and 24 GB unified memory. Gimlet evaluated on an M4 Max, which has substantially more GPU cores and memory bandwidth. Speedups over PyTorch eager MPS are hardware-dependent on both sides of the ratio, because eager MPS kernels and generated kernels scale differently with core count, bandwidth, and thermals, so neither absolute times nor speedup ratios transfer between chips. The protocol records the M4 Pro as a planned deviation, and the reporting rule is explicit: present results as an M4 Pro Gimlet-style reconstruction, never as a head-to-head comparison with Gimlet's published numbers.

Why is pass@k not reported for the KernelBench-to-Metal track?

The protocol forbids it (kernelbench_pass_k_reporting_allowed = false) because the estimator's assumptions do not hold there. The unbiased pass@k formula, one minus the ratio of C(n-c,k) to C(n,k), requires n independent samples. Track B runs exactly one trajectory per task with up to five sequential, feedback-dependent repair attempts that stop at the first success. The attempts are neither independent nor exchangeable, and there is only one trajectory to resample from. Instead, Track B reports attempt 1 as the single-turn view and cumulative attempts 1 through 5 as the multi-turn repair view, with Wilson score intervals. Metal-Sci does report pass@1/4/8/16, but only over genuinely independent units (one-shot samples, or whole repair and optimization trajectories, never individual turns), reporting N/A rather than extrapolating when k exceeds the unit count.

Why is saxpy excluded from the Metal-Sci macro-average?

saxpy is the smoke-test task (y = a·x + y): a trivially memory-bound streaming kernel whose seed already saturates DRAM bandwidth, leaving essentially no headroom for a model to demonstrate optimization capability. It exists to validate the format, compile, dispatch, and timing pipeline, and it doubles as the post-timeout recovery probe. The frozen protocol therefore sets primary_macro_task_count = 11 with primary_macro_excludes = ["saxpy"]. It still runs in the smoke and scored stages, but the primary macro-aggregate covers only the 11 substantive tasks, so a bandwidth-saturated near-1.0x tie neither pads nor dilutes the capability estimate.