RESEARCH / LOCAL BENCHMARK PILOT
Real coding decisions, measured locally.
On a 90-task coding-choice pilot, rotating the candidate options selected the passing implementation 89 times. One pass selected it 85 times. The rotated approach took about three times as long.
| Method / Arm | Correct | Accuracy | Wilson 95% CI | Local Median | Local p95 | Holdout ECE |
|---|---|---|---|---|---|---|
Single pass | 85/90 | 94.4% | 87.6%–97.6% | 97.7 ms | 150.9 ms | 5.6% |
Three rotated choices | 89/90 | 98.9% | 94.0%–99.8% | 291.4 ms | 453.5 ms | 1.6% |
Pairwise tournament | 87/90 | 96.7% | 90.7%–98.9% | 485.4 ms | 747.5 ms | 2.57% |
Tournament + always-test probes Secondary diagnostic | 87/90 | 96.7% | 90.7%–98.9% | 571 ms | 829.9 ms | — |
Cross-Quantization Scorecard (Apple M5 Max, 1,440 Forwards / Model)
BF16 down to 2-BitEvery precision level ran 12 forwards per task over all 120 tasks (1,440 forwards) and is scored on the identical 90 held-out evaluation tasks, extracting slot logits with identical prompts:
| Quant Level | Weights Size | Smoke | Single Pass | Cyclic (ℤ₃) | Tournament | Cyclic ECE | Median Single | Median Cyclic | Flips |
|---|---|---|---|---|---|---|---|---|---|
| BF16 Unquantized 16-bit | 7.83 GB | PASS | 85/90 (94.4%) | 89/90 (98.9%) | 88/90 (97.8%) | 0.79% | 98.7 ms | 294.8 ms | 2/90 (2.2%) |
| Q8 MLX affine 8-bit, g64 | 4.16 GB | PASS | 85/90 (94.4%) | 90/90 (100.0%) | 88/90 (97.8%) | 0.57% | 112.7 ms | 341.9 ms | 2/90 (2.2%) |
| Q6 MLX affine 6-bit, g64 | 3.18 GB | PASS | 85/90 (94.4%) | 89/90 (98.9%) | 88/90 (97.8%) | 1.25% | 124.2 ms | 370.7 ms | 1/90 (1.1%) |
| Q4 MLX affine 4-bit, g64 | 2.20 GB | PASS | 85/90 (94.4%) | 89/90 (98.9%) | 87/90 (96.7%) | 1.60% | 97.7 ms | 291.4 ms | 1/90 (1.1%) |
| Q3 MLX affine 3-bit, g64 | 1.72 GB | PASS | 80/90 (88.9%) | 87/90 (96.7%) | 84/90 (93.3%) | 2.45% | 136.5 ms | 371.7 ms | 8/90 (8.9%) |
| Q2 MLX affine 2-bit, g64 | 1.23 GB | FAIL | 32/90 (35.6%) | 20/90 (22.2%) | 24/90 (26.7%) | 11.51% | 112.6 ms | 334.9 ms | 55/90 (61.1%) |
Local Verdict Decision Fleet vs. Cloud Frontier SOTA
Architecture & Economics| Dimension | Frontier Cloud SOTA | Verdict Decision Local | Delta Advantage |
|---|---|---|---|
| Adjudication Mode | Generative LLM (100B–1T+ params); outputs text rationale before stating choice | Direct slot logit scoring (0.8B–35B params); 0 text tokens generated | Zero token bloat; 100% deterministic slot extraction |
| Venue & Privacy | Public cloud endpoints; snippets traverse WAN | Resident Apple Silicon Unified Memory or local CPU/GPU | Pilot ran on local hardware; the hosted API is a separate venue |
| Decision Latency | Not measured in this pilot | 97.7 ms single pass / 291.4 ms cyclic debiased (M5 Max) | Medians from the pilot summary files |
| Positional Option Bias | Significant first/last choice bias in prompt outputs | Algebraically canceled via Z₃ cyclic group rotations | 98.9%–100% accuracy, ECE down to 0.57% |
| Memory Footprint | Not measured in this pilot | 2.20 GB (4B Q4) · 4.16 GB (4B Q8) | Runs alongside IDE on standard dev laptops |
What this means
In this functional discrimination pilot, rotating the options via the cyclic permutation group Z₃ achieved the highest selection accuracy (89/90, 98.9%), correcting five baseline single-pass mistakes while introducing one new error (exact McNemar p = 0.21875). It also reduced empirical expected calibration error (ECE) to 1.60% and algebraically neutralized slot position bias.
Pairwise round-robin tournaments using Bradley-Terry Maximum Likelihood Estimation proved slower (485 ms median) and scored 87/90. Adding an always-probe execution verification diagnostic (running the first unit assertion on all three candidates) added 82 ms median check time without altering final accuracy on this sample.
All evaluated arms exceeded the strict 35 ms local latency target originally envisioned for sub-millisecond micro-triage; single-pass ran at 97.7 ms median, and cyclic ran at 291.4 ms.
Methodology & isolation
- Split & Tasks: 120 total tasks derived from HumanEval-164: 30 calibration tasks and 90 disjoint evaluation tasks. Each task presented three candidate functions: one canonical solution and two AST mutants killed by the original unit tests.
- Label Isolation: The scoring model received only context docstrings and candidate options in letter slots (A/B/C). Gold answers, test suites, and execution outcomes never entered scoring prompts.
- Sandbox Execution: Candidate functions were graded strictly within a deny-default macOS sandbox enforcing strict CPU, memory, and filesystem boundaries.
- Model & Weights: Evaluated on Mapika/decider-4b v2.1 (MLX 4-bit) (MLX affine 4-bit, group size 64), a 4B parameter specialized adjudicator running on Apple M5 Max, 128 GiB unified memory. Weight SHA-256:
d0ba3ca595387103....
Audit receipts & downloads
All data and receipts are versioned and publicly verifiable: