# Real Coding Decisions Local Pilot Methodology

**Date:** October 7, 2026  
**Venue:** Local Apple M5 Max, 128 GiB unified memory  
**Study ID:** `coding-pilot-2026-10-07`  
**Model:** `Mapika/decider-4b v2.1` MLX 4-bit conversion (Qwen3.5-4B-Base derivative)  
**Weight SHA-256:** `d0ba3ca5953871030be574146a6eef2d8073d556041756647753921112afaa7f`  
**Dataset SHA-256:** `2f7994d4be5d3b80a040dde3422d1dd1dda2422536d3dd79409598769dbe87f9`

## Abstract

This study evaluates the performance, latency, and positional bias effects of decision orchestration methods on functional code selection. On a 90-task evaluation split constructed from HumanEval specifications and test-killed AST mutants, three decision methods were evaluated:
1. **Single pass baseline:** A single forward pass reading letter logits (A/B/C).
2. **Cyclic permutation group ($\mathbb{Z}_3$):** Three sequential shifted prompt evaluations whose slot scores are averaged across shifts.
3. **Round-robin pairwise tournament:** Pairwise comparisons resolved with Bradley-Terry Maximum Likelihood Estimation.

## Findings Summary

| Method | Correct | Accuracy | 95% Wilson CI | Median Latency | p95 Latency | Expected Calibration Error |
| :--- | :---: | :---: | :---: | :---: | :---: | :---: |
| Single pass | 85/90 | 94.4% | 87.6–97.6% | 97.7 ms | 150.9 ms | 5.60% |
| Three rotated choices (Cyclic) | 89/90 | 98.9% | 94.0–99.8% | 291.4 ms | 453.5 ms | 1.60% |
| Pairwise tournament | 87/90 | 96.7% | 90.7–98.9% | 485.4 ms | 747.5 ms | 2.57% |
| Tournament + always-test probes* | 87/90 | 96.7% | 90.7–98.9% | 571.0 ms | 829.9 ms | — |

*\*Secondary diagnostic added after the primary experiment. Latency reflects combined inference and sandbox probe execution.*

## Evaluation Boundaries & Scope

- **Warm Local Model:** Tests were executed locally on Apple Silicon using native MLX inference. Network latency, cloud cold boots, and concurrency overhead are excluded.
- **Gold-Label Isolation:** The decision model only received function signatures, docstrings, and candidate code implementations. Test suites and execution results were strictly isolated.
- **Sandbox Execution:** All verification was performed inside a deny-default macOS sandbox enforcing memory, CPU, and filesystem/network isolation.
- **Public Pretraining Caveat:** While evaluation tasks were held out from calibration, HumanEval is a public benchmark; pretraining contamination cannot be ruled out.