the question
Can a small measured profile of mathematical operations predict performance on unseen compositions? That is a testable, bounded version of the proposed account of model differences.
A complete state-dependent description of operations determines their composed behavior under a common controller. A few accuracy and latency averages do not. Equal one-step error rates can hide shared errors, cancellation and different strategies.
the first instrument
Seed 1729 generates six isolated add, subtract and multiply probes; six unseen compositions across affine recurrence, triangle determinant and cubic differentiation; and twelve paired encoding probes. Each task supplies its mathematical state. Whole-task modes start a fresh transcript for each task; controlled chains start a fresh transcript for each primitive.
All answers are exact integers. Absolute and relative tolerances are zero. Binary64 interpreter arithmetic rejects every intermediate outside the safe integer domain. Independent BigInt checks use a recurrence closed form, shoelace determinant and expanded derivative coefficients.
three whole-task conditions
Model only: one decimal integer. Supplied program: a correct reference program, testing the interpreter. Model program: one model-authored arithmetic tree, parsed and executed once. This last condition measures program construction plus delegation; there is no feedback continuation or repair round. Its prompt supplies three fixed add/sub/mul syntax demonstrations with toy registers.
{"op":"mul","args":[{"ref":"a"},{"ref":"b"}]}
The language has add, sub and mul; its leaves name supplied registers. It has no literals, loops, imports, property access or generated JavaScript. Limits: 16,384 output bytes, 127 tree nodes, depth 16, 63 operations, 16 registers, one tool call and zero retries. Model generation: at most 1,024 input tokens, 256 output tokens and 60 seconds. A terminated worker must be loaded again.
A model can finish generating text and still fail the requested format. Prose rejected as invalid JSON is a parsing failure; no interpreter work occurred. The exact expected answer is computed after the attempt stops, with no correction sent back. Raw output, token use and the generation finish reason stay in the trace.
Program prompts are versioned as register-program-examples/2; integer prompts as integer-answer/1. Earlier unversioned observations keep their original identity. A prompt revision changes the measurement condition; it does not establish improved model performance.
controlled primitive chains
Model chain: the controller requests one primitive at a time and passes each actual parsed output into the next operation. It supports isolated add, sub and mul plus affine recurrences: twenty of the default twenty-four tasks. Each recurrence update is one multiplication and one addition: four or six calls for the supplied recurrences. Geometry and differentiation remain whole-task conditions. Unsupported routes make no model calls.
A wrong but valid integer continues. Invalid output, a budget limit or cancellation stops the route. The verifier runs after completion or halt; it never sends the model an answer, correction or repair instruction. Local correctness checks each operation on its actual operands. Final correctness checks the task's requested result. A later error can cancel an earlier error, so all-step correctness and final correctness stay separate.
The raw trace retains each prompt, completion, routed value, cost and failure, plus the first local error. A missing step after a halt is unobserved. This is an external computation route, not a view inside the model.
Limits: six calls and zero retries; 512 input tokens, 32 output tokens, 1,024 output bytes and 15 seconds per call; 3,072 input tokens, 256 output tokens and 90 seconds for the route. Requested token caps are reserved before each call; actual token usage is a separate measurement and stays unavailable when the adapter cannot report it. No interpreter tool is called during the route.
what a cost means
Construction, generation wall time, parsing, interpreter execution, verification and total wall time are exported separately. Actual tokenizer IDs count input and output tokens. Download and setup are separate session measurements; compilation cannot be isolated and is null. FLOPs are unavailable, not zero. The reference tree describes the task and the controlled route, never hidden model work or optimal mathematical work. Controlled costs aggregate the actual requested primitive calls; their additional prompts make this a distinct protocol.
A measured zero can mean a phase was shorter than the browser clock's resolution. These small interpreter timings are not precise microbenchmarks.
prediction, before observation
Measure and freeze the primitive profile, then run held-out instances. The first predictor multiplies primitive success rates and adds measured generation-plus-execution costs according to the reference tree. Matching inputs, independent step errors and stationary costs are unverified assumptions.
The product predicts all-step correctness under those assumptions. Its use as a final-answer probability is explicitly a naive proxy: errors can cancel. The export retains that proxy's Brier score and cost error. Controlled chains also expose all-step correctness for a separate score; whole-task internal steps remain unobserved. Small calibration cells have broad descriptive Wilson intervals; they do not establish equivalence or a universal account.
Manual exploration stays in the journal. A profile made after seeing its held-out instances is marked as exploratory. Each held-out task is scored only once per frozen profile; interrupted sweeps can resume.
Whole-task generation and controlled primitive chains are separate conditions. Broader controlled families, a state-aware predictor, model-identity residual testing and a confirmatory model comparison remain future work. A failed fixed protocol is not a proof of inability under every protocol.
reproduction
The seed regenerates inputs. The raw JSON retains every attempted trial, prompts, completions, interpreter steps, failure status, budgets and model provenance. Greedy decoding has no supported sampling seed; hardware and runtime scheduling can change results. Loading a model requires explicit consent and never silently changes the backend.
Supplied-program controls and test fixtures are not model evidence. No model results are bundled into this interface. Unsupported model loads remain exportable diagnostics. The formal proof, experiment specification and source review accompany the implementation outside the served experience.