Experimental Reasoning Benchmark (ERB)

Prompt

TASK: Diagnose a forecasting model's failure modes from experimental evidence. You are investigating models that forecast partially observed supply networks. Each simulated world contains firms with inventories, shipments, shipping delays, and capacity limits. Models observe sparse, noisy historical reports. They must predict future quantities and the effects of capacity interventions. At evaluation time, the true network and hidden inventories are unavailable. All errors below use the same normalized metric. Lower is better. Unless explicitly stated, models receive neither the true network nor the simulator equations. EXPERIMENTAL EVIDENCE A. Neural baseline. A neural model estimates relationships, compresses observations into entity states, and predicts through a recurrent decoder. Forecast error: 0.131. B. Error by forecast horizon. The baseline's one-step predictions are competitive with a reference model that uses the correct simulator equations. Approximately 98% of the baseline's excess error relative to that reference emerges at horizons 3 and beyond. Predicted intervention responses are systematically too small and too late. C. Explicit bookkeeping. The baseline is modified to maintain exact inventory and shipment bookkeeping. A recurrent neural network determines shipment quantities. Forecast error: 0.166. D. Historical fitting with the recurrent model. The inferred inputs and state of model C are adjusted to better explain the preceding 24 periods, then used to forecast. Forecast error worsens to 0.184. Even before adjustment, replaying the historical window with model C substantially degrades its forecast relative to its original direct prediction. E. Different dynamics and joint inference. The recurrent shipment mechanism is replaced by a small shared learned rule acting on explicit inventories and shipment queues. For each world, the procedure jointly fits relationships, delays, initial states, and historical shipments to the observed reports. The fitting objective includes report mismatch, prior penalties, and penalties for disagreement between historical shipments and the learned rule. Forecast error: 0.068. F. More training data. Model E is trained on four times as many worlds. Forecast error: 0.064. The improvement over 0.068 is not statistically established. G. More inference compute. Model E's learned rule is kept fixed. Per-world fitting increases from one initialization with 1,600 iterations to four initializations with 6,400 iterations each. Forecast error: 0.033. The result replicates at approximately 0.033 on a fresh panel of worlds. H. Predicting fitted parameters directly. An encoder is trained to predict model E's fitted parameters from the observed history. Encoder without subsequent fitting: forecast error 0.359. Encoder followed by reduced-budget fitting: forecast error 0.073, at approximately half the original fitting cost. INTERPRETATION CONSTRAINTS 1. All worlds come from the same simulator family. No real-world transfer has been tested. 2. Model E returns a fitted point estimate, not a posterior distribution. 3. Experiment G changes both the number of initializations and the number of iterations per initialization. 4. The architecture comparisons change multiple components and do not constitute a complete factorial experiment. 5. The evidence does not include direct measurements proving recovery of the true hidden states or causal graph. 6. Absence of a statistically established improvement is not evidence of equivalence. 7. Do not assume unreported experimental results. QUESTION What causal diagnosis best explains the complete pattern of results? Your answer must: 1. State the strongest supported diagnosis and connect it to specific evidence labels. 2. Explain why fitting historical observations could worsen model C while improving model E. Distinguish a plausible mechanism from a demonstrated cause. 3. Explain what the horizon-dependent errors establish and what they do not establish about the initial representation. 4. Identify at least three tempting conclusions that are not justified by the evidence. 5. Design one controlled follow-up experiment that distinguishes sensitivity to initialization from insufficient convergence as explanations for experiment G. Specify what is held fixed, what is varied, how fitted solutions are selected without future information, what is measured, and which outcomes would favor each explanation.