Simulated Ignorance Fails: A performance gap is not an amount of leakage
The headline 52% is the share of a performance gap left after a prompting intervention. The comparison that produces it changes the model as well as the cutoff, so the number is not an identified leakage quantity.
Simulated ignorance is reported to leave a 52% gap to true ignorance, which reads as a measurement of how much leakage prompting fails to remove.
The 52% is the residual share of a performance gap after the prompting intervention. In §5.1 a comparison between DeepSeek and GPT models on the same questions is described as changing only cutoff status. It changes the model too: architecture, training data, and task-specific capability.
An observed gap can hold leakage plus differences in skill, information recency, and question composition. Similar average performance on some other benchmark imposes no equality on this set. Two models can have errors of +a and −a across two task types and still show the same overall mean.
The paper gives real evidence that the prompts it tested do not erase the observed advantage, which is the useful and widely applicable finding. The cross-model and horizon checks are informative. What is missing is the counterfactual needed to call 52% a causal quantity, and nothing here shows that all retrospective forecasting designs are invalid.
The 52% is a remaining performance gap, not an identified leakage quantity. The cross-model comparison changes architecture, training and task-specific capability alongside cutoff status. The controls are suggestive but do not isolate the causal effect of future information.
← Temporal Leakage in LLM Backtesting · All Leaks Count, Some Count More →