Reviews

Where a paper on the reading list does not support what it claims, and how much of it survives.

The ideas page summarises what each paper claims. This section records what did not hold up on checking. Eight papers have a finding substantial enough for its own page. Each names the exact claim and section, gives the counterexample or the missing assumption, and says what still stands.

A red note means a demonstrable error, a failed inference, or a material contradiction. Ordinary qualifications, for limited generalisability or an unverified priority claim, are folded into the summaries themselves rather than raised here. Three rules of thumb generate most of the findings below: a null experiment is not a universal impossibility proof, a diagnostic is not automatically an identified causal quantity, and a clean output is not evidence that hidden future information went unused.

Paper	Finding	What it comes to
Look-Ahead-Freedom as Temporal Non-Interference The halting reduction does not gate the future value	Proof error	Section 7 claims the constructed pipeline leaks exactly when a simulated machine halts. Under the evaluation rules the paper gives in §6.2, the future value reaches the output in the never-halting case too, so the reduction does not establish what it is used for.
The Geometry of Forgetting Orthogonality does not prove impossibility	Invalid inference	The paper infers that uncertainty-based detectors must be blind to staleness from near-orthogonal probe directions and the failure of the detectors it tested. Near-orthogonality does not support a universal impossibility claim.
Temporal Critique Fine-Tuning Cutoff dependence is compatible with supervised learning	Invalid inference	The paper argues that supervised fine-tuning cannot teach temporal admissibility because admissibility depends on the cutoff. Supervised objectives condition on inputs, and the cutoff can be one of them. The paper's own appendix writes that objective.
TEMPO The gradient alignment claim has a counterexample	Proof error	Proposition 1 asserts that the leakage-mode gradient descends the leakage cost. Its proof changes the weighting distribution without changing the centering means. Two independent witnesses make the inner product positive.
The Memorization Problem Softmax reaches only the interior of the simplex	Mathematical error, repairable	Appendix C.3 claims finite real logits realize every distribution on a finite label set. That fails at the boundary. The error is local and repairable, and the paper's central non-identification result does not depend on it.
Temporal Leakage in LLM Backtesting Three different clocks	Definition mismatch	Leakage is defined against the forecast as-of date, but Assumption 1 treats outcomes resolving after the training cutoff as clean. Those are different boundaries, and news in between can leak.
Simulated Ignorance Fails A performance gap is not an amount of leakage	Identification overclaim	The headline 52% is the share of a performance gap left after a prompting intervention. The comparison that produces it changes the model as well as the cutoff, so the number is not an identified leakage quantity.
All Leaks Count, Some Count More Influence under re-prompting is not internal provenance	Attribution limitation	Shapley-DCLR attributes influence to visible rationale claims by re-prompting on subsets of them. That measures the effect of a later prompt, not what caused the original answer.

Coverage and limits

All 72 paper entries on the ideas page were inventoried. Primary text was available for 68. The broad pass compared each summary against the abstract and conclusions where those could be extracted; selected papers received deeper checks of methods, assumptions and proofs. Four full texts were not reachable: StoriesLM, Sarkar & Vafa, Re(Visiting), and the historical-behavioural-science paper. Those are marked as access gaps and are not treated as verified.

This is a screening audit with targeted proof checks. It is not a line-by-line verification of every paper, and it reproduces no training experiments. “No error identified” is limited to the material inspected.

Per-entry coverage and suggested corrections (72 entries)

The “summary correction” items below have been applied to the ideas page, along with the qualifications for LAP, the fake-date test, alpha decay, FinCAD, divergence decoding, OpenPM, Levy and Total Recall. The four access gaps are now marked as unverified on the entries themselves rather than restated as findings. The list is kept as the record of what was changed and why.

#	Entry and version	Finding / coverage
1	Röttger & Pierrehumbert (2021) · Temporal Adaptation of BERT 2104.08116v2	Screen: summary matches the reported downstream negative result; not a general impossibility.
2	Loureiro et al. (2022) · TimeLMs 2202.03829v2	Screen: the page already explains the within-quarter routing problem.
3	Qiu & Xu (2022) · HistBERT 2202.03612v1	Screen: historical adaptation is distinguished correctly from a clean historical base.
4	Drinkall, Rahimikia, Pierrehumbert & Zohren (2024) · Time Machine GPT 2404.18543v1	Screen: retain the page’s caveats about timestamp validation and clean continuation.
5	Sarkar (2024) · StoriesLM	Access gap: full paper unavailable. The uniqueness claim is unverified.
6	He, Lv, Manela & Wu (2025) · Chronologically Consistent LLMs 2502.21206v3	Screen: the page already explains why cross-model Sharpe differences do not identify leakage.
7	He, Lv, Manela & Wu (2025) · Instruction Tuning Chronologically Consistent LMs 2510.11677v2	Screen: the page correctly distinguishes classifier screening from historical availability.
8	Fittschen, Li, Lippincott & Choshen (2025) · Pretraining for Diachronic Linguistic Change Discovery 2504.05523v3	Summary correction: remove the universal guarantee language; cost comparisons are architecture- and setup-specific.
9	Li et al. (2025) · TiC-LM 2504.02107v3	Screen: 2.6× is a reported compute comparison, not a universal scaling law.
10	Faro, Fan, Alphaidze & Jaggi (2025) · TiMoE 2508.08827v1	Screen: the page already flags leakage through learned routing/co-adaptation.
11	Yan, Tang, Gao & Jiang (2026) · DatedGPT 2603.11838v2	Screen: preserve the distinction between the regression premium and portfolio return.
12	Kelly, Malamud, Schwab & Xu (2026) · Scaling Point-in-Time Language Models 2607.11889v2	Screen: the page already flags the post-training filter’s different criterion.
13	Pilchen, Fabre, Signe Talla & Pérez (2026) · Data Temporality in Pre-training 2605.22769v2	Screen: temporal ordering result is empirical; no general zero-cost guarantee.
14	Luo, Shinnick, Griesshaber & Wang (2026) · TypewriterLM 2606.02991v1	Screen: the page already explains why lexical grounding need not establish semantic provenance.
15	Lopez-Lira & Tang (2023) · Can ChatGPT Forecast Stock Price Movements? 2304.07619v6	Screen: post-cutoff news design supported; inference applies to the evaluated setup.
16	Glasserman & Lin (2023) · Assessing Look-Ahead Bias in GPT Sentiment 2309.17322v1	Summary correction: anonymization’s net benefit does not uniquely decompose distraction and lookahead.
17	Sarkar & Vafa (2024) · Lookahead Bias in Pretrained Language Models	Access gap: full paper unavailable. Candidate biographies alone do not make elections unpredictable; inspect the close-election design before endorsing that claim.
18	Rahimikia & Drinkall (2024) · Re(Visiting) Large Language Models in Finance	Access gap: page admits missing abstract while describing findings. Remove or mark those descriptions as unverified.
19	Ludwig, Mullainathan & Rambachan (2024) · An Applied Econometric Framework 2412.07031v4	Focused check: inspected the basic Bayes-rule and bias decompositions; no algebraic error found there. Universal statements retain their assumptions.
20	Lopez-Lira, Tang & Zhu (2025) · The Memorization Problem 2504.14765v2	Focused math check: finite-softmax reachability error; core non-identification result is narrower and survives.
21	Crane, Karra & Soto (2025) · Total Recall? (Federal Reserve)June 2025 PDF	Focused source check: vintage results supported; within-vintage test is explicitly preliminary and conditional.
22	Paleka, Goel, Geiping & Tramèr (2025) · Pitfalls in Evaluating LM Forecasters 2506.00723v1	Screen: useful concrete evaluation criticism; authors explicitly do not prove all affected performance claims are inflated.
23	Li et al. (2025) · DeepFund 2505.11065v2	Summary correction: live prospective testing is useful, but is not the only valid design; paper also omits execution frictions and has a short sample.
24	Merchant & Levy (2025) · A Fast and Effective Solution to Look-ahead Bias 2512.06607v1	Focused method check: suppression results do not certify absence of all future knowledge.
25	Gao, Jiang & Yan (2025) · Detecting Lookahead Bias in LLM Forecasts 2512.23847v2	Focused proof check: detection relies on exogeneity, conditional moments and monotonicity.
26	Eliseev & Seleznev (2026) · Fake Date Tests 2601.07992v2	Focused assumption check: fake-date rejection is conditional on date invariance.
27	Benhenda (2026) · Look-Ahead-Bench 2601.13770v1	Screen: alpha decay is confounded with regime change and staleness.
28	Kong et al. (2026) · Evaluating LLMs in Finance Requires Explicit Bias ConsiderationEarlier review	Earlier review: taxonomy is proposed, not authoritative or exhaustive; mention frequency is not actual bias prevalence.
29	Zhang, Chen & Stadie (2026) · All Leaks Count, Some Count More 2602.17234v2	Focused metric check: rationale re-prompting attribution is not internal causal provenance.
30	Gao, Jiang & Yan (2026) · Debiasing LLMs by Fine-tuning 2604.02921v2	Screen: evaluates extrapolation bias; do not generalize success to lookahead removal.
31	Levy (2026) · Caution Ahead (Journal of Accounting Research)Wiley, May 2026	Focused source check: restore the code-tool results; correct the Concrete-temperature explanation.
32	Zhang & Stadie (2026) · TEMPOEarlier review	Earlier proof review, counterexample rerun: gradient claim fails; finite-sample gate is not a hard constraint.
33	Zhang, Li, Peng & Chen (2026) · When Alpha Disappears 2605.23959v1	Screen: controlled toggles are useful; violation ranking is specific to tested data/models.
34	Li, Wang & Ma (2026) · FinCAD 2605.24564v2	Focused method/results check: return correction is not identified leakage removal; OOS qualification omitted.
35	Merchant & Levy (2026) · Feature Steering 2606.27199v2	Summary correction: failure to steer selected bias features does not establish unsteerability in general.
36	Fonseca (2026) · Look-Ahead-Freedom as Temporal Non-Interference 2607.04958v1	Focused proof check against HTML and PDF: broken halting reduction and unsupported semantic-decidability frontier.
37	Jia (2026) · HindsightBench 2607.18867v2	Screen: probes audit behavior; no certificate of total parametric cleanliness.
38	Zhang & Stadie (2026) · Temporal Leakage in LLM Backtesting 2608.02985v1	Focused formal check: post-resolution cutoff assumption misses bridge news; corollary prose drops a condition.
39	Cai et al. (2026) · OpenPM 2608.09988v1	Focused scope check: certificate applies to recorded pipeline constraints, not all parametric knowledge.
40	Zhao, Brumbaugh, Wang, Hajishirzi & Smith (2024) · Set the Clock 2402.16797v2	Screen: factual temporal alignment is not a guarantee of historical forecasting validity.
41	Cheng et al. (2024) · Dated Data 2403.12958v2	Summary correction: effective recall boundaries vary; a genuine last-data timestamp can still be a scalar.
42	Park et al. (2025) · Temporal Heads 2502.14258v2	Screen: tested head ablations affect temporal recall; no universal isolation of all date knowledge.
43	Gao et al. (2025) · Can Prompts Rewind Time for LLMs? 2510.02340v2	Screen: failure on tested indirect knowledge is supported; universal failure of every prompt is not proved.
44	Wongchamcharoen & Glasserman (2025) · Do LLMs Understand Chronology? 2511.14214v2	Screen: ordering degradation supported. Exact-match rates also compound ordinary individual errors with length.
45	Pęzik et al. (2025) · LLMLagBench 2511.12116v1	Screen: estimates a behavioral freshness boundary, not an audited last-training timestamp.
46	Li et al. (2026) · Simulated Ignorance Fails 2601.13717v1	Focused design check: residual performance gap is not an identified leakage amount.
47	Ding et al. (2026) · Temporal Critique Fine-Tuning 2605.14636v1	Focused objective/metric check: invalid SFT impossibility argument; QA metric can reward abstention.
48	Asai et al. (2026) · Recall-Based Prompting 2606.05804v1	Summary correction: the tested prompts improve results; no proof they are the best possible prompts.
49	van Adrichem, Bhaskar, Yang & Potts (2026) · Do LMs Consistently Encode the Current Year? 2608.15507v1	Screen: failure of evaluated interventions is supported; no impossibility theorem about joint updating.
50	Lazaridou et al. (2021) · Mind the Gap 2102.01951v2	Summary correction: scale failed to solve drift in the tested models; avoid universal and historical-priority claims.
51	Dhingra et al. (2021) · Time-Aware LMs as Temporal Knowledge Bases 2106.15110v2	Screen: temporal conditioning and refresh results are empirical; no clear summary error identified.
52	Jang et al. (2021) · Continual Knowledge Learning 2110.03215v4	Summary correction: parameter expansion performed best among tested methods; necessity is not established.
53	Jin et al. (2021) · Lifelong Pretraining 2110.08534v3	Screen: reported distillation advantage is scoped to the evaluated streams.
54	Luu et al. (2021) · Time Waits for No One! 2111.07408v2	Screen: summary matches the comparison of continued pretraining and target-period task labels.
55	Agarwal & Nenkova (2021) · Temporal Effects on Pre-trained Models 2111.12790v2	Summary correction: adaptation helped the evaluated settings; not a guarantee for every task or distribution.
56	Zhang & Choi (2023) · Discarding Outdated Facts 2305.14824v3	Screen: duration prediction improves evaluated calibration; not a historical admissibility test.
57	Shin et al. (2025) · TARDIS 2503.18693v2	Screen: the page correctly calls this adaptation rather than leakage removal.
58	Elbadry et al. (2026) · The Geometry of Forgetting 2605.09195v1	Focused logic/method check: empirical orthogonality does not establish universal detector impossibility.
59	Jang et al. (2022) · TemporalWiki 2204.14211v3	Summary correction: reported 12× saving is useful; minimal possible update cost is not proved.
60	Liška et al. (2022) · StreamingQA 2205.11388v1	Summary correction: stale parametric models can limit retrieval systems; retrieval can still improve them.
61	Kasai et al. (2022) · RealTime QA 2207.13332v2	Screen: live QA design and stale-answer observation supported.
62	Park et al. (2024) · ChroKnowledge 2410.09870v3	Screen: temporal benchmark summary broadly supported; priority language is unnecessary.
63	Deng et al. (2024) · NewTerm 2410.20814v1	Screen: the performance drop belongs to the tested term benchmark, not all novel vocabulary.
64	Xian, Cui, Bauer & Abbasi-Asl (2025) · Date-Controlled Tool Use 2503.04188v2	Screen: demonstrates tool-date effects; does not quantify equality of tool and model-cutoff importance.
65	Vladika, Dhaini & Matthes (2025) · Facts Fade Fast 2509.04304v1	Summary correction: all tested models exhibit outdated answers; not all answers from every model are outdated.
66	Yuan, Ding & Vlachos (2025) · Do LMs Update Their Forecasts? 2509.23936v3	Screen: update sizes are compared with human forecasts, which are a reference rather than an oracle posterior.
67	Guan et al. (2026) · TempoMed-Bench 2605.13045v1	Summary correction: observed temporal trend is not a universal linear law of medical knowledge.
68	Zheng, Shao, Ritter & Xu (2026) · Synthetic Worlds 2609.00184v2	Summary correction: fresh synthetic instances reduce memorization risk; published instances can later enter training, and synthetic-to-real transfer needs validation.
69	Dell et al. (2023) · American Stories 2308.12477v1	Summary correction: segmentation improves corpus quality; unsegmented text is not categorically unusable.
70	Silcock, Arora, D'Amico-Wong & Dell (2024) · Newswire 2406.09490v1	Screen: corpus contribution supported. Modern entity metadata needs separate controls if used as training input.
71	Yang, Liu & Wang (2023) · FinGPT 2306.06031v2	Screen: the page correctly says the pipeline is not point-in-time.
72	Varnum, Baumard, Atari & Gray (2024) · Historical LLMs for Behavioral Science	Access gap: full paper unavailable. Period writing is not automatically a representative sample of the period’s population.

Reproducing the numerical checks

reviews/counterexamples.py uses only the Python standard library. It checks the TEMPO gradient, the orthogonality example, the mean-shift example and the missing gate in Fonseca's reduction, and asserts each result. reviews/verify_tempo_prop1.py computes the TEMPO inner product straight from the score functions and reports the dropped covariance term.

This section distinguishes errors in a source paper from overstatements introduced or amplified by the summary on this site. Where a summary was the problem, the summary has been changed rather than the paper blamed.