Temporal Leakage in LLM Backtesting: Three different clocks
Leakage is defined against the forecast as-of date, but Assumption 1 treats outcomes resolving after the training cutoff as clean. Those are different boundaries, and news in between can leak.
§2 defines leakage relative to the forecasting as-of date. Assumption 1 then treats an outcome that resolves after the model's training cutoff as clean.
Those are two different boundaries, and a third clock, the outcome resolution date, is in play as well. A forecast is dated January. The model is trained through June. The outcome resolves in December. News published in March can reveal a great deal about December while being unavailable in January, and December's outcome need never have appeared in training at all. Resolution after the training cutoff rules out direct outcome recall. It does not rule out leakage relative to the as-of date.
The boundary and control estimators additionally require continuity or transportability assumptions. A known date, or a matched clean model on its own, does not identify leakage.
The prose after Corollary 10
The corollary carries a zero-mean-shift condition; the prose that follows drops it. With Y ~ Bernoulli(½), no informative bridge data, an original prediction of 0.2 and a changed prediction of 0.5:
The score improves by 0.09 with no leakage signal extracted at all. The improvement is entirely the correction of a mean shift, which the paper's own decomposition handles through the negative shift term. The corollary with both conditions is not refuted.
The convex-pull identity w(2 − w) is correct for the stated mixture model, and there is no gradient error of the kind in the same authors' TEMPO. The problem is the interpretation of what the model and the estimators establish.
Resolution after the training cutoff does not establish cleanliness relative to an earlier as-of date, because intermediate news can leak. Identification also needs the stated continuity and transportability assumptions. The prose after Corollary 10 omits a necessary zero-mean-shift condition.