pitllm

← Home

Model sequences

Released families of date-stamped checkpoints, what they were trained on, and how strictly they avoid the future.

Two questions decide how useful a sequence is for backtesting. Cadence: yearly checkpoints suit slow-moving questions; monthly or quarterly ones are needed for markets and news. Provenance of the first checkpoint: a model trained from scratch on period text is leakage-free by construction, whereas a model adapted from a modern base (BERT, RoBERTa) carries whatever its base had already read. Both kinds are listed; the distinction is noted for each.

190019201940 1960198020002020 HistBERT · decadal · COHA StoriesLM · yearly (64) · American Stories ChronoBERT / ChronoGPT · yearly (26) · web + news Time Machine GPT · yearly (12) · News Crawl + Wikipedia Scaling PiT LMs · monthly · FineWeb DatedGPT · yearly (12) · web text + per-year instructions TimeLMs · quarterly · Twitter bar = span of checkpoints; light = decade steps; dark = monthly or quarterly cadence

Coverage of the seven released sequences. Nothing open covers 1964–1998 at yearly cadence, and nothing before 2013 at monthly cadence.

ChronoBERT / ChronoGPT — Manela lab

He, Lv, Manela & Wu (2025), arXiv:2502.21206. Twenty-six yearly cut-offs from 1999-12-31 to 2024-12-31, each trained only on text that was available at that date. ChronoBERT is a BERT-style encoder; ChronoGPT a GPT-style decoder. Despite the restriction the models match or outperform BERT on standard NLP benchmarks and stay competitive with larger open-weight models. The paper's finance application shows that look-ahead bias is model- and application-specific: a weaker but chronologically clean model can still deliver the predictability that survives once leakage is removed.

A second family, chrono-gpt-instruct (October 2025), instruction-tunes each yearly ChronoGPT so it can be used through a chat interface while remaining free of post-cutoff data.

Scaling Point-in-Time Language Models — Kelly, Malamud, Schwab & Xu

arXiv:2607.11889 (2026). Decoder-only transformers with up to 4 billion parameters trained on one trillion chronologically filtered tokens from FineWeb, producing monthly checkpoints spanning 2013–2024. The headline result is that the well-known performance gap between point-in-time and unconstrained models narrows substantially with scale: on common-sense reasoning and language-understanding benchmarks the models approach leading open-weight models of similar size.

DatedGPT — Yan, Tang, Gao & Jiang

arXiv:2603.11838 (2026). Twelve 1.3-billion-parameter models trained from scratch on roughly 100 billion tokens each, with strict annual cut-offs from 2013 to 2024, plus DatedInstruct, an instruction-tuning set grounded in each year's documents so that post-training cannot reintroduce leakage. Perplexity probes confirm each model's knowledge stops at its year. The paper's distinctive result is a direct estimate of the “lookahead premium”: on 61,000 firm-day news headlines, models whose training covers the outcome period earn 26.4 basis points per standard deviation more than the clean models, significant at the 1% level.

Time Machine GPT (TiMaGPT) — Oxford

Drinkall, Rahimikia, Pierrehumbert & Zohren (2024), Findings of NAACL, arXiv:2404.18543. Twelve yearly GPT-2 models with cut-offs from 2011-12-31 to 2022-12-31, designed to be “nonprognosticative”: uninformed about future facts and future language. Unlike most predecessors the models are trained from scratch on period data rather than adapted from a modern base, and the exact training set for each year is published alongside the weights.

TimeLMs — Cardiff NLP

Loureiro, Barbieri, Neves, Espinosa Anke & Camacho-Collados (2022), ACL demo, arXiv:2202.03829. RoBERTa-base checkpoints trained on Twitter, released quarterly from 2019-12 through 2022-12 (plus a 2022 RoBERTa-large). The accompanying library has a corresponding mode that routes each tweet to the checkpoint trained only up to that tweet's quarter, which is exactly the point-in-time discipline. Built for studying language-model degradation and cultural shift on social media.

StoriesLM — Sarkar

Sarkar (2024), SSRN 4881024. Sixty-four models, one per year from 1900 to 1963, trained on the American Stories dataset of historical U.S. newspapers. The 1900 model is trained on 1900 text; each subsequent model continues the previous year's checkpoint on the next year's articles, so the pretraining window expands sequentially. The only released sequence covering the first half of the twentieth century at yearly cadence.

HistBERT — Qiu & Xu

Qiu & Xu (2022), arXiv:2202.03612. BERT-base continued on the Corpus of Historical American English, one model per decade from the 1900s to the 2000s, built to track diachronic lexical semantic change. Code and training recipe on GitHub.

Also relevant

Gaps. No open sequence covers 1964–1998 at yearly cadence, although Newswire (1878–1977) now supplies the text for one. Nothing before 2013 exists at monthly cadence. No sequence targets a non-English language or a specialist domain such as medicine or law. The plans page costs out ways to fill some of these.