← Home

Model sequences

Released families of date-stamped checkpoints, what they were trained on, and how strictly they avoid the future.

Two things decide how useful a sequence is for backtesting. Cadence: yearly checkpoints suit slow-moving questions; monthly or quarterly ones are needed for markets and news. What the cut-off claim actually rests on: the families below support that claim to three quite different degrees, and conflating them is the main way these models get oversold.

• Historical adaptation. A modern base (BERT, RoBERTa) continued on period text. The period data shifts vocabulary and style, but the base has already read the future, so this supports claims about historical language, not about knowledge available at a date.
• Restricted pretraining. The pretraining corpus is filtered to text published before the cut-off. This is the strong form. It does not require retraining from scratch every year: continuing a genuinely clean earlier checkpoint preserves chronological consistency, so annual restarts are a convenience rather than a requirement. Released corpora make the claim auditable, but open data by itself does not prove the timestamps are right or the selection unbiased. OCR errors and misdated reprints put future text in nominally historical corpora.
• Verified post-training. Instruction tuning or alignment that is itself constrained and checked. Every instruct variant below rests on a screening or grounding procedure, not a proof, and the procedures differ in what they actually test. Each entry says which.

One further distinction matters for any historical claim made with these models. Applying modern methods to historically available information is one counterfactual; reproducing a strategy that could actually have been deployed at the time, with the tools, data feeds and costs of that period, is another. Both are useful, and the sequences below serve the first far better than the second.

ChronoBERT / ChronoGPT — Manela lab

He, Lv, Manela & Wu (2025), arXiv:2502.21206. Twenty-six yearly cut-offs from 1999-12-31 to 2024-12-31, each trained only on text that was available at that date. ChronoBERT is a BERT-style encoder; ChronoGPT a GPT-style decoder. Despite the restriction the models match or outperform BERT on standard NLP benchmarks and stay competitive with larger open-weight models. The paper's finance application shows that a chronologically restricted pipeline can still deliver useful return predictability, which makes these models a practical alternative to a contaminated one and a control to compare a contaminated one against. The gap between the clean pipeline and a larger modern model is not an identified estimate of look-ahead bias: that comparison also changes architecture, corpus, training budget, language competence, and the interaction with the downstream predictor, and offsetting effects can leave two Sharpe ratios looking similar for unrelated reasons. Quantitative attribution of leakage needs a controlled comparison, with its remaining assumptions stated.

A second family, chrono-gpt-instruct (October 2025), instruction-tunes each yearly ChronoGPT for chat use. The instruction data are not historical documents. The three curriculum stages draw on a from-scratch tutorial set, a GPT-3 self-instruct corpus, and AllenAI's Tulu-3 SFT mixture, all of them modern. Each candidate pair is then screened by a GPT-4.1 classifier and retained only when labelled “knowledge available pre-2000” with sufficient confidence. That is classifier-based anachronism screening with empirical validation, which is a real precaution but does not establish that every retained example contains only historically available information. The authors are explicit about it: the classifier “may misidentify whether certain prompt-response pairs reflect knowledge from before or after the year 2000”. So the instruct models' forecasts are a conservative lower bound only to the extent the screen held, and the “leakage-free” label belongs to the pretraining stage rather than to the whole pipeline.

• Weights: Hugging Face collection — manelalab/chrono-bert-v1-YYYY1231, chrono-gpt-v1-YYYY1231, chrono-gpt-instruct-v1-YYYY1231 (78 repos in total).
• Pretraining: restricted to text published before each cut-off. restricted pretraining
• Post-training (instruct variants): modern instruction data screened by an LLM classifier. screened, not proven
• Try it: the ChronoGPT notebook loads any cut-off and samples from it.

Scaling Point-in-Time Language Models — Kelly, Malamud, Schwab & Xu

arXiv:2607.11889 (2026). Decoder-only transformers with up to 4 billion parameters trained on one trillion chronologically filtered tokens from FineWeb, producing monthly checkpoints spanning 2013–2024. The headline result is that scale substantially narrows the measured benchmark gap between point-in-time and unconstrained models: on common-sense reasoning and language-understanding benchmarks the models approach leading open-weight models of similar size. The full pretraining run took roughly 28,000 GPU-hours, which makes the affordability claim concrete rather than rhetorical.

The two stages need separate verdicts. The pretraining is filtered by crawl date and is the solid part. The instruction-tuned variants are not, because the post-training filter asks the wrong question. Appendix B screens the SFT and DPO sets with gpt-5-nano, keeping only examples judged “timeless”, where the stated test is whether the text states “facts that could BECOME OUTDATED”. The prompt therefore keeps programming tasks and tutorials “even mentioning real tools”, general content about real frameworks, and scientific facts that do not change. But durability and availability are different properties: a fact can be true forever and still have been unknown before the cut-off, and a tutorial can teach a technology that did not yet exist. The prompt's own worked example retains “React 18 introduced concurrent rendering” as a stable technical fact. That is exactly the sort of item a 2013 checkpoint should never have seen. This is a substantive weakness in the post-training argument, and it leaves the chronologically pretrained base models useful while the instruct variants' cut-offs are unverified.

• Cadence: monthly — the finest of any released sequence.
• Pretraining: filtered to text published before each cut-off; roughly 28,000 GPU-hours. restricted pretraining
• Post-training: LoRA instruction tuning, filtered for time-sensitivity rather than availability. wrong criterion
• Weights: see the paper for release details. check paper

DatedGPT — Yan, Tang, Gao & Jiang

arXiv:2603.11838 (2026). Twelve 1.3-billion-parameter models trained from scratch on roughly 100 billion tokens each, with strict annual cut-offs from 2013 to 2024, plus DatedInstruct, an instruction-tuning set grounded in each year's documents, intended to keep post-training inside the cut-off. Perplexity probes are used to check that each model's knowledge stops at its year.

The paper's distinctive result is the “lookahead premium”. Its estimand is narrower than the name suggests. On 61,000 firm-day news headlines the authors split the leaky sentiment signal as s^b = s^f + (s^b − s^f), where s^f is the historically admissible signal (vintage y−1 scoring year y headlines) and s^b the family's latest vintage applied to everything. They then estimate r_i,t+1 = α_i + δ_t + β₁s^f + β₂(s^b − s^f) with firm and date fixed effects. The 26.4 basis points (t = 10.65) is β̂₂ scaled by a standard deviation: the next-day return associated with the residual, the information only the later vintage had, conditional on the admissible signal. It is not an extra portfolio return attributable to leakage. The portfolio comparison is reported separately, an annualised Sharpe of 3.20 lookahead-free against 3.65 lookahead-biased. The leakage interpretation is the authors', and it carries an assumption: even matched vintages of one family differ in language competence and corpus composition, not only in access to the future.

• Pretraining: from scratch on text published before each cut-off. restricted pretraining
• Post-training: DatedInstruct is grounded in each year's own documents, the tightest construction here. period-grounded
• Weights and demo: see the paper. check paper

Time Machine GPT (TiMaGPT) — Oxford

Drinkall, Rahimikia, Pierrehumbert & Zohren (2024), Findings of NAACL, arXiv:2404.18543. Twelve yearly GPT-2 models with cut-offs from 2011-12-31 to 2022-12-31, designed to be “nonprognosticative”: uninformed about future facts and future language. Unlike most predecessors the models are trained from scratch on period data rather than adapted from a modern base, and the exact training set for each year is published alongside the weights.

• Weights: huggingface.co/Ti-Ma — Ti-Ma/TiMaGPT2-2011 … TiMaGPT2-2022, licence CC0.
• Data: Ti-Ma/TiMaGPT2-YYYY and Ti-Ma/wikipedia_YYYY datasets (WMT News Crawl and yearly Wikipedia snapshots).
• Pretraining: from scratch on period data, and because each year's corpus is released the claim is auditable. That is not the same as proven: correct timestamps and unbiased selection still have to hold. restricted pretraining

TimeLMs — Cardiff NLP

Loureiro, Barbieri, Neves, Espinosa Anke & Camacho-Collados (2022), ACL demo, arXiv:2202.03829. RoBERTa-base checkpoints trained on Twitter, released quarterly from 2019-12 through 2022-12 (plus a 2022 RoBERTa-large). The accompanying library has a corresponding mode that matches each tweet to its own quarter's checkpoint. That is a temporal-adaptation interface, not a point-in-time cut-off. Quarters run Jan–Mar, Apr–Jun and so on, and each quarter's model is trained on that whole quarter, so the paper's own example, a tweet dated 1 February 2021 routed to the 2021-Q1 model, scores it with a model that has read late February and March. For a historical forecast, choose a checkpoint whose training window ends before the forecast origin, which usually means the preceding quarter. Built for studying language-model degradation and cultural shift on social media.

• Weights: cardiffnlp/twitter-roberta-base-2019-90m, -mar2020 … -sep2022, -2022-154m.
• Provenance: the 2019 base continues pretraining from RoBERTa-base, which itself saw web text to 2019, and later quarters are incremental. The base's provenance limits any cut-off claim before 2020 regardless of which quarter is selected. base contaminated

StoriesLM — Sarkar

Sarkar (2024), SSRN 4881024. Sixty-four models, one per year from 1900 to 1963, trained on the American Stories dataset of historical U.S. newspapers. The 1900 model is trained on 1900 text; each subsequent model continues the previous year's checkpoint on the next year's articles, so the pretraining window expands sequentially. The only released sequence covering the first half of the twentieth century at yearly cadence.

• Weights: huggingface.co/StoriesLM — StoriesLM/StoriesLM-v1-1900 … -1963, MIT licence.
• Pretraining: sequentially expanding windows over period text; each year continues the previous clean checkpoint rather than restarting, which preserves chronological consistency. restricted pretraining

HistBERT — Qiu & Xu

Qiu & Xu (2022), arXiv:2202.03612. Not a checkpoint sequence, and not one model per decade. Three variants of BERT-base are continued on the Corpus of Historical American English, each over a different historical span: HistBERT-proto is the pilot, HistBERT-5 covers the 1910s–1950s collectively, and HistBERT-10 covers the 1910s–2000s collectively, the 2010s excluded because it would overlap BERT's own pretraining data. Built to track diachronic lexical semantic change. Code and training recipe on GitHub.

• Historical adaptation, not a cut-off claim: every variant continues modern BERT-base, and each spans five or ten decades, so no variant corresponds to a single date. historical adaptation
• COHA itself requires academic access.

Also relevant

• TiMoE — Faro, Fan, Alphaidze & Jaggi (2025), arXiv:2508.08827. A collection of separate GPT-style expert models, each pretrained from scratch on a disjoint two-year slice of 2013–2024, with experts whose window ends after the query date masked at inference. The chronological justification is clearest for TiMoE-Avg, which aggregates the surviving clean experts with fixed weights and needs no extra training. Learned routing is weaker: the router is a one-layer MLP trained on the whole multi-period corpus, and TiMoE-CoAdapt jointly updates the router and the “present” expert for each document. An earlier query can therefore receive mixture weights shaped by later training data even when the later experts are masked. That is a vulnerability in the stated guarantee rather than proof that any particular prediction leaked, but learned routing needs its own cut-off controls, including during co-adaptation. Two-year granularity.
• TypewriterLM — Luo et al. (2026), arXiv:2606.02991. A single 7.24B model trained only on English text before 1913, with a 54B-token corpus and a post-training stage constrained by “lexical grounding”: a response is accepted only if every lexical token is derivable from the source passage plus a small allowlist of function words, with limited morphological normalisation and OCR repair, enforced by a verifier that discards non-compliant responses. Instruction generation and judging still use modern LLMs, and the authors add human review of the small LIMA-style set together with explicit leakage evaluations. Source vocabulary, meaning supported by the source, and historical availability are three different things. The constraint establishes the first only. Old words can be rearranged, and source passages chosen, to convey something learned later. The largest “history language model” released; a sequence could be built the same way.
• Diachronic slice models — Fittschen, Li, Lippincott & Choshen (2025), arXiv:2504.05523. Five small models pretrained on five 10-million-word time slices, for humanities research; argues pretraining beats fine-tuning for period fidelity.
• TiC-LM — Li et al. (2025), arXiv:2504.02107. A time-continual pretraining benchmark over 114 Common Crawl dumps; shows replay schedules that match from-scratch retraining at 2.6× less compute, which is how a monthly sequence could be produced cheaply.
• Pitinf models (PiT-Inference) — a small, medium, and “frontier-level” family of point-in-time LLMs evaluated in Look-Ahead-Bench. Availability outside the benchmark is not documented here. unverified
• Temporal adaptation of BERT — Röttger & Pierrehumbert (2021), arXiv:2104.08116, and Lazaridou et al. (2021), Mind the Gap, trained time-stratified models to study degradation rather than to release a reusable sequence.
• Prompted cut-offs are the cheap alternative to all of the above. The evidence that they are not a substitute is collected under Prompted cut-offs and chronology.