Corpora
Dated text one could train, extend, or evaluate a point-in-time model on, by domain.
A point-in-time model needs text with trustworthy dates and no later edits. That rules out much of the web as crawled today and favours archives with publication dates: newspapers, filings, court opinions, parliamentary records, preprints. The table gives each source's span, the finest temporal granularity it supports, and its licence. open means public domain or an open licence; academic means access is gated but free for research; commercial means a paid licence; mixed means it varies by sub-source or by API terms.
The sequences on the models page used, respectively: FineWeb (Kelly et al.), chronologically filtered web and news (ChronoBERT, DatedGPT), WMT News Crawl and Wikipedia (Time Machine GPT), Twitter (TimeLMs), American Stories (StoriesLM), and COHA (HistBERT). TypewriterLM's 54B-token pre-1913 corpus (Luo et al. 2026) and the TiC-LM Common Crawl dump series are the newest additions. Everything else below is a candidate.
Multilingual and Cross-Lingual Corpora
| Source | Coverage | Granularity | Licence | Notes |
|---|---|---|---|---|
| HPLT (High-Performance Language Technologies) | 75 languages, 5.6T tokens | Snapshot-based (v2.0, 2025-05) | open Open source (OSI compliant) | Multilingual temporal modeling, cross-lingual evolution studies |
| Common Corpus | 500B words public-domain | Snapshot-based | open Public domain | Multilingual pre-training, temporal analysis across languages |
| FineWeb | 2013-present (per Common Crawl dump) | Crawl dump (roughly monthly) | mixed ODC-By | Chronologically filtered by crawl date for Scaling PiT LMs (Kelly et al. 2026) |
| Wikipedia dumps | 2001-present | Twice-monthly dumps; full revision history | open CC BY-SA | Yearly snapshots used by Time Machine GPT; consecutive snapshots used by TemporalWiki |
News and Media Corpora
| Source | Coverage | Granularity | Licence | Notes |
|---|---|---|---|---|
| American Stories Dataset | 1800-1963 (19th-mid-20th century) | Yearly clustering | open Open source | Historical research, cultural studies, media analysis |
| New York Times Annotated Corpus | 1987-2007 | Daily articles | commercial Commercial license required | News analysis, temporal event tracking |
| Reuters News Archive | 1987-present | Real-time updates | commercial Commercial subscription required | Financial analysis, event detection |
| Finance Commons | Public-domain financial subset of Common Corpus | Snapshot-based | open Public domain | Financial modeling, open-source financial analysis |
| Newswire | 1878-1977 | Article date (daily) | open Public domain | 2.7M unique U.S. newswire articles reconstructed from 138M local-newspaper articles; georeferenced and entity-linked (Silcock et al. 2024) |
| Chronicling America (Library of Congress) | 1756-1963 | Issue date (daily) | open Public domain | Raw scans and OCR behind American Stories; historical research |
| WMT News Crawl | 2007-present | Yearly releases, article dates | open Open access for research | Training data for Time Machine GPT |
Social Media Corpora
| Source | Coverage | Granularity | Licence | Notes |
|---|---|---|---|---|
| Twitter TimeLMs Corpus | 2019-Q4 → 2022-Q4 | Quarterly snapshots | open Open source | Social media analysis, language change studies |
| Reddit Historical Data | 2005-present | Monthly dumps | open Open source | Community analysis, trend tracking |
Historical and Literary Corpora
| Source | Coverage | Granularity | Licence | Notes |
|---|---|---|---|---|
| Corpus of Historical American English (COHA) | 1810-2009 | Decade-by-decade sampling | academic Academic access required | Historical linguistics, cultural studies |
| Google Books Ngram Corpus | 1500-2019 | Yearly word frequencies | open Open access | Lexical change, cultural trends |
Scientific and Technical Corpora
| Source | Coverage | Granularity | Licence | Notes |
|---|---|---|---|---|
| PubMed Central Archive | 1781-present | Publication date | open Open access | Scientific progress tracking, medical history |
| arXiv Papers | 1991-present | Publication date | open Open access | Scientific progress analysis, knowledge evolution |
Legal and Government Corpora
| Source | Coverage | Granularity | Licence | Notes |
|---|---|---|---|---|
| U.S. Supreme Court Opinions | 1791-present | Decision date | open Public domain | Legal history, precedent analysis |
| EU Court of Justice Decisions | 1952-present | Decision date | open Open access | EU legal history, cross-jurisdictional analysis |
| UN Treaty Collections | 1945-present | Treaty date | open Open access | International law research, treaty analysis |
| Congressional Record | 1873-present | Daily proceedings | open Public domain | Political history, policy analysis |
| UK Hansard (Parliamentary Proceedings) | 1803-present | Daily proceedings | open Open access | Political history, policy analysis, parliamentary studies |
| EU ParlSpeech | European Parliament proceedings | Session-based | open Open access | EU policy analysis, parliamentary studies |
| Caselaw Access Project | 1658-2018 | Decision date | open Public domain | All published U.S. state and federal case law digitised from Harvard Law Library |
Financial and Economic Corpora
| Source | Coverage | Granularity | Licence | Notes |
|---|---|---|---|---|
| SEC EDGAR Database | 1993-present | Filing date | open Public domain | Financial analysis, regulatory compliance |
| Federal Reserve Economic Data (FRED) | 1774-present | Varies by series | open Open access | Economic analysis, policy research |
Patent and Technical Standards Corpora
| Source | Coverage | Granularity | Licence | Notes |
|---|---|---|---|---|
| USPTO Bulk Patent Data | 1790-present | Patent filing date | open Public domain | Innovation studies, technical evolution, patent analysis |
| EPO BigPatent | European patents | Patent filing date | open Open access | European innovation studies, technical evolution |
Climate and Earth Science Corpora
| Source | Coverage | Granularity | Licence | Notes |
|---|---|---|---|---|
| IPCC Reports | 1990-present | Report publication date | open Open access | Climate science evolution, environmental policy analysis |
| NASA Technical Reports | NASA research publications | Publication date | open Public domain | Space science evolution, Earth science analysis |
How fine can the clock tick?
The cadence of a checkpoint sequence is bounded by the cadence of its data. Most sources support daily dating, but a useful corpus at a given cadence needs enough tokens per step to move the model. The recommendations below, from the granularity guide in the repository, balance temporal precision against tokens per checkpoint.
| Data source | Finest practical | Recommended cadence |
|---|---|---|
| Twitter (academic API) | hourly | quarterly |
| Reddit (Pushshift) | daily | monthly |
| Wikipedia dumps | daily | monthly |
| SEC EDGAR | daily | quarterly |
| Financial news (Common Crawl) | daily | monthly |
| FRED economic releases | daily | monthly |
| arXiv / PubMed | daily | monthly |
| Historical newspapers (Chronicling America) | daily | yearly |
| COHA | yearly | decadal |
| Congressional Record / Hansard | daily | yearly |
| Court opinions | daily | yearly |
Source: PIT-CORPUSES.md and CORPUS-GRANULARITY.md. Some entries in the source list are descriptive rather than pointing at a single dataset; where no canonical link exists the name is left unlinked. Corrections welcome.