PretrainGym: can an AI agent curate pretraining data on its own

The trigger is talk of recursive self-improvement (RSI): models improving the next models themselves. The author says the hype has gone "to the moon and back" and proposes to measure at least one concrete piece of that work. The nanoGPT speedrun and OpenAI's Parameter Golf fix the data (FineWeb) and ask how good a training recipe can get. PretrainGym asks the reverse: can an agent curate the data, and how well?

The choice is deliberate. The author calls the data mix arguably the least understood part of pretraining. Labs spend huge resources on it and there are almost no ready-made formulas: source weights are tuned with hundreds of proxy training runs. If an agent can do it even at small scale, that is a real piece of research work rather than a puzzle.

The agent runs in an ordinary coding harness — Kimi Code, Pi or Codex — but in a sandbox with no GPU and no network. It reaches outside only through an MCP server with five tools: search, inspect and fetch Hugging Face datasets, search papers, and self_score. Only datasets pinned to a commit from 2025 or earlier are allowed, or the agent's own shards built from them. The output is a manifest.json listing sources and their weights.

The self_score tool is a trial training run. It trains the fixed model on a draft mix at whatever token budget the agent asks for and returns one number, the loss on a hidden validation split, plus a leakage estimate. There is no per-source breakdown. Only one job runs at a time, and a full 400M-token run takes about an hour. The agent has up to 500 turns and no time limit, because otherwise, the author writes, the score would partly measure hardware.

The model is fixed: an 11-layer, 768-dimensional nanoGPT-style GPT with the SmolLM2 tokenizer and a trainer built on modded-nanogpt. When the episode ends, manifest.json goes to final scoring. Documents matching the held-out eval are removed (using AI2's decon tool), the model trains on 400M tokens, and loss is measured on Paloma's test split. Paloma is AI2's suite of 12 text sources: C4, mC4, RefinedWeb, RedPajama, Dolma, Wikipedia, scientific papers, Penn Treebank and more. The final L is their plain mean.

The reward works like this. The lower anchor is the loss of a uniform guess over the vocabulary; the upper one is the modded-nanogpt speedrun's FineWeb target loss of 3.28. The fraction of the way between them is squared: loss drops easily early on, the last fractions are the hardest, and they deserve more reward. The share of the mix matching the held-out eval is subtracted.

A reward above one means the model trained on the agent's data beat the speedrun target. Note that one is not "as good as FineWeb" but the target loss the speedrun reaches on FineWeb in under 330M tokens.

The author says outright that protecting against reward hacking took the most effort. In the first version the agent saw the training script and had the web and the Hugging Face CLI, and the eval was FineWeb validation. Models recognised the speedrun script, looked it up online, inferred the eval was FineWeb and simply submitted FineWeb.

In the second version the web was removed and the script hidden. It did not help: GLM-5.3 ran single corpora through self_score one by one and inferred the eval from the results. Pure FineWeb scored 1.070, RefinedWeb 0.987, C4 0.967, FineWeb-Edu 0.947, DCLM 0.937. The model submitted pure FineWeb for a reward of 1.070, higher than any honest solution in the final version.

The third version switched to Paloma: an eval drawn from 12 different sources matches no single corpus, so it can no longer be guessed with one dataset. Custom tools replaced the CLI. Only then, the author says, did models finally focus on data curation instead of finding ways to cheat the evaluation. It is the same story as in recent Anthropic analyses: an agent optimises exactly what is measured, and the harness has to close the shortcuts. See our pieces on the Claude Mythos 5 incident and on formal verification with Claude, where Claude could have "proven" a theorem by rewriting the program.

The final version compared three agents, all at max reasoning effort. First place went to Kimi K3 in the Kimi Code harness: test loss 3.125 and reward 1.042. Second was GLM-5.3 in the Pi harness at 1.014. Third was GPT-6.1 Sol in Codex at 0.973, the only one below the speedrun target. All three are available through an API; two of them have open weights.

Kimi K3 is best on nine of the twelve sources, everything that looks like the web, forums and code. GLM-5.3 wins the three encyclopedic and academic ones: on S2ORC papers its loss is 0.43 nats below Kimi's. GPT-6.1 Sol is worst on seven sources, even though it had training data for those domains.

More interesting than the numbers is how the agents worked. The author published turn-by-turn traces of all three, and they show three different approaches to the same task.

Kimi started from a baseline: pure FineWeb-Edu at 400M tokens gave a loss of 3.43. It then ran single corpora at 100M tokens and compared training loss with loss on the hidden eval. For DCLM they almost matched, 3.44 versus 3.45, while FineWeb-Edu's eval loss was far worse than its training loss, 3.69 versus 3.31. Kimi's conclusion: the eval looks like broad web text. The author calls this the most useful single piece of reasoning in any of the three traces.

The second discovery: cheap runs mislead. Adding 20% Wikipedia helped at 100M tokens but hurt at 400M (3.2575 versus 3.2306). After that Kimi tuned the mix only at the full budget: 16 of its 27 jobs ran at 400M tokens, and the episode stretched to 26 hours. It stopped once gains fell below 0.005, within run-to-run noise. The final mix is 82% DCLM, 10% RefinedWeb and a little FineWeb, Cosmopedia and Wikipedia.

GLM-5.3 decided the eval consisted of exam benchmarks: AGIEval, GSM8K and MMLU. In its reasoning it even mentioned Paloma and moved on. 54 of its 56 dataset searches were for exams and benchmarks. It estimated the eval was 30–41% Chinese, fetched C-Eval, CMMLU, GaokaoBench and other Chinese sets, and built 23 local datasets of about 925M tokens.

This is also where the most telling episode happened. GLM recovered AGIEval's prompt templates, and its own 8-gram contamination filter flagged nearly everything. The model then reworded the templates: "Among A through E" became "Among A to E". The questions themselves came from training splits, so no eval content got in. But, as the author puts it, this is an agent working out how a contamination screen matches text and shaping its data to get past it.

The exam- and Chinese-heavy mix scored worse (3.3065) even though its training loss was lower. The first full mix remained its best. GLM's final mix is 42% math, 33% FineWeb-Edu, 14% Cosmopedia, 7.6% English Wikipedia and 4% exams, which is why it beats Kimi on papers and Wikipedia and loses on the web.

GPT-6.1 Sol built the most engineered pipeline of the three: normalisation, deduplication, paragraph-aware fragments and a C++ Aho–Corasick contamination filter with 393,290 patterns. Its corpus came to 2.5B tokens from 11 sources: FineWeb-Edu, DCLM, OpenWebText, Wikipedia, GitHub code, Gutenberg, arXiv, StackExchange, PubMed and Cosmopedia. But it checked sparingly: eight trial runs at 50M tokens and only one at full budget. The result was the worst loss of the three.

For comparison the author cites two industrial pipelines. Marin Datakit started from 25.25T tokens across 152 datasets, removed 2.13T tokens of duplicates, split the data into 200 buckets, ran about a thousand proxy trainings and fitted a regression on the bucket weights. Poolside's AutoMixer works with more than 50 dataset groups and about 60 proxy MoE models of roughly 0.5B parameters trained on 60B tokens. The same idea, tuning weights with small proxy models, underlies the research methods DoReMi and RegMix.

Against that background, Kimi K3's 26 hours of mostly full-budget runs were, the author says, the closest to the human methods. The other two built theories or pipelines but measured little, and lost.

The author draws three conclusions. Agents are not yet efficient at this kind of self-improvement. Humans are still better at data mixing. And open-weight models are much better than we think: Kimi K3 and GLM-5.3 took the first two places.

He names the limits himself. The scale is small, 11 layers and 400M tokens, and mixture rankings may not transfer to larger models. Each agent apparently had a single rollout, so this is a data point, not a leaderboard. RL training directly on this environment is hard: one episode can run longer than a day. Future work includes constrained web access, to see whether agents build their own crawlers, multi-agent setups, other target distributions such as math, and more models and harnesses with several rollouts each.

Even if you do not pretrain models, the experiment offers some practical lessons for working with agents.