Announcement

AI web text reportedly helped only data-starved models in COLM 2026 study

In X posts, Jenna Russell said Pangram labeled 31% of FineWeb-filtered web tokens as AI-generated in August 2026, up from 10% in June 2024. She said a study of 800 language models found AI tokens lowered loss on human text for data-starved models under about 10 human tokens per parameter, but raised it almost from Ch

12 Sources, 10d ago, first seen 10d ago

TLDR

In X posts, Jenna Russell said Pangram labeled 31% of FineWeb-filtered web tokens as AI-generated in August 2026, up from 10% in June 2024. She said a study pretraining 800 language models found AI tokens lowered loss on human text only for data-starved models under about 10 human tokens per parameter, then raised it almost immediately from Chinchilla-optimal settings onward. Mohit Iyyer’s no-new-data comparison was his own summary.

Combined views

13.9K

12 Sources, first seen 10d ago

222 likes22 comments57 saves87 reposts

Announcement

AI web text reportedly helped only data-starved models in COLM 2026 study

12 Sources, 10d ago, first seen 10d ago

TLDR

Today's Rank

—

Not ranked yet

Combined views

13.9K

12 Sources, first seen 10d ago

222 likes22 comments57 saves87 reposts

Jenna Russell said in an X thread that Pangram labeled 31% of FineWeb-filtered web tokens as AI-generated in August 2026, up from 10% in June 2024.

Featured Source

In the same post, Russell said her team pretrained 800 language models ranging from 19.9 million to 973 million parameters on mixes of human and AI web text to measure what those mixes do to pretraining.

In a follow-up post, Russell said that for data-starved models "under ~10 human tokens per param," adding AI tokens lowered loss on human text.

She said the pattern changed from "Chinchilla-optimal (20) on": adding AI tokens raised loss on human text almost immediately, while adding human tokens continued to lower the loss.

Mohit Iyyer later offered his own summary of the work on X, writing that across most standard training budgets, adding AI text makes models worse than if no new training data were added at all.

That no-new-data comparison was Iyyer's summary of the work, not Russell's wording in the cited posts.

Sentiment

Positive——Negative

Summary

Not enough discussion yet.

No sentiment analysis available yet.