Common Crawl

“If you take like any old, like the pile or one of the more modern ones, basically any common crawl derived, scrape, open web text, any of these things, you're going to find that almost all documents are very short. There's some long context documents, but the …”

“We found that there's more AI generated content on the internet than human generated content. So back to the common crawl study, we looked at a 100,000 different URLs over the past five years. And then you can see this curve where AI generated is now higher th…”

“Grok five and for sure grok six will not use common crawl. It will not use the internet.”

“For example, here we measured like these words ratio in different dumps of common crawl, and we can see that like the ratio really increased after chat GPT's release.”

“So what we did is we trained different models on these different dumps, and we then computed their performance on popular like NLP benchmarks, and then we computed the aggregated score. And surprisingly, you can see that the latest dumps are actually even bett…”

“What they found is, as a reaction to, like, the close like, of the existence of closed models, like OpenAI or Cloud GPT or Cloud a lot of content owners have blanket blocked any type of crawling to their website.”

“In YouTube growing by one to two petabytes per day, which makes YouTube's data repository 300 times larger than common crawl, which makes it bigger than anything else that anyone else has.”