Generalization improved significantly when models were trained on broader datasets like internet scrapes (e.g., Common Crawl, Reddit) rather than narrow corpora.
It was only when you trained over all the tasks on the internet — when you did a general internet scrape from something like Common Crawl or scraping links in Reddit, which is what we did for GPT-2 — that you started to get generalization.