GPT-2: Language Models Are Unsupervised Multitask Learners

Created in October 09, 2023

2023   ·   llm   ·   paper-review

GPT-2 (Radford et al., 2019) demonstrates that scaling a simple autoregressive language model (predict next token) on a diverse web corpus yields surprising zero-shot capability across tasks like reading comprehension, summarization, and translation — without any task-specific fine-tuning.

Key contributions:

  • WebText: 40GB filtered Reddit-linked corpus for high-quality training data
  • 1.5B parameter model (large for 2019)
  • Zero-shot on CoQA, CNN/DM, Winograd: competitive with supervised baselines
  • The model learns task formatting implicitly from the training distribution

The paper’s core argument: “Language models are multitask learners” — a single model doing many things well is a property of scale + data diversity, not architecture. This set the stage for GPT-3’s few-shot learning and the modern foundation model paradigm.



Enjoy Reading This Article?

Here are some more articles you might like to read next:

  • The Expressive Power of Transformers with Chain of Thought
  • Landscape of Thoughts — Visualizing Where LLM Reasoning Actually Goes
  • Magellan — Guided MCTS for Escaping the Gravity Wells of LLM Creativity
  • PriorZero — Injecting LLM Priors into MuZero-Style World Models at the MCTS Root
  • SuperThoughts — Reasoning Tokens in Superposition