GPT-3: Language Models Are Few-Shot Learners

Created in October 09, 2023

2023   ·   llm   ·   paper-review

GPT-3 (Brown et al., 2020) scales GPT-2’s approach to 175 billion parameters and introduces in-context learning: given a few examples in the prompt (few-shot), the model generalizes to new inputs without any parameter updates.

Key results:

  • Few-shot performance approaches or exceeds fine-tuned smaller models on many benchmarks
  • Meta-learning interpretation: in-context learning as implicit gradient descent in the forward pass
  • Strong performance on TriviaQA, HellaSwag, SuperGLUE

Limitations acknowledged by the authors:

  • Still struggles with arithmetic and formal reasoning
  • In-context sample efficiency is poor compared to fine-tuning
  • High inference cost
  • Difficult to update knowledge after training

GPT-3 established in-context learning as a primary interface for LLMs and triggered a wave of follow-up work on prompt engineering, instruction tuning, and RLHF.



Enjoy Reading This Article?

Here are some more articles you might like to read next:

  • The Expressive Power of Transformers with Chain of Thought
  • Landscape of Thoughts — Visualizing Where LLM Reasoning Actually Goes
  • Magellan — Guided MCTS for Escaping the Gravity Wells of LLM Creativity
  • PriorZero — Injecting LLM Priors into MuZero-Style World Models at the MCTS Root
  • SuperThoughts — Reasoning Tokens in Superposition