Emergent Abilities of Large Language Models

Created in July 22, 2023

2023   ·   llm   theory   ·   paper-review

This paper documents emergent abilities in LLMs: capabilities that are not present in smaller models but appear sharply once a critical scale is reached, often with little improvement before that threshold.

Examples documented:

  • Multi-digit arithmetic
  • Multi-step word problems
  • Following complex chained instructions
  • Programmatic transformations

The argument: scaling is not just a smooth quantitative improvement; some capabilities exhibit phase transitions as a function of compute or parameters.

Subsequent Debate

A follow-up paper (Schaeffer et al., NeurIPS 2023) argued emergence may be partly a measurement artifact: discontinuous metrics (e.g., exact-match accuracy) hide smooth underlying improvements that would be visible under continuous metrics (e.g., per-token log-likelihood).

The empirical question is unresolved: even with smoother metrics, qualitative shifts in what tasks the model can do at all remain — but the precise role of metric choice in the appearance of emergence is now part of the discussion.



Enjoy Reading This Article?

Here are some more articles you might like to read next:

  • The Expressive Power of Transformers with Chain of Thought
  • Landscape of Thoughts — Visualizing Where LLM Reasoning Actually Goes
  • Magellan — Guided MCTS for Escaping the Gravity Wells of LLM Creativity
  • PriorZero — Injecting LLM Priors into MuZero-Style World Models at the MCTS Root
  • SuperThoughts — Reasoning Tokens in Superposition