Auto-Regressive Next-Token Predictors Are Universal Learners

Created in October 09, 2023

2023   ·   llm   theory   ·   paper-review

This paper provides a theoretical justification for why next-token prediction (the pre-training objective of GPT-style models) is so general: it proves that autoregressive next-token predictors are universal learners in a formal sense — they can approximate any computable function on sequences given sufficient capacity and data.

The result connects the empirical success of LLMs to a clean theoretical guarantee: the objective is not accidentally powerful, but fundamentally capable of learning arbitrary sequential mappings. This also provides a theoretical framing for in-context learning as implicit Bayesian inference.



Enjoy Reading This Article?

Here are some more articles you might like to read next:

  • The Expressive Power of Transformers with Chain of Thought
  • Landscape of Thoughts — Visualizing Where LLM Reasoning Actually Goes
  • Magellan — Guided MCTS for Escaping the Gravity Wells of LLM Creativity
  • PriorZero — Injecting LLM Priors into MuZero-Style World Models at the MCTS Root
  • SuperThoughts — Reasoning Tokens in Superposition