Deriving Language Models from Masked Language Models

Created in July 06, 2023

2023   ·   information-theory   nlp   ·   paper-review

MLMs like BERT define conditional distributions $p(x_i \mid x_{\setminus i})$, but these conditionals do not generally correspond to a valid joint distribution — extracting a coherent language model from an MLM requires care.

This paper compares methods for constructing joint distributions from MLM conditionals:

  • Markov Random Field (MRF) factorization, treating the unmasked tokens as cliques
  • Pseudo-likelihood approximations
  • Other consistency-enforcing methods

Evaluated through metrics like P-PPL (pseudo-perplexity) and U-PPL (unigram-conditioned perplexity), the analysis reveals tradeoffs in how each method handles the conditional independence assumptions baked into the MLM.

The paper proposes regularization techniques to relax these assumptions, sitting at the intersection of information theory, stochastic processes, and probabilistic modeling — a useful theoretical lens on why MLMs are powerful but not “true” language models.



Enjoy Reading This Article?

Here are some more articles you might like to read next:

  • The Expressive Power of Transformers with Chain of Thought
  • Landscape of Thoughts — Visualizing Where LLM Reasoning Actually Goes
  • Magellan — Guided MCTS for Escaping the Gravity Wells of LLM Creativity
  • PriorZero — Injecting LLM Priors into MuZero-Style World Models at the MCTS Root
  • SuperThoughts — Reasoning Tokens in Superposition