Mask More and Mask Later (ACL 2022)

Created in July 16, 2023

2023   ·   nlp   llm   ·   paper-review

Core Idea

The paper identifies two types of information in each token: token information and position information. Through attention, masking creates four distinct information flows combining masked/unmasked entities with self-loop and transfer mechanisms.

The key observation: two flows are particularly valuable for pre-training —

  1. Unmasked tokens gathering context from each other
  2. Transferring that context to masked positions for prediction

Proposals

Mask More: Use a higher masking rate than the standard 15%, since the masking rate determines which flows dominate. Higher rates strengthen the valuable context-transfer flow.

Mask Later: Use token embeddings during final estimation while ignoring the [MASK] token’s own information flow during intermediate computations. This disentangles the [MASK] token’s dual role as both a target and a context provider.

Result

Restructuring which information flows are emphasized during pre-training achieves greater computational efficiency while maintaining model quality — effectively doing more with each training step.



    Enjoy Reading This Article?

    Here are some more articles you might like to read next:

  • The Expressive Power of Transformers with Chain of Thought
  • Landscape of Thoughts — Visualizing Where LLM Reasoning Actually Goes
  • Magellan — Guided MCTS for Escaping the Gravity Wells of LLM Creativity
  • PriorZero — Injecting LLM Priors into MuZero-Style World Models at the MCTS Root
  • SuperThoughts — Reasoning Tokens in Superposition