AMOM: Adaptive Masking over Masking (AAAI 2023)

Created in July 17, 2023

2023   ·   nlp   llm   ·   paper-review

Background

Conditional Masked Language Models (CMLM) apply MLM-style objectives to the decoder in seq2seq architectures, allowing non-autoregressive generation. This is faster than autoregressive decoding but typically lags in quality.

Two Masking Types

Adaptive X Masking (input side): Applied to source sequences, similar to BERT-style masking on the encoder input.

Adaptive Y Masking (output side): Applied during decoding to mask low-confidence predictions, which are then iteratively refined. Unlike autoregressive models that generate left-to-right, this targets uncertain positions directly.

Why “Adaptive”?

Two sources of adaptiveness:

  1. The X and Y masking strategies interact with each other — information from one affects the other
  2. Masking ratios are dynamic rather than fixed throughout training and inference

This flexibility allows the model to allocate prediction effort where uncertainty is highest, improving generation quality without sacrificing the parallelism of non-autoregressive decoding.



    Enjoy Reading This Article?

    Here are some more articles you might like to read next:

  • The Expressive Power of Transformers with Chain of Thought
  • Landscape of Thoughts — Visualizing Where LLM Reasoning Actually Goes
  • Magellan — Guided MCTS for Escaping the Gravity Wells of LLM Creativity
  • PriorZero — Injecting LLM Priors into MuZero-Style World Models at the MCTS Root
  • SuperThoughts — Reasoning Tokens in Superposition