Masked Latent Semantic Modeling (MLSM)

Created in July 11, 2023

2023   ·   nlp   llm   ·   paper-review

MLSM extends masked language modeling into the latent space. Rather than masking surface tokens, the method applies knowledge distillation and sparse coding techniques to mask semantic representations during training. This shifts the pre-training signal from token-level prediction to latent-level prediction, encouraging richer semantic representations.

Key ideas:

  • Knowledge distillation provides the latent targets
  • Sparse coding selects which latent features to mask
  • Training objective operates in representation space rather than vocabulary space


Enjoy Reading This Article?

Here are some more articles you might like to read next:

  • The Expressive Power of Transformers with Chain of Thought
  • Landscape of Thoughts — Visualizing Where LLM Reasoning Actually Goes
  • Magellan — Guided MCTS for Escaping the Gravity Wells of LLM Creativity
  • PriorZero — Injecting LLM Priors into MuZero-Style World Models at the MCTS Root
  • SuperThoughts — Reasoning Tokens in Superposition