Frequency Effects on Syntactic Rule Learning in Transformers

Created in July 24, 2023

2023   ·   nlp   theory   ·   paper-review

This ACL 2021 paper investigates how the frequency of tokens and constructions in pre-training data shapes the syntactic rules that Transformers learn to generalize. The central finding is that frequency has a non-trivial effect on syntactic generalization: models trained on natural language distributions may learn frequency-based heuristics rather than genuine syntactic rules, complicating interpretations of “grammatical knowledge” in language models.

Understanding frequency effects is important for designing pre-training corpora and interpreting probing experiments.



Enjoy Reading This Article?

Here are some more articles you might like to read next:

  • The Expressive Power of Transformers with Chain of Thought
  • Landscape of Thoughts — Visualizing Where LLM Reasoning Actually Goes
  • Magellan — Guided MCTS for Escaping the Gravity Wells of LLM Creativity
  • PriorZero — Injecting LLM Priors into MuZero-Style World Models at the MCTS Root
  • SuperThoughts — Reasoning Tokens in Superposition