A Closer Look at How Fine-tuning Changes BERT

Created in July 13, 2023

2023   ·   nlp   llm   ·   paper-review

This ACL 2022 paper systematically examines how fine-tuning modifies pre-trained BERT across three dimensions: performance outcomes, internal representations, and layer-specific behavior.

Key findings:

  • Fine-tuning adapts higher layers more dramatically than lower layers
  • Task-specific information emerges primarily in upper layers
  • The pre-trained representations provide a strong scaffold that fine-tuning reshapes incrementally rather than overwriting

Understanding these dynamics is important for deciding how many layers to fine-tune and for explaining transfer learning success.



Enjoy Reading This Article?

Here are some more articles you might like to read next:

  • The Expressive Power of Transformers with Chain of Thought
  • Landscape of Thoughts — Visualizing Where LLM Reasoning Actually Goes
  • Magellan — Guided MCTS for Escaping the Gravity Wells of LLM Creativity
  • PriorZero — Injecting LLM Priors into MuZero-Style World Models at the MCTS Root
  • SuperThoughts — Reasoning Tokens in Superposition