BERT: Pre-training of Deep Bidirectional Transformers

Core Idea

BERT achieves deep bidirectionality through its pre-training objective rather than architecture. Unlike GPT (left-to-right) or ELMo (shallow bidirectional concatenation), BERT uses Masked Language Modeling to let every token attend to all others during pre-training.

Pre-training Tasks

Masked Language Modeling (MLM): 15% of tokens are randomly selected. Of those:

• 80% replaced with [MASK]
• 10% replaced with a random token
• 10% kept unchanged

This mixture prevents the model from simply learning “mask = predict this token.”

Next Sentence Prediction (NSP): Given two sentences A and B, predict whether B follows A in the original text. Trains the [CLS] token to encode document-level relationships.

Input Representation

Three embeddings summed:

• Token: WordPiece vocabulary
• Segment: differentiates sentence A vs. B
• Position: learned absolute positional embeddings

Special tokens: [CLS] at start (for classification tasks), [SEP] between sentences.

Fine-tuning

A single task-specific linear layer added on top of [CLS] (classification) or token representations (sequence labeling). All parameters fine-tuned end-to-end. This makes BERT remarkably general — the same pre-trained weights adapt to SQuAD, GLUE, NER, etc. with minimal task-specific engineering.

Takeaways

• Bidirectionality from pre-training objective, not architecture
• 15% mask rate balances learning signal and mismatch with fine-tuning
• [CLS] as a sentence-level representation is central to downstream use