- llm
- nlp
- reasoning
- rl
- rlhf
- ssm
- theory
- information-theory
- representation-learning
- optimization
- rag
- alignment
- paper-review
- survey
- notes
•
•
•
•
•
•
•
•
•
•
•
•
•
•
-
Blessing of Class Diversity in Pre-training
Theoretical analysis of why MLM works as a pre-training objective through the lens of class diversity.
-
Deriving Language Models from Masked Language Models
Computing valid joint distributions from MLMs via Markov Random Fields, with regularization to relax conditional independence.
-
Neural Collapse: Terminal Phase of Deep Network Training
The four manifestations of neural collapse and what they reveal about deep learning's implicit biases.
-
BERT: Pre-training of Deep Bidirectional Transformers
Review of BERT — bidirectional pre-training via MLM and NSP, and fine-tuning for downstream tasks.
-
Attention Is All You Need
Review of the Transformer paper — self-attention mechanism, encoder-decoder architecture, and why it replaced RNNs.