← All papers

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Devlin, Chang, Lee, Toutanova · 2018 · NAACL 2019

TrainingRead on arXiv

Showed how bidirectional pretraining with masked language modeling can dramatically improve language understanding. BERT set new benchmarks across 11 NLP tasks and popularized the pretrain-then-finetune paradigm.

Key Idea

BERT (Bidirectional Encoder Representations from Transformers) pretrains a Transformer encoder by jointly conditioning on both left and right context in all layers. Previous models like GPT were unidirectional (left-to-right), which limited their ability to understand context.

Pretraining Objectives

  • Masked Language Model (MLM): randomly mask 15% of input tokens and predict them. This forces the model to learn bidirectional representations
  • Next Sentence Prediction (NSP): given two sentences, predict whether the second follows the first. Helps with tasks like question answering and natural language inference

Architecture & Fine-tuning

  • Uses only the Transformer encoder (no decoder)
  • Two model sizes: BERT-Base (110M params) and BERT-Large (340M params)
  • Fine-tuning is straightforward: add a task-specific head on top of the pretrained encoder and train end-to-end on downstream data

Why It Matters

  • Established the pretrain → finetune paradigm that all modern NLP follows
  • Bidirectional context leads to much richer representations than left-to-right models
  • Single model achieves SOTA on diverse tasks: classification, NER, question answering, similarity
  • Spawned a family of models: RoBERTa, ALBERT, DistilBERT, DeBERTa

Key Takeaways for Interviews

  • MLM is the key innovation, bidirectional context through random masking
  • BERT is an encoder-only model (not generative)
  • Fine-tuning a pretrained BERT is far more sample-efficient than training from scratch
  • NSP was later shown to be less important (RoBERTa dropped it)