Develops a way of reading small attention-only transformers as sums of interpretable paths, and uses it to identify induction heads — a mechanism for in-context copying.

This is an external publication hosted at transformer-circuits.pub. AI Governance LK did not write it; the summary and notes on this page are our own.

How to read it

This is demanding. Work through it with a notebook open, reproducing the small models as you go. TransformerLens was built partly to make that practical.

What you will learn

  • Read a transformer as a residual stream with read/write components
  • Explain QK and OV circuits
  • Describe how induction heads work

Before you start

Suggested background. This is editorial guidance, not a requirement.

  • Linear algebra
  • How transformers work

How to cite

Elhage, N., Nanda, N., Olsson, C., et al. (2021). A Mathematical Framework for Transformer Circuits. Transformer Circuits Thread.