Develops a way of reading small attention-only transformers as sums of interpretable paths, and uses it to identify induction heads — a mechanism for in-context copying.
This is an external publication hosted at transformer-circuits.pub. AI Governance LK did not write it; the summary and notes on this page are our own.
How to read it
This is demanding. Work through it with a notebook open, reproducing the small models as you go. TransformerLens was built partly to make that practical.
What you will learn
- Read a transformer as a residual stream with read/write components
- Explain QK and OV circuits
- Describe how induction heads work
Before you start
Suggested background. This is editorial guidance, not a requirement.
- Linear algebra
- How transformers work
How to cite
Elhage, N., Nanda, N., Olsson, C., et al. (2021). A Mathematical Framework for Transformer Circuits. Transformer Circuits Thread.