Getting Started with Mechanistic Interpretability
A reading order and a first experiment for engineers who want to understand what happens inside a model.
Technical AI Safety
Reverse-engineering the internal computations of neural networks.
A reading order and a first experiment for engineers who want to understand what happens inside a model.
Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, Shan Carter, Distill, 2020
A highly visual introduction to the idea that neural networks can be understood in terms of features and the circuits that connect them, using examples from an image classifier.
Nelson Elhage, Neel Nanda, Catherine Olsson, et al., Transformer Circuits Thread, 2021
Develops a way of reading small attention-only transformers as sums of interpretable paths, and uses it to identify induction heads — a mechanism for in-context copying.
Nelson Elhage, Tristan Hume, Catherine Olsson, et al., Transformer Circuits Thread, 2022
Uses small synthetic models to show how networks can represent more features than they have dimensions by storing them in superposition, and maps when and how this happens.
Trenton Bricken, Adly Templeton, Joshua Batson, et al., Transformer Circuits Thread, 2023
Applies sparse autoencoders to a small transformer and recovers many features that are far more interpretable than individual neurons, offering a practical response to superposition.
Neel Nanda and open-source contributors, GitHub, 2022
A Python library for mechanistic interpretability of GPT-style language models that makes it easy to cache, inspect and edit internal activations.