Reading Group Notes: Concrete Problems in AI Safety, Ten Years On
Notes from our reading group on which of the paper’s five problems have become research fields — and which remain under-studied.
Technical AI Safety
Getting AI systems to pursue the goals we actually intend, and specifying those goals well.
Notes from our reading group on which of the paper’s five problems have become research fields — and which remain under-studied.
Lauro Langosco, Jack Koch, Lee Sharkey, Jacob Pfau, David Krueger, ICML 2022, 2021
Demonstrates agents that keep their capabilities out of distribution but pursue the wrong goal — for example, an agent that learned to run to the end of a level rather than to collect the coin that was always placed there…
Evan Hubinger, Carson Denison, Jesse Mu, et al., arXiv, 2024
Deliberately trains models with hidden, trigger-dependent behaviour and finds that standard safety training — fine-tuning, reinforcement learning and adversarial training — does not reliably remove it.
Ethan Perez, Sam Ringer, Kamilė Lukošiūtė, et al., arXiv, 2022
Uses language models to generate large evaluation datasets cheaply, then uses them to surface behaviours such as sycophancy that become more pronounced with scale or with RLHF.
Stuart Russell, Viking, 2019
A leading AI researcher argues that machines designed to pursue fixed objectives are dangerous, and proposes instead systems that are uncertain about human preferences and learn them by observing people.
Brian Christian, W. W. Norton, 2020
A narrative history of the problem of getting machine-learning systems to do what we intend, connecting fairness, reinforcement learning, imitation and inverse reinforcement learning through interviews with researchers.
BlueDot Impact, 2024
An organisation offering free courses on AI safety and governance, combining curated readings with facilitated small-group discussion.