A foundational paper that reframed AI safety as a set of practical machine-learning research problems, from reward hacking to robustness under distributional shift.
This is an external publication hosted at arxiv.org. AI Governance LK did not write it; the summary and notes on this page are our own.
Why it matters
Before this paper, much discussion of AI risk was abstract. The authors grounded it in five research problems that can be studied with today's systems: avoiding negative side effects, avoiding reward hacking, scalable oversight, safe exploration, and robustness to distributional shift. Each is illustrated with a simple cleaning-robot example that makes the failure mode easy to picture.
How to read it
Read the introduction and the section on reward hacking first; those ideas recur throughout the field. Then skim the remaining problems and note which ones feel most relevant to systems you know. The paper is a good companion to Unsolved Problems in ML Safety, which revisits the agenda five years later.
What you will learn
- Name the five research problems the paper defines
- Explain how each problem shows up in present-day systems
- Recognise the vocabulary later work builds on
Before you start
Suggested background. This is editorial guidance, not a requirement.
- Basic familiarity with supervised and reinforcement learning
How to cite
Amodei, D., Olah, C., Steinhardt, J., Christiano, P., Schulman, J., & Mané, D. (2016). Concrete Problems in AI Safety. arXiv:1606.06565.