A foundational paper that reframed AI safety as a set of practical machine-learning research problems, from reward hacking to robustness under distributional shift.

This is an external publication hosted at arxiv.org. AI Governance LK did not write it; the summary and notes on this page are our own.

Why it matters

Before this paper, much discussion of AI risk was abstract. The authors grounded it in five research problems that can be studied with today's systems: avoiding negative side effects, avoiding reward hacking, scalable oversight, safe exploration, and robustness to distributional shift. Each is illustrated with a simple cleaning-robot example that makes the failure mode easy to picture.

How to read it

Read the introduction and the section on reward hacking first; those ideas recur throughout the field. Then skim the remaining problems and note which ones feel most relevant to systems you know. The paper is a good companion to Unsolved Problems in ML Safety, which revisits the agenda five years later.

What you will learn

  • Name the five research problems the paper defines
  • Explain how each problem shows up in present-day systems
  • Recognise the vocabulary later work builds on

Before you start

Suggested background. This is editorial guidance, not a requirement.

  • Basic familiarity with supervised and reinforcement learning

How to cite

Amodei, D., Olah, C., Steinhardt, J., Christiano, P., Schulman, J., & Mané, D. (2016). Concrete Problems in AI Safety. arXiv:1606.06565.