Concrete Problems in AI Safety was published in 2016. It named five problems: avoiding negative side effects, avoiding reward hacking, scalable oversight, safe exploration, and robustness to distributional shift. Our reading group asked a simple question: what happened to each?

Problems that became fields

Scalable oversight and robustness are now substantial research areas, with dedicated papers, benchmarks and teams. Reward hacking has become a practical concern in the training of language models, where models can learn to satisfy a reward model in ways their designers did not intend.

Problems that remain quieter

Negative side effects and safe exploration received less attention in the language-model era, partly because the systems in question were not acting in the world. As models are increasingly deployed as agents with tools, both are becoming relevant again.

Questions for discussion

  • Which of the five problems is most relevant to systems deployed in your organisation?
  • What would a modern version of the paper add?