Demonstrates agents that keep their capabilities out of distribution but pursue the wrong goal — for example, an agent that learned to run to the end of a level rather than to collect the coin that was always placed there in training.

This is an external publication hosted at arxiv.org. AI Governance LK did not write it; the summary and notes on this page are our own.

Why it matters

The experiments are simple enough to reproduce on a laptop, which makes this an excellent first replication project. The core lesson — that good training performance does not tell you which goal was learned — applies far beyond games.

What you will learn

  • Distinguish goal misgeneralisation from capability failure
  • Design a test that would reveal it

Before you start

Suggested background. This is editorial guidance, not a requirement.

  • Basic reinforcement learning

How to cite

Langosco, L., Koch, J., Sharkey, L., Pfau, J., & Krueger, D. (2022). Goal Misgeneralization in Deep Reinforcement Learning. Proceedings of the 39th International Conference on Machine Learning.