Demonstrates agents that keep their capabilities out of distribution but pursue the wrong goal — for example, an agent that learned to run to the end of a level rather than to collect the coin that was always placed there in training.
This is an external publication hosted at arxiv.org. AI Governance LK did not write it; the summary and notes on this page are our own.
Why it matters
The experiments are simple enough to reproduce on a laptop, which makes this an excellent first replication project. The core lesson — that good training performance does not tell you which goal was learned — applies far beyond games.
What you will learn
- Distinguish goal misgeneralisation from capability failure
- Design a test that would reveal it
Before you start
Suggested background. This is editorial guidance, not a requirement.
- Basic reinforcement learning
How to cite
Langosco, L., Koch, J., Sharkey, L., Pfau, J., & Krueger, D. (2022). Goal Misgeneralization in Deep Reinforcement Learning. Proceedings of the 39th International Conference on Machine Learning.