Introduces the distinction between outer and inner alignment and the idea of a “mesa-optimiser”: a learned model that is itself an optimiser, whose objective may differ from the one it was trained on.
This is an external publication hosted at arxiv.org. AI Governance LK did not write it; the summary and notes on this page are our own.
Why it matters
Much of the vocabulary used in alignment discussions comes from this report. It is conceptual rather than empirical, so read it alongside Goal Misgeneralization in Deep Reinforcement Learning, which provides concrete examples.
What you will learn
- Define inner and outer alignment
- Explain why deceptive alignment is a concern
Before you start
Suggested background. This is editorial guidance, not a requirement.
- Solid grasp of how models are trained by optimisation
How to cite
Hubinger, E., van Merwijk, C., Mikulik, V., Skalse, J., & Garrabrant, S. (2019). Risks from Learned Optimization in Advanced Machine Learning Systems. arXiv:1906.01820.