Introduces the distinction between outer and inner alignment and the idea of a “mesa-optimiser”: a learned model that is itself an optimiser, whose objective may differ from the one it was trained on.

This is an external publication hosted at arxiv.org. AI Governance LK did not write it; the summary and notes on this page are our own.

Why it matters

Much of the vocabulary used in alignment discussions comes from this report. It is conceptual rather than empirical, so read it alongside Goal Misgeneralization in Deep Reinforcement Learning, which provides concrete examples.

What you will learn

  • Define inner and outer alignment
  • Explain why deceptive alignment is a concern

Before you start

Suggested background. This is editorial guidance, not a requirement.

  • Solid grasp of how models are trained by optimisation

How to cite

Hubinger, E., van Merwijk, C., Mikulik, V., Skalse, J., & Garrabrant, S. (2019). Risks from Learned Optimization in Advanced Machine Learning Systems. arXiv:1906.01820.