Agentic AI Systems: A Safety Checklist for Builders
Practical questions to answer before giving a language-model agent access to tools, data or the internet.
Technical AI Safety
Keeping autonomous, tool-using systems within safe and controllable bounds.
Practical questions to answer before giving a language-model agent access to tools, data or the internet.
Richard Ngo, Lawrence Chan, Sören Mindermann, arXiv (ICLR 2024), 2022
Argues that systems trained with today’s methods could learn to exploit flaws in their reward signal, internalise goals that differ from those intended, and pursue power-seeking strategies — and explains why each is hard to detect.
Lauro Langosco, Jack Koch, Lee Sharkey, Jacob Pfau, David Krueger, ICML 2022, 2021
Demonstrates agents that keep their capabilities out of distribution but pursue the wrong goal — for example, an agent that learned to run to the end of a level rather than to collect the coin that was always placed there…