Agentic AI Systems: A Safety Checklist for Builders
Practical questions to answer before giving a language-model agent access to tools, data or the internet.
Security and misuse
Attacks on AI systems and the use of AI to cause harm.
Practical questions to answer before giving a language-model agent access to tools, data or the internet.
Evan Hubinger, Carson Denison, Jesse Mu, et al., arXiv, 2024
Deliberately trains models with hidden, trigger-dependent behaviour and finds that standard safety training — fine-tuning, reinforcement learning and adversarial training — does not reliably remove it.
Andy Zou, Zifan Wang, J. Zico Kolter, Matt Fredrikson, arXiv, 2023
Shows that automatically optimised text suffixes can make safety-trained language models comply with harmful requests, and that such suffixes often transfer between models.
OWASP Foundation, OWASP, 2023
A community-maintained list of the most important security risks in applications built on language models, such as prompt injection and insecure handling of model output, with mitigations for each.
MITRE, 2021
A knowledge base of adversary tactics and techniques against AI-enabled systems, modelled on MITRE ATT&CK and illustrated with case studies.