Safety Evaluations for Sinhala and Tamil
Most published safety evaluations are written in English. This project develops a small, carefully translated and locally adapted evaluation suite in Sinhala and Tamil, and a protocol for…
Technical AI Safety
Behaving safely under distribution shift, adversarial pressure and edge cases.
Most published safety evaluations are written in English. This project develops a small, carefully translated and locally adapted evaluation suite in Sinhala and Tamil, and a protocol for…
Notes from our reading group on which of the paper’s five problems have become research fields — and which remain under-studied.
Most safety evaluations are written in English. What changes when we test in Sinhala, Tamil or Arabic?
Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, Dan Mané, arXiv, 2016
A foundational paper that reframed AI safety as a set of practical machine-learning research problems, from reward hacking to robustness under distributional shift.
Dan Hendrycks, Nicholas Carlini, John Schulman, Jacob Steinhardt, arXiv, 2021
A research agenda that organises machine-learning safety into four areas — robustness, monitoring, alignment and systemic safety — with concrete open problems in each.
Lauro Langosco, Jack Koch, Lee Sharkey, Jacob Pfau, David Krueger, ICML 2022, 2021
Demonstrates agents that keep their capabilities out of distribution but pursue the wrong goal — for example, an agent that learned to run to the end of a level rather than to collect the coin that was always placed there…
Stephanie Lin, Jacob Hilton, Owain Evans, ACL 2022, 2021
A benchmark of questions designed so that common human misconceptions produce wrong answers. In the original experiments, larger models were often less truthful, not more.
Andy Zou, Zifan Wang, J. Zico Kolter, Matt Fredrikson, arXiv, 2023
Shows that automatically optimised text suffixes can make safety-trained language models comply with harmful requests, and that such suffixes often transfer between models.