A Python framework for writing and running reproducible evaluations of language models, including multi-turn and tool-using agent evaluations.
This is an external publication hosted at inspect.aisi.org.uk. AI Governance LK did not write it; the summary and notes on this page are our own.
Why it matters
Shared tooling makes evaluation results easier to reproduce and compare. Inspect is used by government evaluators and independent researchers, and its task format is a good default for new evaluation projects on this platform.
What you will learn
- Write a simple evaluation task with a dataset, solver and scorer
- Run it against more than one model and compare logs
Before you start
Suggested background. This is editorial guidance, not a requirement.
- Python
- Access to a model API or a local model
How to cite
UK AI Security Institute. (2024). Inspect: An open-source framework for large language model evaluations.