A Python framework for writing and running reproducible evaluations of language models, including multi-turn and tool-using agent evaluations.

This is an external publication hosted at inspect.aisi.org.uk. AI Governance LK did not write it; the summary and notes on this page are our own.

Why it matters

Shared tooling makes evaluation results easier to reproduce and compare. Inspect is used by government evaluators and independent researchers, and its task format is a good default for new evaluation projects on this platform.

What you will learn

  • Write a simple evaluation task with a dataset, solver and scorer
  • Run it against more than one model and compare logs

Before you start

Suggested background. This is editorial guidance, not a requirement.

  • Python
  • Access to a model API or a local model

How to cite

UK AI Security Institute. (2024). Inspect: An open-source framework for large language model evaluations.