A benchmark of questions designed so that common human misconceptions produce wrong answers. In the original experiments, larger models were often less truthful, not more.
This is an external publication hosted at arxiv.org. AI Governance LK did not write it; the summary and notes on this page are our own.
Why it matters
TruthfulQA is a compact example of good benchmark design: it targets a specific failure, explains its construction, and reports surprising results. It is also a useful case study in how benchmarks age once models are trained on the internet that contains them.
What you will learn
- Explain how adversarially written questions probe imitative falsehoods
- Discuss benchmark saturation and contamination
Before you start
Suggested background. This is editorial guidance, not a requirement.
- None
How to cite
Lin, S., Hilton, J., & Evans, O. (2022). TruthfulQA: Measuring How Models Mimic Human Falsehoods. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics.