Evaluates many language models across a broad set of scenarios and several metrics at once — including calibration, robustness, fairness, bias and toxicity — rather than accuracy alone.

This is an external publication hosted at arxiv.org. AI Governance LK did not write it; the summary and notes on this page are our own.

Why it matters

HELM’s taxonomy of scenarios and metrics is a useful checklist when designing any evaluation, and its coverage gaps — including limited coverage of South Asian languages — are a reminder of how much remains to be measured.

What you will learn

  • Explain why multi-metric evaluation matters
  • Identify gaps in scenario coverage, such as languages

Before you start

Suggested background. This is editorial guidance, not a requirement.

  • Basic understanding of benchmarks

How to cite

Liang, P., Bommasani, R., Lee, T., et al. (2023). Holistic Evaluation of Language Models. Transactions on Machine Learning Research.