Safety evaluations shape how models are trained and whether they are deployed. Yet the large majority are written in English. For users in Sri Lanka and the Gulf, that raises an obvious question: do the results transfer?

Why translation is not enough

Translating an English evaluation item word for word can produce something no native speaker would write. Cultural context matters too: a request that is clearly harmful in one setting may be ambiguous in another. Good multilingual evaluation needs native speakers involved in writing items, not only translating them.

Open questions

  • How should over-refusal — refusing benign requests — be measured across languages?
  • How large does an evaluation need to be before differences between languages are meaningful?
  • How do we avoid confusing translation artefacts with genuine differences in model behaviour?

What we are doing

Our project Safety Evaluations for Sinhala and Tamil is building a small, carefully reviewed suite to explore these questions. It has not produced results yet; when it does, we will publish the method alongside them.