Status: In Progress. Nothing on this page has been peer reviewed. Treat any outputs as preliminary.

Most published safety evaluations are written in English. This project develops a small, carefully translated and locally adapted evaluation suite in Sinhala and Tamil, and a protocol for comparing model behaviour across the three languages.

Research question

Do safety-relevant behaviours of widely used language models — refusing harmful requests and answering benign ones — differ between English, Sinhala and Tamil, and if so, how?

Motivation

If safety behaviour degrades in the languages spoken by most of Sri Lanka’s population, users and deployers need to know. Evidence of this kind is scarce for South Asian languages.

Methodology

Items are drafted in English, translated by native speakers, then adapted for cultural context rather than translated literally. Each item is reviewed by two annotators. The suite is implemented as Inspect tasks so that runs are logged and reproducible. A pilot of roughly one hundred items will be used to refine the protocol before scaling.

Findings

No findings have been published for this project.

Limitations

  • Small item counts limit statistical power.
  • Translation choices can introduce artefacts that look like model differences.
  • Results will describe specific model versions at a specific time.

Current stage

The item-writing guidelines and the annotation protocol are drafted, and a prototype set of tasks runs end to end in Inspect. No results have been produced. Nothing on this page should be read as a finding.

How to get involved

We are looking for native Sinhala and Tamil speakers to review items, and for anyone with experience of evaluation design in other low-resource languages.