Regional LLM versus Dedicated Neural MT for Waray–English: A Bidirectional FLORES-200 Evaluation of a Low-Resource Philippine Language

Abstract

Waray (Winaray), an Austronesian language of the eastern Philippines with about three million speakers, is computationally low-resourced and rarely evaluated in machine translation (MT). This study compares two MT approaches on Waray–English using FLORES-200: NLLB-200, a dedicated multilingual neural model that supports Waray, and Gemma-SEA-LION-v4-27B-IT, a Southeast Asian regional large language model (LLM) for which Waray is unsupported. Both were evaluated bidirectionally with spBLEU and chrF++, and a blind two-rater expert evaluation assessed the into-Waray direction. A directional asymmetry emerged: the LLM exceeded the dedicated model by 16.5 chrF++ into English but trailed by 4.5 into Waray. Yet two native experts found no significant difference in the systems’ Waray output, suggesting the metric gap may largely reflect single-reference divergence rather than quality. A consistent domain split favored the LLM on news and the dedicated model on instructional text. The findings caution against surface-metric-only evaluation for low-resource generation.