Sailor: Open Language Models for South-East Asia

PAPER GITHUB HUGGING FACE DEMO

Introduction

Sailor is a suite of Open Language Models tailored for South-East Asia (SEA), focusing on languages such as 🇮🇩Indonesian, 🇹🇭Thai, 🇻🇳Vietnamese, 🇲🇾Malay, and 🇱🇦Lao. Developed with careful data curation, Sailor models are designed to understand and generate text across the diverse linguistic landscapes of the SEA region. Built from Qwen 1.5, Sailor encompasses models of varying sizes, spanning from 0.5B to 14B versions for different requirements.

Key features of Sailor include:

• Continually pretrained on 200 Billion to 400 Billion tokens over 7 languages, including Indonesian, Thai, Vietnamese, Malay, Lao, English, and Chinese.
• Various model sizes (0.5B, 1.8B, 4B, 7B, 14B) to support different requirements.
• Strong performance on SEA benchmarks such as XQuAD, TydiQA, XCOPA, Belebele, and M3Exam.
• No restrictions on research and commercial use, but must comply with the Qwen 1.5 license.

Built from Open-Source Community

Sailor owes its existence to the open-source community. It is crafted by continually pre-training from language models like the remarkable Qwen 1.5 models, which already have great performance on SEA languages. The pre-training corpus heavily leverages the publicly available corpus, including SlimPajama, SkyPile, CC100, and MADLAD-400.

By employing aggressive data deduplication and careful data cleaning on the collected corpus, we have attained a high-quality dataset spanning various languages. Through systematic experiments to determine the weights of different languages, Sailor models undergo training from 200B to 400B tokens, tailored to different model sizes. The approach boosts their performance on SEA languages while maintaining proficiency in English and Chinese without significant compromise. Finally, we continually pre-train the Qwen1.5-0.5B model with 400 Billion tokens, and other models with 200 Billion tokens to obtain the Sailor models.

For most of the models, we use 200 billion tokens, with the effective tokens for each language as shown below. For models utilizing 400 billion tokens, they are doubled accordingly.

Commitment to Open-Source Community

The release of Sailor models marks the beginning of our commitment to open-source. In the coming weeks, we plan to release:

• Training recipes
• Code for pre-training
• Pipeline for data cleaning and deduplication
• Pre-training corpus

Benchmarking Performance

Sailor models are evaluated across several high-quality benchmarks, encompassing four kinds of different tasks: question answering, commonsense reasoning, reading comprehension, and examination. We gratefully acknowledge the contributions of all dataset authors. For evaluation, following established protocols, we employed the awesome evaluation platform OpenCompass for comprehensive evaluation. The performance of all models is assessed based on the 3-shot Exact Match performance, with prompts provided in local languages (e.g., Indonesian task description for Indonesian tasks).

We acknowledge and respect the release of several SEA language models before, including SEA-LION, SeaLLMs, Typhoon, and VinaLLaMA. Here we mainly selected SeaLLMs-7B-Hybrid, its base model Llama-2-7B, SeaLLMs-7B-v2, and its base model Mistral-7B-v0.1 for performance comparison. Evaluation results for more models will be presented in our paper. Our reporting strictly adheres to the same evaluation methodology to ensure fair comparison, and we make much effort to closely match the reported results of the baseline.

Evaluation Tasks

• Question Answering: XQuAD (Thai, Vietnamese) and TydiQA (Indonesian).
• Commonsense Reasoning: XCOPA (Indonesian, Thai, Vietnamese).
• Reading Comprehension: Belebele (Indonesian, Thai, Vietnamese).
• Examination: M3Exam (Javanese, Thai, Vietnamese).

Question Answering

All models are evaluated on the XQuAD and TydiQA benchmarks, with the 3-shot Exact Match (EM) and F1 score reported. Baselines which have better performance than Sailor models are highlighted in green. Sailor-14B’s XQuAD (th) result seem abnormal, with its predictions tending to be semantically equivalent but longer than the groundtruth, leading to lower EM and F1 scores compared to Qwen1.5.

Commonsense Reasoning

All models are evaluated on the XCOPA benchmark, with the 3-shot accuracy reported.

Reading Comprehension

All models are evaluated on the Belebele benchmark, with the 3-shot Exact Match (EM) reported. Baselines which have better performance than Sailor models are highlighted in green.

Examination

All models are evaluated on the M3Exam benchmark, with the 3-shot Exact Match (EM) reported. The code jv is short for Javanese, which is a language spoken in Indonesia.

Contributors

• Longxu Dou, Sea AI Lab
• Qian Liu, Sea AI Lab
• Guangtao Zeng, SUTD
• Jia Guo, NUS
• Jiahui Zhou, Sea AI Lab
• Ziqi Jin, SUTD
• Xin Mao, NTU
• Wei Lu, SUTD
• Min Lin, Sea AI Lab

Contact Us

Sailor models are free for research and commercial use, but you should also obey the Qwen 1.5 license. We encourage you to use Sailor models in your research and applications.

For questions or collaboration, please:

• Raise an issue in our GitHub repository
• Contact us at:
  • doulx@sea.com
  • liuqian.sea@gmail.com

Citation

@article{dou2024sailor,
  title={Sailor: Open Language Models for South-East Asia},
  author={Dou, Longxu and Liu, Qian and Zeng, Guangtao and Guo, Jia and Zhou, Jiahui and Lu, Wei and Lin, Min},
  journal={arXiv preprint arXiv:2404.03608},
  year={2024}
}