YTL AI Labs, in collaboration with Universiti Malaya (UM), has successfully developed the first MalayMMLU (Massive Multi-task Language Understanding), which has recently been accepted at the prestigious Empirical Methods in Natural Language Processing 2024 (EMNLP 2024) conference. The MalayMMLU is a comprehensive benchmark designed to evaluate large language models (LLMs) in Bahasa Melayu.

By providing a standardised mechanism for assessing LLMs across multiple language tasks, the MMLU benchmark facilitates fair and thorough evaluations, thereby driving excellence in LLM development. The MalayMMLU consists of over 24,000 questions covering the entire K-12 spectrum across 22 subjects from the Malaysian education curriculum.

This benchmark is vital for enhancing LLMs' understanding of Bahasa Melayu and advancing applications in education, healthcare, and public services, thus fostering culturally relevant AI solutions for Malaysia and Southeast Asia. YTL AI Labs and Universiti Malaya plan to publicly release the MalayMMLU dataset along with the evaluation code, enabling the wider AI community to contribute to the development of more inclusive and effective AI technologies.

Dato Seri Yeoh Seok Hong, Managing Director of YTL Power International, stated, "Malaysia is rapidly building an ecosystem that will support the development and adoption of AI in the country. Our first AI data centres are being built and our universities are focusing on building AI talent. The release of the MalayMMLU is an exciting milestone that will undoubtedly accelerate Malaysia's journey to becoming an AI Nation. YTL AI Labs is proud to be part of this journey."

Professor Ir. Dr Chan Chee Seng, Dean of the Faculty of Computer Science and Information Technology at Universiti Malaya, who co-led the development of the MalayMMLU with Foong Chee Mun, CEO of YTL AI Labs, added, "Acceptance of the MalayMMLU at the EMNLP 2024 conference marks a significant milestone in the development of LLMs tailored to the Malaysian context. We now have a benchmark for the Bahasa Melayu language which hitherto did not exist. We believe that we will now begin to see a proliferation of applications built on the back of this benchmark. It is a great privilege for Universiti Malaya to be working with YTL AI Labs to be the first in the world to set a universal benchmark for our national language."