Arabic LLM Evaluation Framework

Project Overview

This project develops a systematic evaluation framework for Arabic Large Language Models (LLMs) specifically designed for educational conversational applications. The work addresses a critical gap in Arabic NLP research and is currently under review at Language Resources and Evaluation (Q1 journal).

Technical Details & Achievements

The Challenge

Existing LLM benchmarks focus primarily on English and use quantitative metrics that miss nuanced aspects of educational value. LLMs in genreal and specially Arabic LLMs need evaluation that captures the core purpose of the LLMs use cause in this case educational conversational applications:

• Pedagogical effectiveness
• Cultural and age appropriateness
• Conversational quality in educational contexts

Framework Approach

Our framework introduces a rubric-based evaluation system that enables qualitative and holistic assessment:

Left: Comparative evaluation of Arabic LLMs. Right: Analysis comparing different evaluation methods.

Key Contributions

• Novel Evaluation Rubric: Multi-dimensional assessment beyond simple accuracy metrics
• Arabic-Specific Focus: Addresses unique challenges of Arabic language processing in education
• Inter-Rater Reliability: Robust validation ensuring consistent evaluation across evaluators
• Comparative Framework: Systematic comparison of multiple Arabic LLMs

Inter-Rater Reliability Analysis

Statistical validation of evaluation consistency across human raters

Research Team

Led by Dr. Shaimaa Lazem (SRTA-City) and Dr. Christine Basta (Polytechnic University of Catalonia), in collaboration with undergraduate students from Alexandria University. My role focused on framework design, experimental setup, and analysis.

Code Repository

github.com/yosftag1/arabic-llm-evaluation-framework

Publication Status

Under Review at Language Resources and Evaluation (Q1)
Preprint available upon request