A Comparative Framework for Evaluating Arabic LLMs in Educational Conversational Applications
Published in Language Resources and Evaluation (Q1), 2025
This research presents a comparative framework designed to evaluate Arabic Large Language Models (LLMs) specifically for educational conversational applications, addressing the unique challenges of Arabic language processing in educational contexts.
Evaluation Framework
Comparative evaluation of Arabic LLMs across multiple pedagogical dimensions
Key Contributions
- Novel Rubric-Based Evaluation: Multi-dimensional assessment beyond simple accuracy metrics
- Arabic-Specific Focus: Addresses unique challenges of Arabic NLP in education
- Inter-Rater Reliability: Robust validation ensuring consistent evaluation
Statistical validation of evaluation consistency across human raters
Research Details
Authors: Shaimaa Lazem, Youssef Tageldin, Aly Nasr, et al.
Status: Under Review at Language Resources and Evaluation (Q1 Journal)
Code Repository: github.com/yosftag1/arabic-llm-evaluation-framework
Preprint: Available upon request
Recommended citation: Shaimaa Lazem, Youssef Tageldin, Aly Nasr, et al. (2025). A Comparative Framework for Evaluating Arabic LLMs in Educational Conversational Applications. Language Resources and Evaluation (Under Review).
Download Paper