CrammedBERTurk: Pretraining/Finetuning a New Language Model for Turkish Question Answering on Limited Budget

Abstract

A comprehensive evaluation of transformer-based models for Turkish Question Answering (QA) is conducted, introducing the novel pretraining and fine-tuning of CrammedBERTurk for the first time in this domain. The CrammedBERTurk model was pretrained on a single consumer GPU within 48 hours, showcasing efficient language model (LM) training under constrained computational resources. In contrast, pretraining BERT-base required 16 TPUs over four days. For Turkish QA, CrammedBERTurk was compared against BERTurk, XLM-RoBERTa, and ALBERT using Exact Match (EM), F1, ROUGE scores, and LLM-as-Judge assessment. CrammedBERTurk achieved competitive results compared to BERTurk and, in some cases, outperformed it across multiple datasets. On TQuAD, CrammedBERTurk achieved an EM score of 67.94% and an F1 score of 85.21%, representing a relative improvement of 4.3% in EM and 4.8% in F1 compared to BERTurk. Similarly, on THQuAD, CrammedBERTurk set a new state-of-the-art with an EM score of 69.4% and an F1 score of 86.58%, marking a relative improvement of 8.3% in EM and 5.3% in F1 over BERTurk. On the closed-domain EMUQuAD dataset, it achieved an EM score of 41.32% and an F1 score of 73.73%. The study provides insights into developing efficient transformers for low-resource languages under constrained computational budgets.

Affiliated Institutions

• Eastern Mediterranean University CY

Related Publications