Investigating Students’ Academic Writing Proficiency Using ChatGPT. The Case Study of First Year Master’s Students.

Abstract

Chat Generative Pre-trained Transformer is an AI-based natural language generative model, able to produce humanized texts with a personalized style/type of writing requested. This study investigates the effect of ChatGPT on students’ academic writing proficiency within EFL higher education, where it addresses two significant gaps: a practical gap as no study investigated the effectiveness of ChatGPT as a sentence enhancer, vocabulary generator and ideas expander for academic writing support; the second gap is a contextual gap as no research has been conducted on introducing ChatGPT to academic writing support in the Algerian context. It is hypothesized that students will build upon their objectivity, formality and syntactic complexity, more specifically in terms of objective tone, formal vocabulary, elimination of slangs, idioms, abbreviations, contractions, acronyms, implementing higher noun phrases and longer T-units. A total of 19 Algerian first year master’s students majoring in Didactics and Applied Linguistics were part of an experiment about using ChatGPT as an assistant in sentence enhancement, vocabulary generation and ideas expansion for five session. ChatGPT was available inside the classroom using mobile phones with Internet access. A pre-questionnaire was administrated to the samples in order to identify their writing performance, areas of lacks and strengths as well as their familiarity with AI writing tools. A pre-test was carried out after the pre-questionnaire to examine students’ initial academic writing performance. Over a span of five sessions, students were exposed to ChatGPT as an assistant in their process of writing. A follow-up post-test was administered under identical conditions to the pre-test. Later on, the samples responded to a survey to report their perceptions and opinions about the chatbot, their academic writing improvement as well as the quality of teaching they were exposed to. The results showed that the samples generally showed different level of writing proficiency between the pre and post-test scores. They performed significantly better in the post-test in regard to syntactic complexity, formal vocabulary, exclusion of idioms and slangs, suggesting that ChatGPT can be beneficial in improving academic writing skills to a certain degree. Considering objectivity and the absence of abbreviations, contractions and acronyms, ChatGPT did not achieve the desired outcomes.