Assessing a Graded Writing Assignment in University: Comparison Between Human Evaluation and Evaluation by a Generative Artificial Intelligence Model
Article Sidebar
Main Article Content
Abstract
Generative artificial intelligence (GAI) can support the evaluative judgment exercised by university faculty when assessing students’ written productions. However, its use raises issues that go beyond score concordance alone. To better understand the conditions under which GAI can provide such support, this article presents an exploratory experiment carried out in a university course focused on the mastery of written French. The study compares human evaluation and GAI‑supported evaluation of the same diagnostically oriented written task. The analysis is grounded in a three‑axis framework of evaluative judgment, encompassing the functions of assessment, its stages, and its ethical dimension, understood as inseparable from the evaluator’s responsibility. The results show an overall convergence between the two modes of evaluation in terms of scores and number of identified errors, while revealing significant discrepancies in error identification, categorization, and the sequencing of diagnosis, scoring, and feedback as error complexity increases. The study shows that GAI cannot replace evaluative judgment. Nevertheless, it can support certain evaluative operations under clear methodological, institutional, and ethical guidelines, combined with faculty training and ongoing professional vigilance. It also opens avenues for further research, notably through comparison between different generative models.
Downloads
Article Details
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.
References
Ahangama, N. (2026). Designing assessments in the generative AI era: A tailored assessment framework for ICT tertiary education. International Journal of Educational Technology in Higher Education, 23(9). https://doi.org/10.1186/s41239-026-00582-0 DOI: https://doi.org/10.1186/s41239-026-00582-0
Álvarez Méndez, J. M. (2001). Evaluar para conocer, examinar para excluir. Morata.
An, Y., Yu, J. H. et James, S. (2025). Investigating the higher education institutions’ guidelines and policies regarding the use of generative AI in teaching, learning, research, and administration. International Journal of Educational Technology in Higher Education, 22(10). https://doi.org/10.1186/s41239-025-00507-3 DOI: https://doi.org/10.1186/s41239-025-00507-3
Anistyasari, Y., Hidayati, S. C., Suparji, S., Ekohariadi, E. et Kusumaningtyas, D. A. (2025). Comparing AI and human assessment of academic writing skills: A kappa analysis. E3S Web of Conferences, 645. https://doi.org/10.1051/e3sconf/202564506014 DOI: https://doi.org/10.1051/e3sconf/202564506014
Audran, J. (2024). Cinq enjeux d’évaluation face à l’émergence des IA génératives en éducation. Mesure et évaluation en éducation, 47(1), 6-26. https://doi.org/10.7202/1114564ar DOI: https://doi.org/10.7202/1114564ar
Bardin, L. (2013). L’analyse de contenu. Presses Universitaires de France. https://doi.org/10.3917/puf.bard.2013.01 DOI: https://doi.org/10.3917/puf.bard.2013.01
Bittle, K. et El Gayar, O. (2025). Generative AI and academic integrity in higher education: A systematic review and research agenda. Information, 16(4), 296. https://doi.org/10.3390/info16040296 DOI: https://doi.org/10.3390/info16040296
Black, P. et Wiliam, D. (1998). Assessment and Classroom Learning. Assessment in Education: Principles, Policy & Practice, 5(1), 7-74. https://doi.org/10.1080/0969595980050102 DOI: https://doi.org/10.1080/0969595980050102
Black, P. et Wiliam, D. (2009) Developing the theory of formative assessment. Educational Assessment, Evaluation and Accountability, 21, 5-31. https://doi.org/10.1007/s11092-008-9068-5 DOI: https://doi.org/10.1007/s11092-008-9068-5
Bloom, B. S. (1971). Handbook on formative and summative evaluation of student learning. McGraw-Hill.
Broadfoot, P. (1996). Education, assessment and society: A sociological analysis. Open University Press.
Brookhart, S. M. (2011). Educational assessment knowledge and skills for teachers. Educational Measurement: Issues and Practice, 30(1), 3-12. https://doi.org/10.1111/j.1745-3992.2010.00195.x DOI: https://doi.org/10.1111/j.1745-3992.2010.00195.x
Brookhart, S. M. (2013). Classroom assessment in the context of standards-based reform. Teachers College Record, 115(11), 1-16. DOI: https://doi.org/10.1177/016146811311500805
Dabis, A., et Csáki, C. (2024). AI and ethics: Investigating the first policy responses of higher education institutions to the challenge of generative AI. Humanities and Social Sciences Communications, 11, 1006. https://doi.org/10.1057/s41599-024-03526-z DOI: https://doi.org/10.1057/s41599-024-03526-z
De Ketele, J.-M. (1986). L’évaluation : approche descriptive ou normative? De Boeck.
De Ketele, J.-M. (2006). La recherche en évaluation : propos synthétiques et prospectifs. Mesure et évaluation en éducation, 29(1), 99-118. https://doi.org/10.7202/1086970a DOI: https://doi.org/10.7202/1086970ar
De Ketele, J.-M. et Gérard, F.-M. (2007). La qualité et la validation des évaluations. De Boeck.
De Ketele, J.-M. (2010a). Ne pas se tromper d'évaluation. Revue française de linguistique appliquée, 15(1), 25-37. https://doi.org/10.3917/rfla.151.0025 DOI: https://doi.org/10.3917/rfla.151.0025
De Ketele, J.-M. (2010b). L’évaluation : une problématique complexe. De Boeck.
Collin, S. et Marceau, E. (2023). Enjeux éthiques et critiques de l’intelligence artificielle en enseignement supérieur. Éthique publique, 24(2). https://doi.org/10.4000/ethiquepublique.7619 DOI: https://doi.org/10.4000/ethiquepublique.7619
Conseil de l’innovation du Québec (2024). L’intelligence artificielle générative : comprendre, encadrer et agir. Gouvernement du Québec. https://www.ethique.gouv.qc.ca/publications/iagenerative/
Elshall, A. S. et Badir, A. (2025). Balancing AI-assisted learning and traditional assessment: The FACT assessment in environmental data science education. Frontiers in Education, 10. https://doi.org/10.3389/feduc.2025.1596462 DOI: https://doi.org/10.3389/feduc.2025.1596462
Enseigner à l’UQAM. (s. d.). La démarche évaluative et les valeurs qui s’y rattachent. https://enseigner.uqam.ca/evaluer/demarche-et-valeurs/
García-Varela, F., Nussbaum, M., Mendoza, M., Martínez-Troncoso, C. et Bekerman, Z. (2025). ChatGPT as a Stable and Fair Tool for Automated Essay Scoring. Education Sciences, 15(8), 946. https://doi.org/10.3390/educsci15080946 DOI: https://doi.org/10.3390/educsci15080946
Geçkin, V., Kızıltaş, E. et Çınar, Ç. (2023). Assessing second-language academic writing: AI vs. human raters. Journal of Educational Technology & Online Learning, 6(4), 1096-1108. http://doi.org/10.31681/jetol.1336599 DOI: https://doi.org/10.31681/jetol.1336599
Ghimire, A., Edwards, J. (2024). From Guidelines to Governance: A Study of AI Policies in Education. Dans Olney, A.M., Chounta, IA., Liu, Z., Santos, O.C., Bittencourt, I.I. (dir.) Artificial Intelligence in Education. Posters and Late Breaking Results, Workshops and Tutorials, Industry and Innovation Tracks, Practitioners, Doctoral Consortium and Blue Sky. AIED 2024. Communications in Computer and Information Science (vol. 2151, p.299-307). Springer, Cham. https://doi.org/10.1007/978-3-031-64312-5_36 DOI: https://doi.org/10.1007/978-3-031-64312-5_36
Gipps, C. V. (1994). Beyond testing: Towards a theory of educational assessment. Falmer Press.
Gipps, C. V. (1999). Socio-cultural aspects of assessment. Review of Research in Education, 24, 355-392. DOI: https://doi.org/10.2307/1167274
Hadji, C. (1997a). L’évaluation démystifiée. ESF.
Hadji, C. (1997b). Pour une éthique de l’agir évaluationnel. Mesure et évaluation en éducation, 20(2), 7-26. https://doi.org/10.7202/1091380ar DOI: https://doi.org/10.7202/1091380ar
Hadji, C. (2012). Faut-il avoir peur de l’évaluation? De Boeck. DOI: https://doi.org/10.3917/dbu.hadji.2012.01
Hannachi, R. et Araújo, S. (2025). Vers une évaluation humaine et automatisée de l’écrit avec l’intelligence artificielle générative. Les Langues modernes, 119(1). https://hal.science/hal-05220332v1
Huba, M. E. et Freed, J. E. (2000). Learner-centered assessment on college campuses. Allyn & Bacon.
Ilieva, G., Yankova, T., Ruseva, M. et Kabaivanov, S. (2025). A Framework for Generative AI-Driven Assessment in Higher Education. Information, 16(6), 472. https://doi.org/10.3390/info16060472 DOI: https://doi.org/10.3390/info16060472
Jorro, A. et Mercier-Brunel, Y. (2022). Évaluation. Dans A. Jorro (dir.), Dictionnaire des concepts de la professionnalisation (p. 197-200). De Boeck Supérieur. DOI: https://doi.org/10.3917/dbu.jorro.2022.01.0197
Kemal, S. et Liman Kaban, A. (2024). Comparative Analysis of Human Graders and AI in Assessing Secondary School EFL Journal Writing. Asian Journal of Distance Education, 20(1). https://www.asianjde.com/ojs/index.php/AsianJDE/article/view/796
Kılınç, S. (2024). Comprehensive AI assessment framework: Enhancing educational evaluation with ethical AI integration. Journal of Educational Technology & Online Learning, 7(4), 521-540. https://doi.org/10.31681/jetol.1492695 DOI: https://doi.org/10.31681/jetol.1492695
Koo, T. K. et Li, M. Y. (2016). A Guideline of Selecting and Reporting Intraclass Correlation Coefficients for Reliability Research. Journal of Chiropractic Medicine, 15(2), 155-163. https://jcm.kglmeridian.com/view/journals/jcm/15/2/article-p155.xml DOI: https://doi.org/10.1016/j.jcm.2016.02.012
Leroux, J. L. et Bélair, L. (2015). Exercer son jugement professionnel. Dans J. L. Leroux (dir.), Évaluer les compétences (p. 65-107). Chenelière.
Li, H., Chen, C. H., Fan, K., Young-Johnson, C., Lim, S. ET Feng, Y. (2025). Agreement between large language models and human raters in essay scoring: A research synthesis. arXiv. https://doi.org/10.48550/arXiv.2512.14561
Lund, B. D., Lee, T. H., Mannuru, N. R., et Arutla, N. (2025). AI and academic integrity: Exploring student perceptions and implications for higher education. Journal of Academic Ethics, 23, 1545-1565. https://doi.org/10.1007/s10805-025-09613-3 DOI: https://doi.org/10.1007/s10805-025-09613-3
Lundgren, M. (2024). Large Language Models in Student Assessment: Comparing ChatGPT and Human Graders. arXiv. https://doi.org/10.48550/arXiv.2406.16510 DOI: https://doi.org/10.2139/ssrn.4874359
McDonald, N., Johri, A., Ali, A., Hingle, A., 2024.Generative artificial intelligence in higher education: Evidence from an analysis of institutional policies and guidelines. Computers in Human Behavior: Artificial Humans, 3. https://doi.org/10.1016/j.chbah.2025.100121 DOI: https://doi.org/10.1016/j.chbah.2025.100121
Messick, S. (1989). Validity. Dans R. L. Linn (dir.), Educational measurement (3e éd., p. 13-103). Macmillan.
Messick, S. (1994). The Interplay of Evidence and Consequences in the Validation of Performance Assessments. Educational Researcher, 23(2), 13-23. https://doi.org/10.3102/0013189X023002013 DOI: https://doi.org/10.3102/0013189X023002013
Microsoft (2024). Microsoft Copilot. https://www.microsoft.com/copilot
Miles, M. et Huberman, A. (2009). Analyse des données qualitatives. De Boeck.
Mpolomoka, D. L. (2025). Utilizing artificial intelligence for assessment. Pedagogical Research, 10(3). https://doi.org/10.29333/pr/16677 DOI: https://doi.org/10.29333/pr/16677
OpenAI. (2024). GPT 4 technical report. ArXiv. https://arxiv.org/abs/2303.08774
Organisation de coopération et de développement économiques (OCDE). (2021). Recommandation du Conseil sur l’intelligence artificielle. OCDE. https://legalinstruments.oecd.org/fr/instruments/OECD-LEGAL-0449
Page, E. B. (1968). The use of the computer in analyzing student essays. International Review of Education, 14, 210-225. https://doi.org/10.1007/BF01419938. DOI: https://doi.org/10.1007/BF01419938
Perrenoud, P. (1998). L’évaluation des élèves : de la fabrication de l’excellence à la régulation des apprentissages. De Boeck Université.
Perrenoud, P. (2004). Évaluer des compétences. De Boeck.
Peterson, S. (2025). Addressing student use of generative AI in schools and universities through academic integrity reporting. Frontiers in Education, 10. https://doi.org/10.3389/feduc.2025.1610836 DOI: https://doi.org/10.3389/feduc.2025.1610836
Posit team (2025). RStudio: Integrated Development Environment for R. Posit Software, PBC, Boston, MA. URL http://www.posit.co/
Roblez, A. (2025, 25 février). Intelligence évaluative et évaluation artificielle : réflexions sur la relation entre l’IA et l’évaluation [communication orale]. Webinaire du Réseau de la pédagogie, Institut catholique de Lille. https://hal.science/hal-04967764v1
Santos Guerra, M. A. (2010). La evaluación como aprendizaje. Kairós.
Scallon, G. (2015). Des savoirs aux compétences : exploration en évaluation des apprentissages. De Boeck.
Scriven, M. (1967). The methodology of evaluation. Dans R. W. Tyler, R. M. Gagne et M. Scriven (dir.), Perspectives of curriculum evaluation (p. 39-83). Rand McNally.
Scriven, M. (1991). Evaluation thesaurus (4e éd.). Sage.
Steiss, J., Tate, Tamara., Graham, S., Gruz, J., Hebert, M., Wang, J., Moon, Y., Tseng, W., Warschauer, M., Olsan, C.B. (2024). Comparing human and ChatGPT feedback. Learning and Instruction, 91. https://doi.org/10.1016/j.learninstruc.2024.101894 DOI: https://doi.org/10.1016/j.learninstruc.2024.101894
Tremblay, C., Miklohoun, S. et Poellhuber, B. (2025). Recension des usages d’intelligences artificielles génératives (IAg) pour offrir de la rétroaction en enseignement supérieur. International Journal of Technologies in Higher Education, 22(1). https://doi.org/10.18162/ritpu-2025-v22n1-11 DOI: https://doi.org/10.18162/ritpu-2025-v22n1-11
UNESCO (2023). Guidance for generative AI in education and research. UNESCO.
Wanlin, P. (2007). L’analyse de contenu comme méthode d’analyse qualitative d’entretiens : une comparaison entre les traitements manuels et l’utilisation de logiciels. Recherches qualitatives, hors série, 3, 243-272. https://www.erudit.org/en/books/collection-hors-serie-les-actes-de-la-revue-recherches-qualitatives/bilan-et-prospectives-de-la-recherche-qualitative-en-sciences-humaines-et-sociales/5175co/
Zhao, J., Chapman, E. et Ghassemi Pour Sabet, P. (2024). Generative AI and educational assessments: A systematic review. Education Research & Perspectives, 51(31), 124-155. https://doi.org/10.70953/ERPv51.2412006 DOI: https://doi.org/10.70953/ERPv51.2412006