AI Literacy in Chemistry Education: Rasch Analysis of Pre-Service Chemistry Teachers’ Ability To Evaluate The Conceptual Accuracy of AI-Generated Answers

Muhammad Eka Putra Ramandha; Andi Mutia  Fitri

doi:10.58258/1agh6950

Authors

Muhammad Eka Putra Ramandha Universitas Islam Negeri Mataram
Andi Mutia Fitri Universitas Mataram

DOI:

https://doi.org/10.58258/1agh6950

Keywords:

artificial intelligence; AI literacy; chemistry education; pre-service chemistry teachers; Rasch model; conceptual evaluation.

Abstract

Generative artificial intelligence (AI) has increasingly entered higher education, including chemistry education. Although AI can provide rapid and apparently coherent explanations, its outputs may contain factual, conceptual, representational, and reasoning errors. This study examines pre-service chemistry teachers’ ability to evaluate the conceptual accuracy of AI-generated answers to chemistry problems using the Rasch measurement model. The study was designed as a quantitative descriptive evaluation involving 120 pre-service chemistry teachers and 30 scenario-based items. Each item presented a chemistry problem followed by an AI-generated response containing different levels and types of accuracy. Rasch analysis was used to estimate person ability, item difficulty, reliability, separation, and item fit. The descriptive interpretation focused on the occurrence of response patterns and the characteristics of the most difficult and easiest items. The simulated analysis produced person reliability of 0.87 and item reliability of 0.96, with person and item separation indices of 2.56 and 4.89, respectively. The most difficult items involved organic structure, chemical equilibrium, and acid-base reasoning, whereas factual errors and simple calculation errors were more readily identified. The findings indicate that the ability to use AI should not be equated with the ability to critically evaluate AI outputs. Domain-specific AI literacy in chemistry requires students to mobilize conceptual knowledge and chemical reasoning to verify apparently plausible AI-generated explanations.

Abstract Views 94PDF Downloads 10

References

Clark, T. M. (2023). Investigating the Use of an Artificial Intelligence Chatbot with General Chemistry Exam Questions. Journal of Chemical Education, 100(5), 1905–1916. https://doi.org/10.1021/acs.jchemed.3c00027

Dickson-Karn, N. M., Soltanirad, C., & Tafini, N. (2023). Comparing the Performance of College Chemistry Students with ChatGPT for Calculations Involving Acids and Bases. Journal of Chemical Education, 100(10), 3934–3944. https://doi.org/10.1021/acs.jchemed.3c00500

Dong, Y., Xu, W., Huang, J., & Yann, K. (2025). Validating and refining a multi-dimensional scale for measuring AI literacy in education using the Rasch Model. Humanities and Social Sciences Communications, 12, 1317. https://doi.org/10.1057/s41599-025-05670-6

Erümit, A. K., & Özdemir Sarıalioğlu, R. (2025). Artificial intelligence in science and chemistry education: A systematic review. Discover Education, 4, 178. https://doi.org/10.1007/s44217-025-00622-3

Feldman-Maggor, Y., Blonder, R., & Alexandron, G. (2025). Perspectives of Generative AI in Chemistry Education Within the TPACK Framework. Journal of Science Education and Technology, 34, 1–12. https://doi.org/10.1007/s10956-024-10147-3

Holme, T. A., (2024). Students’ Experience of a ChatGPT Enabled Final Exam in a Non-Majors Chemistry Course. Journal of Chemical Education. https://doi.org/10.1021/acs.jchemed.4c00161

Lintner, T. (2024). A systematic review of AI literacy scales. npj Science of Learning, 9, 50. https://doi.org/10.1038/s41539-024-00264-4

Nayyar, P., & Lewis, S. E. (2025). Why Am I Learning This? Students Discovering the Usefulness of General Chemistry to Their Career Interests via ChatGPT. Journal of Chemical Education, 102(12), 5409–5415.

Priyadi, D., Suyanta, S., Suwardi, S., & Isa, I. M. (2025). Exploring the Role of ChatGPT in Chemistry Learning: A Systematic Literature Review. Jurnal Pendidikan Sains Indonesia, 13(3), 648–662. https://doi.org/10.24815/jpsi.v13i3.45118

Roman, A., Simaitis, M., Sheely, K., Roth, Y. M., Tansey, J. T., & Cogan, J. C. (2025). Learning Tools Using ChatGPT in the Biochemistry Class: Creating Notes and Performance on Exams. Biochemistry and Molecular Biology Education, 53(4), 381–388. https://doi.org/10.1002/bmb.21904

Sorenson, B., & Hanson, K. (2024). Identifying generative artificial intelligence chatbot use on multiple-choice, general chemistry exams using Rasch analysis. Journal of Chemical Education, 101(8), 3216–3223. https://doi.org/10.1021/acs.jchemed.4c00165

Tang, K.-S. (2024). Informing research on generative artificial intelligence from a language and literacy perspective: A meta-synthesis of studies in science education. Science Education. https://doi.org/10.1002/sce.21875

Uçar, S.-Ş., Lopez-Gazpio, I., & Lopez-Gazpio, J. (2025). Evaluating and challenging the reasoning capabilities of generative artificial intelligence for technology-assisted chemistry education. Education and Information Technologies, 30, 11463–11482. https://doi.org/10.1007/s10639-024-13295-6.

West, J. K. (2023). An Analysis of AI-Generated Laboratory Reports across the Chemistry Curriculum and Student Perceptions of ChatGPT. Journal of Chemical Education, 100(11), 4351–4359. https://doi.org/10.1021/acs.jchemed.3c00581

Young, J. D., Dawood, L., & Lewis, S. E. (2024). Chemistry students’ artificial intelligence literacy through their critical reflections of chatbot responses. Journal of Chemical Education, 101(6), 2466–2474. https://doi.org/10.1021/acs.jchemed.4c00154.

Yuriev, E., Orgill, M., & Holme, T. (2025). Generative AI in Chemistry Education: Current Progress, Pedagogical Values, and the Challenge of Rapid Evolution. Journal of Chemical Education, 102(9), 3773–3776. https://doi.org/10.1021/acs.jchemed.5c01105.