Interpretable Multimodal Learning Analytics for Personalized Augmented Reality Vocabulary Learning in Children with Autism Spectrum Disorder: A Secondary Single-Subject Analysis

Abstract

This paper performs a secondary analysis of multimodal learning analytics on augmented reality (AR)-enhanced vocabulary acquisition sessions that have been previously collected from children with autism spectrum disorder (ASD). Although AR provides visual and interactive support for vocabulary learning, standard performance measures offer little information about the differences in learners’ behaviors during and between different intervention stages. Nine children with mild ASD, aged 8–12 years, participated in a single-subject withdrawal design including familiarization, baseline, intervention, and maintenance phases. The proposed Phase-aware Multimodal Bayesian Learning Analytics approach incorporates session-level learning curves, application logs, screen capture videos, audio data, and systematic behavioral observations. A Bayesian hierarchical model was used to estimate personalized effects of intervention and maintenance phases taking into account heterogeneity of participants, dependence of sessions, items’ difficulties, tasks’ types, teacher instructions, and engagement predictors. Seven out of nine participants reached effective or highly effective performance with values of data above the median, however, maintenance effects varied between participants. In the leave-one-participant-out validation, the full multimodal model achieved 0.852 accuracy, 0.814 macro F1-score, 0.889 area under the receiver operating characteristic curve (AUC), 0.148 Brier score, and 0.061 expected calibration error (ECE). Multimodal engagement features improved calibration and interpretation of results, although a sensitivity analysis showed a decrease in performance when the best responding participants were removed.