Presentation
PS7 - Inferring Comprehension of Social Interaction from Eye Tracking in Neurodiverse Individuals in Rural Settings
SessionPoster Session 1
DescriptionBackground:
Successful social interaction is important for individuals of all ages, and can increase longevity by ~50% (Holt-Lunstad et al., 2010). Cognitive abilities mediating social interaction include the awareness that other individuals have different thoughts, feelings, and points of view than one's own. This cognitive construct is termed “Theory of Mind” (ToM). ToM develops gradually in neurotypical children at 3-5 years old. ToM is also impaired in some neurodivergent individuals, e.g., Autism Spectrum Disorders (ASD) and/or Attention Deficit Hyperactivity Disorder (ADHD), but not in language impairments (e.g., Developmental Language Disorder (DLD)). Usually, ToM is measured using comprehension of verbal stories placing additional cognitive and language processing burden on participants. Indeed, language impairment can negatively affect ToM (Durrleman et al., 2022). However, using a non- (low-) verbal ToM task in autistic vs. DLD individuals can reliably indicate level of ToM separately from language skills (Colle et al., 2007). Training with social communication may improve ToM in ASD (Gabbatore et al., 2022).
Thus, non-/low- verbal ways of measuring ToM are important, as a proxy for social interaction, especially in 25-35% of ASD individuals who are minimally verbal or nonspeaking (Hughes et al., 2023). Using eye tracking technology enables examination of participant's attentional processes based on where they look at key timepoints of social interaction as indication of their ToM capabilities. This ensures inclusion of individuals with a wide range of cognitive abilities, and allows investigation of the relationship between ToM, language, and ASD. We include rural individuals with ASD who receive less services which might exacerbate cognitive challenges and outcomes (Gupta et al., 2023).
Methods:
Using participatory design and human-centered design processes, we adapted Colle et al.’s (2007) nonverbal (low-verbal) ToM task as 26 videos (to ensure consistency in presentation), shown to participants while their eye gaze and non/verbal answers were recorded. These videos are 15-60 seconds long and feature nonverbal social interaction (mediated by head nods and looking to each other) between Hider hiding toy Bear under one of two boxes out of view of participant, and Communicator communicating via pointing to the location of Bear determined by what they can see. Then, Hider asked participants, “Where’s Bear?”, who then responded by either pointing or verbally indicating which box had Bear. Pilot testing suggested that Hider’s eye gaze sometimes gave away the location of Bear, so Hider’s eyes during hiding phrases were digitally masked in videos.
Three sets of conditions were tested: pre-test, control, and belief. Pre-test conditions, visible and invisible hiding (behind a screen), taught participants that Communicator is trustworthy and tested whether participants understood the basic principles of ToM game. Control conditions established if participant was able to keep track of Bear if its location was switched, regardless of what Communicator indicated. Finally, belief conditions tested participant’s grasp of ToM indicated by their ability to use information from Communicator.
The key condition is false-belief (FB), where Hider switched the box locations while Communicator stepped out. When Communicator returned, they falsely indicated the location of Bear, based on their own knowledge that did not match reality. To show intact ToM, participant must correctly infer that Communicator is false and identify the correct box. In contrast, in true belief condition, boxes are switched in view of Communicator, so their answer is correct, and in control belief, no box switch occurs while Communicator is away. During videos, we used Gazepoint eye tracking software, with analysis in RStudio (eyetrackingR and Stargazer). All research was conducted either in participants’ homes or university classroom settings.
Participants:
We evaluated 30 child participants in rural and urban settings, ages 1.9 to 17, 11 with ASD (some with additional DLD or ADHD), 7 without ASD but with DLD or ADHD, and 12 neurotypical (some siblings). 7 adults (ages 18-31 years, 4 autistic and 3 neurotypical) were also tested, and showed successful knowledge of all ToM conditions. Eyetracking was only successfully analyzed for 12 child participants, ages 6 to 17, 6 with ASD (including 1 nonspeaking), 2 without ASD but DLD or ADHD, and 4 neurotypical.
Behavioral results:
Behavioral analysis in child participants of the mean correct scores on FB condition, separated by diagnosis (ASD vs. non-ASD) revealed a difference approaching significance (p=.06). ASD group (N=11, Mean Chronological Age=10.4, Mean Nonverbal Mental Age=10.6 years) scored 45% (SE=13%) correct on false belief. But non-ASD group (N=19, Mean CA=7.9, Mean NVMA=9.9) scored 73% (SE=6%) for FB. Both groups passed pretest and control conditions (63-95% correct), and these groups only differed in their levels of receptive vocabulary percentile (p=.04), and caregiver questionnaires: SCQ autism questionnaire and CCC-2 General Communication Composite score (p<.0001). Their levels of nonverbal IQ (KBIT), receptive grammar (TACL), executive function (nonverbal motor inhibition “Luria’s hand game”), phonological working memory (CNRep), chronological age, and rurality (RUCA) did not differ.
However, generalized linear mixed models across diagnoses for the whole child group, with grouped binomial for number of correct behavioral responses, and ToM conditions and other factors as predictors, revealed significant effect of ToM conditions (p<.001) and nonverbal IQ raw score (p<.001). Posthoc analyses revealed lower ToM scores in participants with lower nonverbal IQ raw scores (indicating ToM challenges in younger participants or those with more cognitive challenges). All other factors, including diagnoses, were not significant.
Eye tracking results:
Eye tracking results are consistent with behavioral results. Of 12 child participants who were analyzed, on behavioral responses to FB condition, 9 passed (including 3 with ASD, 2 with ADHD, 3 neurotypical, 1 DLD) and 3 failed (all with ASD and language impairment). We focused on the time frame from when final question of “Where’s Bear?” was posed including time of participant's answer, generating ArcSin values (proportion of eye gaze collapsed across 5 seconds). In FB condition, Pass group demonstrated more gaze to Areas of Interest (AOI) of Hider’s face (49%), distractor box (49%, vs. 30% in other conditions), and target box (46%), and less to Communicator’s face (16%). These numbers indicate participants’ engagement with Hider, some attention to incorrect location (thereby acknowledging awareness of false information from Communicator), and certainty of final answer. Linear model of ArcSin revealed no effect of conditions, but significant effect of AOI (p<.0001).
In contrast, Fail group in FB condition showed more looks to both Communicator and Hider (48% each), with only 14% to target box and 20% to distractor box. Also, for control belief condition, Fail group showed equal distribution of eye gaze towards Communicator, Hider, and distractor box (25-35%), but only 8% to target box. These numbers indicate that Fail participants are aware that they do not know where Bear is in FB condition, and engage with (pay attention to) actors in video to find the answer. However, distribution of looks in control belief and other conditions suggest that Fail group may not engage consistently with the task. Linear model of ArcSin showed neither condition nor AOI were significant, suggesting random gazing behaviors.
Conclusions:
We find no evidence of correlation between diagnoses of ASD, ADHD, or DLD with ability to successfully complete nonverbal ToM task. Only nonverbal IQ significantly predicted the ability to pass ToM. Eye-tracking was informative for studying attentional patterns in Pass vs. Fail groups, indicating inconsistent social engagement in Fail group. However, successful eye tracking analysis was hindered by age as participants under 6 years were unable to remain stationary enough to record over 50% of gaze. Next steps of this project will use EEG (electroencephalography) to record changes in brain activity during observing social interaction, with the goal of tailoring possible interventions to needs of participants with diverse cognitive abilities.
Successful social interaction is important for individuals of all ages, and can increase longevity by ~50% (Holt-Lunstad et al., 2010). Cognitive abilities mediating social interaction include the awareness that other individuals have different thoughts, feelings, and points of view than one's own. This cognitive construct is termed “Theory of Mind” (ToM). ToM develops gradually in neurotypical children at 3-5 years old. ToM is also impaired in some neurodivergent individuals, e.g., Autism Spectrum Disorders (ASD) and/or Attention Deficit Hyperactivity Disorder (ADHD), but not in language impairments (e.g., Developmental Language Disorder (DLD)). Usually, ToM is measured using comprehension of verbal stories placing additional cognitive and language processing burden on participants. Indeed, language impairment can negatively affect ToM (Durrleman et al., 2022). However, using a non- (low-) verbal ToM task in autistic vs. DLD individuals can reliably indicate level of ToM separately from language skills (Colle et al., 2007). Training with social communication may improve ToM in ASD (Gabbatore et al., 2022).
Thus, non-/low- verbal ways of measuring ToM are important, as a proxy for social interaction, especially in 25-35% of ASD individuals who are minimally verbal or nonspeaking (Hughes et al., 2023). Using eye tracking technology enables examination of participant's attentional processes based on where they look at key timepoints of social interaction as indication of their ToM capabilities. This ensures inclusion of individuals with a wide range of cognitive abilities, and allows investigation of the relationship between ToM, language, and ASD. We include rural individuals with ASD who receive less services which might exacerbate cognitive challenges and outcomes (Gupta et al., 2023).
Methods:
Using participatory design and human-centered design processes, we adapted Colle et al.’s (2007) nonverbal (low-verbal) ToM task as 26 videos (to ensure consistency in presentation), shown to participants while their eye gaze and non/verbal answers were recorded. These videos are 15-60 seconds long and feature nonverbal social interaction (mediated by head nods and looking to each other) between Hider hiding toy Bear under one of two boxes out of view of participant, and Communicator communicating via pointing to the location of Bear determined by what they can see. Then, Hider asked participants, “Where’s Bear?”, who then responded by either pointing or verbally indicating which box had Bear. Pilot testing suggested that Hider’s eye gaze sometimes gave away the location of Bear, so Hider’s eyes during hiding phrases were digitally masked in videos.
Three sets of conditions were tested: pre-test, control, and belief. Pre-test conditions, visible and invisible hiding (behind a screen), taught participants that Communicator is trustworthy and tested whether participants understood the basic principles of ToM game. Control conditions established if participant was able to keep track of Bear if its location was switched, regardless of what Communicator indicated. Finally, belief conditions tested participant’s grasp of ToM indicated by their ability to use information from Communicator.
The key condition is false-belief (FB), where Hider switched the box locations while Communicator stepped out. When Communicator returned, they falsely indicated the location of Bear, based on their own knowledge that did not match reality. To show intact ToM, participant must correctly infer that Communicator is false and identify the correct box. In contrast, in true belief condition, boxes are switched in view of Communicator, so their answer is correct, and in control belief, no box switch occurs while Communicator is away. During videos, we used Gazepoint eye tracking software, with analysis in RStudio (eyetrackingR and Stargazer). All research was conducted either in participants’ homes or university classroom settings.
Participants:
We evaluated 30 child participants in rural and urban settings, ages 1.9 to 17, 11 with ASD (some with additional DLD or ADHD), 7 without ASD but with DLD or ADHD, and 12 neurotypical (some siblings). 7 adults (ages 18-31 years, 4 autistic and 3 neurotypical) were also tested, and showed successful knowledge of all ToM conditions. Eyetracking was only successfully analyzed for 12 child participants, ages 6 to 17, 6 with ASD (including 1 nonspeaking), 2 without ASD but DLD or ADHD, and 4 neurotypical.
Behavioral results:
Behavioral analysis in child participants of the mean correct scores on FB condition, separated by diagnosis (ASD vs. non-ASD) revealed a difference approaching significance (p=.06). ASD group (N=11, Mean Chronological Age=10.4, Mean Nonverbal Mental Age=10.6 years) scored 45% (SE=13%) correct on false belief. But non-ASD group (N=19, Mean CA=7.9, Mean NVMA=9.9) scored 73% (SE=6%) for FB. Both groups passed pretest and control conditions (63-95% correct), and these groups only differed in their levels of receptive vocabulary percentile (p=.04), and caregiver questionnaires: SCQ autism questionnaire and CCC-2 General Communication Composite score (p<.0001). Their levels of nonverbal IQ (KBIT), receptive grammar (TACL), executive function (nonverbal motor inhibition “Luria’s hand game”), phonological working memory (CNRep), chronological age, and rurality (RUCA) did not differ.
However, generalized linear mixed models across diagnoses for the whole child group, with grouped binomial for number of correct behavioral responses, and ToM conditions and other factors as predictors, revealed significant effect of ToM conditions (p<.001) and nonverbal IQ raw score (p<.001). Posthoc analyses revealed lower ToM scores in participants with lower nonverbal IQ raw scores (indicating ToM challenges in younger participants or those with more cognitive challenges). All other factors, including diagnoses, were not significant.
Eye tracking results:
Eye tracking results are consistent with behavioral results. Of 12 child participants who were analyzed, on behavioral responses to FB condition, 9 passed (including 3 with ASD, 2 with ADHD, 3 neurotypical, 1 DLD) and 3 failed (all with ASD and language impairment). We focused on the time frame from when final question of “Where’s Bear?” was posed including time of participant's answer, generating ArcSin values (proportion of eye gaze collapsed across 5 seconds). In FB condition, Pass group demonstrated more gaze to Areas of Interest (AOI) of Hider’s face (49%), distractor box (49%, vs. 30% in other conditions), and target box (46%), and less to Communicator’s face (16%). These numbers indicate participants’ engagement with Hider, some attention to incorrect location (thereby acknowledging awareness of false information from Communicator), and certainty of final answer. Linear model of ArcSin revealed no effect of conditions, but significant effect of AOI (p<.0001).
In contrast, Fail group in FB condition showed more looks to both Communicator and Hider (48% each), with only 14% to target box and 20% to distractor box. Also, for control belief condition, Fail group showed equal distribution of eye gaze towards Communicator, Hider, and distractor box (25-35%), but only 8% to target box. These numbers indicate that Fail participants are aware that they do not know where Bear is in FB condition, and engage with (pay attention to) actors in video to find the answer. However, distribution of looks in control belief and other conditions suggest that Fail group may not engage consistently with the task. Linear model of ArcSin showed neither condition nor AOI were significant, suggesting random gazing behaviors.
Conclusions:
We find no evidence of correlation between diagnoses of ASD, ADHD, or DLD with ability to successfully complete nonverbal ToM task. Only nonverbal IQ significantly predicted the ability to pass ToM. Eye-tracking was informative for studying attentional patterns in Pass vs. Fail groups, indicating inconsistent social engagement in Fail group. However, successful eye tracking analysis was hindered by age as participants under 6 years were unable to remain stationary enough to record over 50% of gaze. Next steps of this project will use EEG (electroencephalography) to record changes in brain activity during observing social interaction, with the goal of tailoring possible interventions to needs of participants with diverse cognitive abilities.
Authors
Event Type
Poster Presentation
TimeMonday, March 314:45pm - 6:15pm EDT
LocationFrontenac Foyer
Digital Health (DH)
Simulation and Education (SE)
Hospital Environments (HE)
Medical and Drug Delivery Devices (MDD)
Patient Safety and Research Initiatives (PS)