An interdisciplinary team at the College of Staten Island led by Dr. Kristen Gillespie (Psychology) and Dr. Deborah Sturm (Computer Science) are developing a game using the Kinect interface to help autistic people collaborate with their siblings and peers and recognise complex emotions. More generally, this video game aims to help autistic youth improve their social understanding and collaboration skills in two ways:
Along with Computer Science and Psychology students, autistic college students are involved throughout the project in designing and evaluating the game.
“The meaningful inclusion of autistic college students in the design and evaluation of this game allows them to gain valuable employment readiness skills and also ensures that the game is well-matched to the interests of the individuals it is designed to support.” says Dr. Gillespie-Lynch.
The video-game builds on FaceSay, a computer-based face processing training program that has consistently been associated with improvements in the real-world social skills of autistic children. Standing near one another, participants complete collaborative emotion matching puzzles by moving images on a screen using Kinect technology.
“Because participants are acting together upon a digital world while standing next to one another in the ‘real world’, they have many opportunities to engage with one another in person while receiving digital scaffolding to help them solve increasingly complex collaborative emotion matching tasks.” adds Dr. Gillespie-Lynch.
Each puzzle depicts the outline of a figure in an emotionally valenced context (e.g., a bear in the background to depict fear). Once the players have constructed the body of the figure, they must agree on the correct emotion for the face of the figure by choosing from three displayed emotions. The games come with increasingly complex levels of difficulty (in terms of emotions) while the emotions depicted shift from cartoons to photorealistic displays. Additionally, audio-visual scaffolding and feedback are provided to support participants in completing increasingly more difficult emotion matching tasks.
More details about the game design and formative evaluation with autistic college students and their mentors can be found in the Gillespie-Lynch et al. (2017) paper, referenced bellow. Some of the findings from the formative evaluation studies suggest that the participants were highly visually engaged with the screen and they typically attended to the screen for the entire duration of each observation period. Although they rarely looked at their partners during game play, 60% of the autistic students looked at their partner at least once. Moreover, generally, participants were not familiar with how to grab, move and release objects, suggesting the importance of providing tutorials for helping them practice using the Kinect interface.
Usability evaluation is planned to determine if the Kinect-based game is more engaging than identical in-person emotion-matching tasks and gather further user feedback from dyads of autistic college and high school students. Similarly, more focused and intensive evaluation studies are planned to examine benefits of repeated engagement with the program for minimally verbal younger autistic individuals and their siblings. More details about the next steps in this project can be found in the reference bellow.
The researchers are interested in identifying potential collaborators who specialise in Kinect, human-computer interaction, game-based learning, and/or technological interventions for minimally verbal people on the spectrum. If you are interested, please, email Dr. Gillespie-Lynch directly, at Kristen.Gillespie@csi.cuny.edu.
Gillespie-Lynch, K., Goldstein, G., Smith, D. S., Riccio, A., Kholodovsky, M., Merendino, C., … & Sturm, D. (July, 2017). Connecting through Kinect: Designing and Evaluating a Collaborative Game with and for Autistic Individuals. To appear In the International Conference on Human-Computer Interaction. Springer, Cham.
This work is funded by a CUNY Interdisciplinary Research Grant (IRG) Program, entitled “Connecting through Kinect: Evaluating a Game to Support Emotion Recognition and Collaboration among Autistic Individuals” awarded to Dr. Gillespie and Dr. Sturm.