PIs: Marcelo Worsley, David Uttal
Northwestern University
Award Details

Spatial reasoning and computational thinking are essential knowledge for future workers in Science, Technology, Engineering and Mathematics (STEM). Researchers in this project will design and study an intelligent multimodal interface to support elementary and middle school students to learn these cognitive skills on a collaborative digital platform based on Minecraft. The new interface will include audio and speech, gaze with eye tracking, and tangible physical objects. The combination of these human-computer interaction modalities may support learning for a broader diversity of learners and facilitate collaboration between peers in a more naturalistic and productive setting. Enabling interacting with computers not only through keyboard commands but also through speech, physical blocks and eye gaze, learners will have the opportunity to practice and improve their spatial reasoning and computational thinking skills in an informal, flexible and accessible context. More broadly, this project investigates the future of multimodal interfaces. Some of the key components of the next-generation multimodal, voice-enabled interface that this project explores include the ability to interpret everyday human language and to express an idea using speech or gesture during collaboration. In addition to developing new technologies, this project examines how to structure different digital learning experiences in ways that promote spatial reasoning and computational thinking. Several hypotheses related to spatial reasoning and computational thinking will be tested through laboratory studies and afterschool learning club studies in partnership with the Evanston Public School District in Illinois. This project is funded by the Cyberlearning for Work at the Human-Technology Frontier program, which seeks to fund exploratory and synergistic research in learning technologies to prepare learners to excel in work at the human-technology frontier.

The research project will investigate four important research questions. (1) In what ways does a multimodal interface enable or promote increased participation in the afterschool game-based learning club among current non-participants, especially younger students, people from underrepresented groups and girls? (2) How might a multimodal interface change existing patterns of behavior among current learners; specifically, in what ways do multimodal interfaces introduce, support, and disrupt collaborative learning and building practices? (3) Do these changes in participation and in-game practices correlate with the development of spatial thinking, computational thinking and the pursuit of computing careers? (4) What design principles and technological and/or algorithmic developments are needed to support collaborative work through multimodal interfaces? Researchers will employ an iterative design process. Design modifications will be informed by mixed-method analyses of video data, surveys, learning assessments, user activity logs, and multimodal sensory data on how learners engage in multimodal, collaborative problem solving. This research has the potential to discover learning pathways to STEM careers for underrepresented young learners. Around 400 students will participate and experience the proposed multimodal learning system during the three years. The resulting software and design documents will be open source for the public in order to reach a much larger number of institutes to broadening participation of STEM learning.