PIs: Nicholas Giudice, Justin Dimmel, Stacy Doore
University of Maine
Award Details

There are many limitations for students who are blind or visually impaired (BVI) in accessing complex STEM graphical information in the classroom or workplace. This longstanding problem arises due to reliance on inaccessible and outdated learning materials, the need for costly specialized devices, and an adherence to an outdated educational service model. To address these issues, this project will investigate the development and evaluation of an innovative remote learning system based on the use of multiple sensory channels to strategically present information from auditory, linguistic, touch, and enhanced visual sensing. The research will focus specifically on the optimization of multimodal information presentation and perception, separating sensory output based on its unique information processing characteristics for conveying different types of stimuli. The first project goal is to increase the quality of STEM instruction for BVI students by determining perceptually motivated learning supports that promote non-visual knowledge acquisition of STEM graphical and spatial information (learning goal). The second project goal is to increase access to graphical and spatial STEM content through creation of an innovative remote multimodal interface for communicating the conceptual meaning of visual information (technology goal). The project outcomes will contribute to theories of non-visual learning and multisensory processing, and a clear translational path to development of more efficient, intuitive, and usable multimodal interfaces for both blind and sighted users. The application of the results will help to address the severe under-representation of BVI individuals in STEM-related disciplines, and the 70% unemployment rate of this demographic, by providing a new, low-cost, and accessible technology platform for communicating non-visual graphical STEM materials.

The researchers will answer the following inter-connected questions: 1) What is the best information content to be conveyed by different modal outputs for maximizing perceptual saliency, learnability, interpretation, and representation of STEM graphical materials? Once optimized in the lab, 2) How well does the optimized multimodal learning system perform in a remote deployment environment in conveying graphical STEM materials to BVI learners; and 3) Does the remote learning system increase the level of comprehension of STEM graphical content as compared to traditional BVI instructional methods? Both quantitative and qualitative data about the optimization process and the remote technology system will be collected and analyzed including user response metrics on speed/accuracy, user experience data, and STEM graphical assessment instruments adapted for BVI students. The first phase of the research will investigate multimodal information processing in order to establish best practices for information delivery and non-visual graphical learning efficiency with experiments comparing graphical information presented in different modalities for three core STEM graphical themes: graphs, diagrams, and maps. The second phase of experiments will investigate the remote learning system’s efficacy as well as evaluating user performance on graphical STEM learning measures and key usability and satisfaction metrics. This project has the broader goals of increasing independence for BVI learners and other students with or without disabilities who might benefit from a remote multimodal learning environment, and the development of a new tool for supporting large-scale research and assistive technology evaluation with BVI human subjects, thereby dramatically increasing scientists’ ability to recruit and work with a much larger population of BVI users than is currently possible from lab-based studies.