PIs: Heather Pon-Barry
Mount Holyoke College
Award Details

The activities in this Faculty Early Career Development project are rooted in a social educational robot that collaborates with students in STEM learning. The robot acts as a peer learner who asks students to teach them how to approach mathematics and computer science problems. The process of explaining one?s reasoning has the pedagogical benefit of deepening understanding. Educational robots have the potential to reach a broad population of students, but the benefits depend on facilitating strong engagement in the learning activity. In existing research on human-robot collaborative activities, much attention has been directed towards robots that execute natural language commands or respond to stand-alone questions; less attention has been given to robots that sustain longer conversational interactions. This project develops a framework for improving user engagement—social connection to the robot and involvement in the task—in conversational human-robot interaction. Improving human-robot conversational engagement will enable scientific progress from robots that focus on atomic interactions to socially intelligent robots that sustain interaction over time. This project integrates substantial computer science education activities to expand research opportunities for students who are underrepresented in computing, to broaden perspectives of computing by teaching a seminar on talking robots, and to extend the impact of a curriculum established by the investigator for training peer mentors in creating inclusive learning environments.

This project aims to improve educational robots’ ability to sustain student engagement in STEM learning activities by characterizing how multiple modalities of perceptual inputs and of robot actions can be effectively combined in physically-situated interaction. A significant challenge in combining modalities is defining appropriate temporal segments that capture meaningful units and align in a way that allows them to complement one another. The project addresses three interconnected aspects of engagement for social educational robots. First, it explores machine learning methods for estimating engagement using speech, vision, and lexical inputs, with a focus on the temporal dynamics of aligning the perceptual data. Second, the project develops strategies for adaptive response generation based on perceived engagement, drawing on corpus analyses of human-human and human-robot peer-learning dialogues. Third, the project takes a reinforcement learning approach to fostering engagement by learning dialogue policies that integrate verbal and non-verbal robot actions such as head movements, spoken backchannels, and gaze gestures. Taken together, the results provide a foundation for enabling educational robots to interact with greater social intelligence and maintain long-term engagement, and ultimately could transfer to social conversations with robots in other fields of education as well as non-educational domains.