PIs: William Cope, ChengXiang Zhai, Duncan Ferguson, Willem Els
University of Illinois at Urbana-Champaign
Award Details

This project will develop online software tools to assess and offer feedback to learners communicating complex scientific or technical information. “Complex epistemic performance” here refers to knowledge representations in reports or case studies which involve not only facts and theoretical concepts that might produce correct answers but also arguments, interpretations and conclusions that are matters of disciplinary or professional judgment. Development and testing of these tools will take place using clinical case studies in medicine and veterinary medicine. Medical students will write analyses of specific cases of sick people and animals, marshaling evidence and making diagnoses based on this evidence. Peers will offer “second opinions”, followed by revision. Students will receive a combination of human feedback and machine feedback.

The principal technical innovation in this project will be the development and testing of machine learning algorithms that offer useful feedback to learners and support instructor assessment. The software will analyze student-created cases in relation to the case objectives and a clinical evaluation rubric. It will compare review text item by item to the rubric criteria. The rubric-criterion ratings and overall ratings assigned by users (students and instructors) will train the software via a process of supervised machine learning. In this way, the software will be able to make progressively more accurate assessments of new texts, as well as evaluate the quality of peer reviews. The software will also use unsupervised machine learning techniques, highlighting as-yet unclassified patterns that may warrant investigation — in other words, it will ask students and instructors to interpret patterns of response that may be of interest but which they may not have noticed. One key research outcome of this project will be whether and how machine-supported and machine-mediated formative assessment processes improve learning outcomes in science and related professions, as exemplified in university-level medical education. The algorithms developed in this project could also have broader applicability in a wide range of areas of scientific and technical endeavor.