PIs: Steven Dow, University of California-San Diego (Award Details)
Brian Bailey, University of Illinois at Urbana-Champaign (Award Details)
Elizabeth Gerber, Northwestern University (Award Details)

Educating the next generation of design innovators is essential for ensuring U.S. economic and social prosperity. To prepare students for this critical work, design innovation instructors rely on project-based learning where learners construct and iterate on preliminary solutions to real-world problems and try to acquire feedback from stakeholders who depend on the outcomes. This project aims to create a scalable method for obtaining formative feedback in the classroom by leveraging online crowds through social media and crowdsourcing platforms via a new online learning platform called CRAFT (CRowd Aided Feedback Technology).

The CRAFT project involves an integrated social science research and technology design plan. Studies will examine how to optimize the acquisition of feedback, how learners do sensemaking of provided feedback, and how and when they can reflect on the feedback in their own learning process. The technology design will utilize an iterative rapid prototyping strategy based on XML templates that will allow quick reconfiguration of interface elements and functionality. Pilot usability testing will be done with face-to-face volunteers, students in courses, and via Amazon’s Mechanical Turk. Students in courses on product design, graphic design, and interface design will serve as participants in a series of studies. A 3×2 experiment will examine the relationship between types of incentives for feedback (intrinsic, financial, and social), design stage (early- or late-stage design), and impact of the feedback on learners. A second, 3×2 within-subjects experiment will examine the relationship between different types of rubrics, early- and late-stage design, and learner impact. A third experiment will compare individual use of scaffolded crowd feedback tools, paired dialogue with annotation tools, and crowd use of annotation tools, and a control condition to examine qualitatively the differences in learner outcomes. The fourth experiment will be a between-subjects analysis of the impact of the tool on design outcomes using both an analysis of the designs themselves and an analysis of a reflective essay submitted by the participants. The CRAFT tool will be iterated and refined on the basis of each of these experiments. A final, summative evaluation will test the final version of the CRAFT environment using a mixed factorial design with independent variables including scaffolding and early- vs. late-stage of design.