Overview

For innovative educational programs, products, and policies to be implemented successfully across diverse settings, they need to be adapted to address the unique needs and characteristics of those environments. Design-Based Implementation Research (DBIR) is a systematic approach for working with stakeholders to iteratively design research that investigates the implementation process so that it answers questions of importance in the local context. DBIR does not specify a particular method or analytic approach, recognizing that a range of different methods is appropriate in different circumstances and in different phases of the research and development lifecycle. The goal of this form of research is to build the capacity of local stakeholders to improve their practice. DBIR grew out of Design-Based Research (DBR), a methodological approach created by learning scientists in which researchers investigate solutions to educational problems by designing interventions and iteratively testing them in real-world settings to see how they function in formal or informal educational settings. DBIR draws on this approach, but rather than exploring how a specific intervention works to address a particular educational challenge, it explores how to design an implementation process that addresses multiple challenges within a complex system.

The DBIR approach was established in response to evidence that research-based innovations are often difficult to sustain or use at scale in real-world classrooms, schools, and districts, even when they proved effective in small-scale studies. This is because traditional scale-up approaches did not always take into account the critical role that local adaptation and stakeholder commitment play in successful implementation of innovative programs, products, and policies. An example of a DBIR project is CSR Colorado, a collaboration between Denver Public Schools and researchers at University of Colorado, Boulder. The district wanted to integrate an innovative program called Collaborative Strategic Reading that was developed previously. The researchers and district worked together to co-design a process for adapting this program to work within the norms of a large, diverse district and conducting research to ensure that the program was achieving the district’s goals.

Four principles underlie DBIR:

• A focus on persistent problems of practice, as experienced from multiple stakeholders’ perspectives;
• A commitment to iterative, collaborative design in realistic contexts;
• A concern with developing theory and knowledge related to both classroom learning and implementation through systematic inquiry; and
• A concern with developing capacity for sustaining change in educational systems.

Key Lessons

Key lessons from cyberlearning research:

• Researchers engaged in DBIR not only investigate and build theory about an innovation’s impact on learning, but also about how and why an innovation is implemented differently in different settings.
• DBIR researchers use techniques such as program mapping to understand the organizational structures of the settings into which innovations are introduced.
• Researchers engaged in DBIR not only iteratively and collaboratively design programs with stakeholders, but they also iteratively and collaboratively design the implementation plan, the outcomes, and the measures.
• Scaling an innovation requires building capacity within systems to sustain that innovation and transfer ownership to local stakeholders, using techniques such as establishing professional routines and supporting communication networks.
• By looking at implementation variation across multiple contexts, DBIR projects can inform policy-making at the system level, which can enhance sustainability.

Issues

Establishing trusting partnerships. Productive DBIR work requires trusting relationships among researchers, practitioners, and district leaders, but trust is something that usually is only built over time. It can be difficult for researchers to initiate DBIR projects with new stakeholders because they do not have those existing relationships. Educators are used to being evaluated, but are not used to engaging with and helping develop a research agenda. Researchers often are not aware of the multiple, conflicting demands on educators that can make it difficult to prioritize research. DBIR researchers need to establish practices that help all parties to build confidence in each other and the process.

Iterative research design. Traditional research is conducted with a set of research questions and a fully articulated research and analysis plan before the research begins. DBIR uses an iterative approach to research that is responsive to the interests of stakeholders and to the interim findings of the research. This is an approach that is not always understood or valued by funders–who may want to know exactly what the research will look like before they fund it–or other researchers–who may question why input from people without research expertise influences the research design. This can make it challenging to find funding for DBIR research and to publish about it in peer reviewed journals. In addition, to be flexible enough to select the most appropriate research techniques for the task at hand requires DBIR researchers to have a breadth of knowledge about a wide range of methods that many researchers do not have.

Clash of cultures. DBIR projects can be challenging because of the differences between the expectations, norms, and incentives in the cultures of school districts and universities or research institutions. For example, turnover of educators and administrators in large districts can make it difficult to keep partnerships going. District priorities and policies can change dramatically when leadership changes, which may lead to calls for different kinds of research or even a lack of interest in research. If they are planning to use research to inform practice, school districts may want findings more quickly than researchers are accustomed to producing. In addition, higher education institutions often do not reward faculty who engage in these kinds of long-term, evolving partnerships, instead favoring high publication rates and specialization of their research focus.

Projects

Examples of NSF Cyberlearning projects that overlap with topics discussed in this primer (see project tag map).

Design-based Research

• Cyberlearning: Sensei: High-Fidelity, Non-Invasive Classroom Sensing for Professional Development
• EXP: Readily Available Learning Experiences: Turning the Entire Web into Progressive Examples to Bridge Conceptual Knowledge Gaps for Novice Web Developers
• NetStat: EAGER: A Representation and Communication Infrastructure for Classroom Collaboration in Data Modeling and Statistics
• EXP: Inclusive Design for Engaging All Learners (IDEAL): Designing Technology for Cultural Brokering
• CAREER: Designing a New Nexus: Examining the Social Construction of Electronics and Computing Toolkits to Broaden Participation and Deepen Learning

More posts: design-based-research

Resources

Design as Scholarship: Case Studies from the Learning Sciences, edited by Vanessa Svihla & Richard Reeve. For researchers in the learning sciences, there is sparse literature how we actually go about designing. Design as Scholarship: Case Studies from the Learning Sciences addresses this need by providing design stories of how researchers actually do their work—how they identified and met needs, how they collaborated across disciplinary boundaries, and how they took advantage of emergence or opportunism in their work. The book includes chapters on designing technologies for learning, community co-design, and more.

Learndbir.org. This website provide a wide range of tools and instruments for engaging in DBIR, as well as examples of DBIR projects taking place in various contexts, and publications on DBIR studies.

Strategic Education Research Partnership (SERP) Institute. This institute was established to foster long-term relationships among researchers, practitioners, and designers so they can work together to develop solutions to challenging educational problems. The site provides information about their projects as well as the educational products and programs that have resulted from those initiatives.

Carnegie Foundation for the Advancement of Teaching. This organization has been a leader in improvement science and networked improvement communities (NICS). The site provide resources that explain the principles of improvement science and examples of successful efforts to employ those principles at scale.

Readings

References and key readings documenting the thinking behind the concept, important milestones in the work, foundational examples to build from, and summaries along the way.

See also: publications from NSF-funded cyberlearning projects.

Key Readings

Fishman, B., Penuel, W. R., Allen, A., & Cheng, B. H., & Sabelli, N. (2013). Design-Based Implementation Research: An Emerging Model for Transforming the Relationship of Research and Practice. In Fishman, Penuel, Allen, & Cheng (Eds.), Design-based implementation research: Theories, methods, and exemplars. National Society for the Study of Education Yearbook, Vol. 112(2), pp. 136-156. New York: Teachers College Record.

Fishman, B. J., Penuel, W. R., Allen, A. R., & Cheng, B. H. (Eds.). (2013). Design-based implementation research: Theories, methods, and exemplars. New York: Teachers College Record.

Penuel, W., Fishman, B., Cheng, B., & Sabelli, N., (2011). Organizing research and development at the intersection of learning, implementation, and design. Educational Researcher, 40(7), 331–337.

Bryk, A. S., Gomez, L. M., & Grunow, A. (2011). Getting ideas into action: Building networked improvement communities in education. Stanford, CA: Carnegie Foundation for the Advancement of Teaching.

Publications from NSF-funded Cyberlearning Projects

Clarke, P. J., Pava, J., Davis, D., Hernandez, F., & King, T. M. (2012). Using WReSTT in SE courses: An empirical study. In Proceedings of the 43rd ACM Technical Symposium on Computer Science Education, (pp. 307-312). New York, New York: Special Interest Group in Computer Science Education.

Goswami, A., Walia, G. S., & Abufardeh, S. (2014). Using a Web-Based Testing Tool Repository in Programming Course: An Empirical Study. In Proceedings of the International Conference on Frontiers in Education: Computer Science and Computer Engineering (p. 1). Las Vegas, Nevada: Computer Engineering and Applied Computing.

Clarke, P., Davis, D., Lau, R., King, T. (2014). Student Learning and Use of Tools in an Undergraduate Software Testing Class. In Proceedings of the 121st ASEE Annual Conference & Exposition. Indianapolis, IN: American Society for Engineering Education.

Ding, M. (2016). Developing preservice elementary teachers’ specialized content knowledge: the case of associative property. International Journal of STEM Education, Vol. 3(1), pp. 1.

Johnson, R., Severance, S., Leary, H., Miller, S. (2014). Mathematical tasks as boundary objects in design-based implementation research. In Polman, J. L., Kyza, E. A., O’Neill, D. K., Tabak, I., Penuel, W. R., Jurow, A. S., O’Connor, K., Lee, T., and D’Amico, L. (Eds.), Learning and becoming in practice: The International Conference of the Learning Sciences (ICLS) 2014, Vol. 2 (pp. 1127-1131). Boulder, CO: International Society of the Learning Sciences.

Johnson, R., Leary, H., Severance, S., Penuel, W. R., Sumner, W., Devaul, H., & Dibie, O. (2014, November). Capacity for Customization: Algebra Teachers, Curriculum Design, and Common Core. Poster session presented to University of Colorado Academic Affairs, Boulder, CO.

Hellmann, J. D. (2015). DataSnap: Enabling Domain Experts and Introductory Programmers to Process Big Data in a Block-Based Programming Language (Doctoral dissertation). Retrieved from Virginta Tech Elextronic Theses and Dissertations. (16249). Retrieved from http://hdl.handle.net/10919/54544.

Citation

Primers are developed by small teams of volunteers and licensed under a Creative Commons Attribution 4.0 International License.

Suggested citation (originally published 2014; updated 2019);

Fishman, B., Cheng, B., & Penuel, W. (2019). CIRCL Primer: Design-Based Implementation Research. In CIRCL Primer Series. Retrieved from http://circlcenter.org/dbir/

After citing this primer in your text, consider adding: “Used under a Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/).”