Ambitious Mashups: Research Methods

Authors: Judi Fusco & Patricia Schank; Contributor: Jeremy Roschelle

“I see exciting possibilities in Cyberlearning for thinking about tool design and analytics design together, such that learning environments produce useful data and are designed to take advantage of the data to support teachers and students as well as their own continual self-improvement.”
– Alyssa Wise (excerpt from 6/1/20 reflections on her work and field)

Definition

Cyberlearning projects explore the frontiers of learning with technology, and to do so, the methods they use need to be innovative and informative. While methods are part of every project, they sometimes become the prime focus of the project—for example, when the methods are uniquely innovative or being refined as part of the work. Even established methods can take center stage when they deeply shape the work being done. Because of the interdisciplinarity of Cyberlearning projects, we also see methods from different fields combined in new ways to provide insights to questions. In reviewing the portfolio, the most common or growing methodological approaches found were:

  • Design-Based Research (DBR). Researchers create learning experiences and study them to investigate potential advances and to better understand what the target users and communities need. In Cyberlearning, DBR is a methodology that rigorously explores which design features have the most potential to improve learning (Brown, 1992; Hoadley, 2002).
  • Learning Analytics. Learning analytics can provide insights to the learning process and are often linked to formative assessment, sometimes in Intelligent Tutoring Systems (ITS), to provide students with direct feedback and/or give teachers more information to help their students. In addition, learning analytics have been used to support group problem solving by helping learners acquire content knowledge or to support collaborative processes.
  • Multimodal analytics. Building on learning analytics, multimodal analytics is a method that involves the integration and analysis of multiple data sources (e.g., visual, audio, gestural, movement, eye gaze, heart rate, and/or other types) to help researchers measure learning in new ways and gain insight into learners’ abilities, emotions, needs, and preferences and how these impact learning interventions.

Please see the full report for additional information on how this theme changed over time in cyberlearning research and for some questions that arose as we investigated the theme.


Project Examples and Resources

Design-based Research (DBR)

  • 22 projects
  • 4 sessions at cyberlearning convenings

Stimulating Quotes and Snippets

  • “I use design-based research approaches to co-create learning experiences that promote disciplinary engagement mediated by practitioners’ tools and cyberlearning innovations” – Ale Magana
  • “The curriculum, which being co-designed by teachers, scientists, and curriculum developers, is being implemented in the New York City public schools, the largest school district in the U.S. To me, the most important component is the lasting impact, the sustainability of that impact, and integration within the community.” – Lauren Birney

Example Project Abstracts:

Related Primers/Spotlights/Reports:

Showcase Videos and/or Gallery Posters:

Cross Connections:
AI (Robotics), Emerging / Smart & Connected

Multimodal

  • 11 Projects
  • 5 sessions at cyberlearning convenings

Stimulating Quotes and Snippets

  • “Multimodal Learning Analytics (MLA)… creates a more comprehensive and holistic framework to study learning processes wherever and however they happen.. its main strength is to be able to analyze face-to-face, hands-on, unbounded, and analog learning settings such as classrooms, collaborative groups and labs.” – Xavier Ochoa
  • “MLA opens the door to exploring learning environments, whether classrooms or design studios, that have been difficult to investigate before. New types of sensors and data mining techniques make it possible to capture the process data that is usually lost in the activities in these spaces.” – Marcelo Worsley

Example Project Abstracts:

Related Primers/Spotlights/Reports:

Showcase Videos and/or Gallery Posters:

Cross Connections:
AI, esp. Speech & Robotics, Learning theories, esp. Embodied Learning, Representations

Learning Analytics

  • 34 Projects
  • 14 sessions at cyberlearning convenings

Stimulating Quotes and Snippets

  • “I think people sometimes have a false sense of learning analytics as a set of fully automated techniques that turn raw data into meaningful information. In reality there are three critical things that are often overlooked: The importance of human decision making… The importance of conceptual framing… [and] The importance of understanding the data in depth.” – Alyssa Wise
  • “While issues of data privacy are very much on people’s minds today, it’s important to recognize that using big data doesn’t mean becoming Big Brother- it can become a means for understanding how to tailor learning experiences to meet the needs of every student” – Stephanie Teasley

Example Project Abstracts:

Related Primers/Spotlights/Reports:

Showcase Videos and/or Gallery Posters:

Cross Connections:
AI, ITS, and Machine Learning, Assessment

Other Resources of Note

Exit Survey Highlights (30 total responses)

  • Projects tagged as one or more of the methodological approaches (DBR; Multimodal analytics; Data analytics/data mining): 17
    • DBR: 13
    • Multimodal analytics: 2
    • Data analytics/data mining: 3
    • Some overlap between DBR and data analytics.
  • Gender of PIs: 10/17 Female
    • CL is a place where women are well-represented among PIs
  • Project Implementation Setting:
    • Majority (6/10) of these projects were implemented in either an informal learning setting, or an informal and formal learning setting.
  • Special populations targeted:
    • 9/17 projects specifically targeted a special population (i.e., Learners in special education or with a disability; Learners in low-performing districts or schools; ELLs, women/girls; Underrepresented minorities)
  • Explicit focus on cyberlearning in preparation for and within the context of the work setting: 10/17
    • 6/10: specifically related to supporting the current and future work of teachers in classrooms and other related settings;
    • 4/10: Design and develop future learning environments to educate/re-educate workers for new worker environments and experiences in collaboration with advanced technology.
  • Project included teacher/practitioner partnerships: 5/10
    • Only 50% of these projects included a practitioner partnership, which is a lower percentage than projects addressing other CL themes.
  • Project included 2 or more grad students on project staff: 13/17; CL projects focused on methodological approaches provide good opportunities for engaging and training grad students, with several projects employing 5+ grad students.
  • PI proposals and awards
    • PI received new Cyberlearning award: 5
    • PI submitted proposal to Future of Work program: 3 (more than projects addressing the other CL themes)
    • PI received an NSF award for a program other than Cyberlearning: 8
    • More PIs received awards from other NSF programs, such as CS for All, AISL, STEM+C, and ITEST, than CL awards.
  • Project Publications:
    • 8/17 projects indicated that they published project findings in scholarly journals, including: Technological Innovations in Statistics Education; Journal of Computer Assisted Learning; Multimodal Technologies and Interaction; Journal of Computer-Supported Collaborative Learning; Cognition & Instruction; Journal of the Learning Sciences; Race, Ethnicity & Education; Anthropology & Education Quarterly