CIRCL perspectives offer a window into the different worlds of various stakeholders in the cyberlearning community — what drives their work, what they need to be successful, and what they think the community should be doing. Share your perspective.
Neil Heffernan is a professor in the Computer Science Department at Worcester Polytechnic Institute, and directs the PhD and Masters Programs in Learning Sciences and Technologies. Dr. Heffernan is well known for ASSISTments, an online tutoring program that he created. ASSISTments has been used as a shared scientific instrument to conduct randomized controlled trials comparing different ideas on how to improve student learning.
What drives your work?
Scientists and physicists have expensive shared scientific instruments like the Hubble space telescope or particle colliders. In educational research, nothing equivalent exists. We should spend more time focusing on platforms, to enable researchers in cyberlearning to work together.
A decade ago, I started ASSISTments as a platform. Since then, I’ve been investigating research questions around middle school math with funding from NSF and the US Dept. of Education. And I got several of my colleagues and friends to eventually do studies with me. More recently, we’ve made this research platform really easy for others to submit studies. Now we have funding from NSF to help researchers to create a problem set in our world, deliver it to teachers and students, analyze their data, and write up their study. We’ve done webinars and workshops to help people learn to use the platform. The process for doing studies with ASSISTments is also available on our web site. See 5 Stages for Doing Research with ASSISTMents.
What are key strategies for platform projects like yours?
A key thing we’re doing is simple student level randomization. Begin with the assumption that all of your designs are going to be wrong, and run lots of experiments. Most projects in the cyberlearning community that have serious sites, with hundreds of thousands of users, haven’t built out a student-level randomization component. But companies like Google and Facebook are doing user-level randomization. I’m reading a really good paper by folks at Bing; they are running 100 experiments a day. Other platforms, like Pearson, could easily do this. It’s not technically hard. I think it’s politically hard. If you’re a company, you don’t want to tell your clients that you are experimenting on your children. The dominant idea is that “We already know how this goes – this product works”, when in fact all these products are flawed and could all be improved. Too may people just validate that the product is doing what it says it does. Some researchers may be great designers, and some maybe aren’t so great, but the person who is designing stuff usually isn’t the same one who wants to do the hard work of rigorous methodology, including randomization, to see if it works. We’re trying to build an infrastructure to make it easy for designers to put in their ideas and test them rigorously.
Besides helping designers and experimenters, we also want to help content people by assembling a really good list of measures of common core state standards. We’re trying to make it so that so you don’t also have to be a measurement expert to measure student learning, because that’s also yet another skill that is hard to do well. Some people are really good at measuring student learning and understanding, but that’s not the same set of skills needed to design pedagogically useful interventions. To be adopted, a platform needs to be attractive to people across a project team, and also attractive to teachers.
How do you get teachers to adopt your platform?
To be adopted by teachers, we had to build a platform that teachers like. Teachers were part of the design team. My wife and I used to be middle school math teachers. When I began this project a decade ago, I asked some teachers in the Worcester public school system, “How do you want the computer to respond when your children are getting these questions wrong?” I learned from these teachers how they wanted to break a problem down into steps. Then a week later I’d come back with something they could try with their kids. I built a simple authoring tool that allowed us to write the content, hint messages, and quickly involved teachers in the design. Eventually, when I’d bring it in, I’d say that I want to study such-and-such, so I’d like you to give a pre- and post-test.
A lot of work went into building the platform and building up the subject pool. Some of it was marketing our work at teacher conferences to get teachers to become aware of us. Some of the time feels wasted. Not a lot of people have the time necessary to build a product that’s attractive and useful for teachers that doesn’t pay off immediately for their research projects. That’s why it’s sad that we don’t have shared scientific instruments.
What should the cyberlearning community be doing?
We don’t really have a good mechanism by which academics can get credit for building and sharing platforms like this. I happened to be able to pull this off because I’m a tenured professor. But for the young and untenured, how can you afford to “waste” time helping others get their research done? Incentive structures for academics and platforms are not well aligned. I wish that NSF and the US Dept. of Education were more interested in putting out research RFPs that were not just for solo researchers to go do a study, but for research organizations to build tools, where the intellectual merit comes mostly from the researchers who are using it. Good science is needed to build the Hubble space telescope, but mostly the people who build the telescope are not the ones that have the ideas of what to do with it. So I want to see NSF actually fund more projects like ours.
We also need to encourage other platforms to embrace experimental methodology, and also openness. We are wicked-crazy open! All of the data we produce is anonymized. You can get your own data, and you can get others’ data, but it’s all anonymized. We have a really unusually IRB situation. Our IRB is excited about the work we do! In the form that we make researchers fill out, they agree to not de-anonymize the data. Our IRB allows us to give out our data. Some researchers use our tool because we actually protect them from IRB. We make it very easy for them to get their study run. When a researcher does a study, they get an automatically-generated document with their data, and can compare conditions. I sometimes have conversations with them to help them refactor their work so they get more useful comparisons or have a version that more reliably helps kids.
Companies out there don’t enable easy access to data. I have a friend who did a study at a large education company. He had to download their source code, modify it, and work with their production engineers to reboot their whole site with this new code. Then to get the data out, they have to process all these logs. It would take a month just to get the data out. There’s also the publication problem. He’s sitting on a study, but he does not have approval to publish, largely because it’s not a positive result, or really, really small.
If you use ASSISTments, we make it very clear: my name does not go on the paper you publish. If you get your study approved by us, we have no right to interfere. There’s no pre-publication review where I have to first look at your paper that it somehow does not speak negatively towards ASSISTments. I agree to keep their data private for a year, until the study is over, because I want them to be able to publish something about it first. But I also recognize that many of their studies will not be published, for example, if they get a null result. I want all that data to go public eventually.
Some of the things we’re doing are really important to having a shared scientific instrument. A young whippersnapper researcher that NSF wants to encourage just won’t have time to do all these things on their own.