An interview with Marie Bienkowski, computer scientist and deputy director of SRI International’s Center for Technology in Learning, about the applying evidence centered design (ECD) to create assessments that support valid inferences about computational thinking practices.

What is the big idea of your project work?

Our work focuses on a principled approach to designing assessment tasks that can generate valid evidence of students’ abilities to think computationally. Assessing computational thinking broadly is a long-term goal, but for practical reasons, we’ve begun by assessing computational thinking instantiated in a particular course, Exploring Computer Science (ECS). What is unique about ECS is that it has a focus on creating equal access to computing for students who are traditionally underrepresented in the computing workforce. ECS is typically a 9th or 10th grade course, and you can imagine a pathway through high school for computer science which is ECS, then Computer Science Principles (CSP), the new advanced placement course that was launched this year, and then the regular AP Computer Science A and B. Alternatively, students could take ESC and CSP to get a general overview and good exposure to computer science concepts along with some practical skills in programming. We do get asked the question a lot, “Where is your assessment of computational thinking?” The answer is: it’s hard to assess that independent of a particular curriculum or module or lesson.

The other big idea of our work is that we collaborate with the curriculum developers, who are staunch advocates of equity. Jane Margolis wrote two books on equity: Stuck in the Shallow End: Education, Race, and Computing and Unlocking the Clubhouse: Women in Computing and she influenced the curriculum authors, Joanna Goode and Gail Chapman. They made it very clear in the beginning that they wanted us to build assessments that would push students to do more critical thinking and focus on practices as opposed to rote memorization­­––in other words, to help drive the values they’re trying to teach in the curriculum. They were adamant about no multiple choice; only items that made you think, explain, understand, and do reasoning. We ended up moving toward measuring constructs that the field calls computational thinking practices, which you might think of as a subset of computational thinking. The practices are the “do” part of computational thinking (just as concepts are the “what I know” and “what I do”). The computational thinking practices were what they wanted us to focus on for measurement of learning in ECS.

How did you identify the computational thinking practices?

We ended up adopting the computational thinking practices of CSP, which was developed by a community of K-12 and college instructors in conjunction with the College Board. CSP was developed to address a lowering enrollment in computer science, which was often seen as “just programming.” The computational thinking practices in CSP include:

• Connecting computing to other disciplines
• Creating computational artifacts
• Abstracting and modeling
• Analyzing computational problems and artifacts
• Communicating
• Collaborating

How did you develop your assessments of computational thinking practices?

We began with our typical evidence-centered design (ECD) process, where we do domain analysis and domain modeling, and articulate the assessment argument, implementation, and delivery. So it’s kind of like a software development process where you have your requirements, identify the constraints you’re working with, build out the skeleton, implement it, test, and iterate. By instantiating these computational thinking practices in a particular curriculum, like ECS, you can do measurement.

In our 2015 report, Assessment design patterns for computational thinking practices in secondary computer science: A first look, we walk through our domain analysis and modeling process and describe how these computational thinking practices play out in sample curricula. We describe the sources we used to do the domain analysis, unpack the design patterns, give examples of what’s an observation and what’s a work product, and so on, so you can understand ECD a little more. Our focal knowledge, skills, and abilities (focal KSAs) are mostly abilities, since we focus on practices. Then we move into how this would play out in other projects, such as the the computational thinking activities in the Computational Thinking in STEM (CT-STEM) project at Northwestern University, and curriculum for Scratch, Alice, and AgentSheets. Do the focal KSAs from our design patterns map to these as well? CT-STEM looks at computational thinking in different courses in high school like math, science, physics, and chemistry; our design pattern on “analyze the effects of developments in computing” is applicable there. We also have a design pattern “analyze computational work and the work of others”. When you’re reading someone else’s code, trying to find a bug, that’s what you’re doing. In the Scratch environment, they encourage kids to reuse and remix others’ code, which involves doing analysis of computational work. We could instantiate our “abstractions and modeling” design pattern to produce assessments for Agentsheets, and our “design and implement creative solutions and artifacts” pattern for making a game in Alice. These are just a few examples of how you can measure things that experts say are important about computational thinking practices in the context of different curriculum.

How do computational practices play out in the ECS Curriculum?

There is a lot of focus in ECS curriculum on problem solving. But a lot of ways to get kids interested in computer science are more about creative expression: We have them build a blog, tell a story, make a character dance on the stage. So what we’ve tried to do is to walk the line between solving a problem and designing something; creative design and creative problem solving. Sometimes you don’t have requirements for an artifact that are based on something the user needs; you just want to design something to have creative expression.

There is also a focus on collaboration and communication. Collaborating with peers on a computing activity involves things like “developing a shared understanding of the problem and requirements” or “understanding and valuing multiple perspectives on a problem and its solution”. Those are things specific to a design and problem solving activity. We also have students using collaborative tools like Dropbox, Google Docs, and techniques like peer programming and code reviews. Similarly, with communication: can students explain why they took a particular approach or use different representations (like a narrative or a storyboard) to describe their code? We hope that teachers might look at these and come up with ideas about what productive collaboration and communication might look like.

How has your work evolved and what is unique about it?

We started by planning for the assessment development with the ECS authors, then built and tested (multiple times!) paper and pencil assessments. Now we are putting the assessments online as part of our work with the CS10K Community of Practice to make them more available to more students. Teachers increasingly want their students to practice taking things online because of high-stakes tests. The CS10K Community provides a rich array of resources for teachers, including our assessments. And we are using the assessments to study implementation: how do different implementations of ECS affect student outcomes?. Generally when people do studies of computing education they look at things like whether the students are more interested in computing or see themselves as a computer scientist. We are really interested in using these assessment measures to look at things like the impact of teacher training and experience on student learning.

How do you work with teachers and students?

We did codesign of the items with the authors of the ECS curriculum to begin with. With our items, we try to be very careful about the equity issue, for example, not writing about things (like an iPhone) that only a special class of people would have access to. The curriculum authors had a lot of experience with that, working with LA Unified. So they were very sensitive to those things, and gave us feedback along the way.

In piloting, we started with cognitive think alouds with students as they took the assessment. Then we had students in a class take it, and asked teachers to tell us how they thought their students would do for each item. We wanted to see what teachers expected students to do after they had taught the content. And then we scored the tests and modified the items based on what saw in the scores. Scoring isn’t easy, because the assessments are scenario based—just reading their handwriting can be tough to do! We also had some IISME teachers work with us over the summer.

Now we’ve had thousands of students use the assessments – mostly students in California (Santa Clara and LA), and piloting in Chicago and Boston. It’s really exciting. We had a moment of debate where we didn’t want to release the assessments widely—we just wanted to use it for our research purposes–but then we decided we’d license them for use by any teacher using ECS.

The other thing is that you don’t just write the assessment. You also have to write the rubric for it. That’s another thing where looking at the student data helps you understand what you should actually count as evidence of a correct answer to the question. That went through refinement as well. The package that teachers get now are the assessments and the rubrics.

More broadly, we’re looking at how teachers who are new to teaching computer science find evidence of student learning throughout the course. Do they develop their own rubrics? Give assignments that are more challenging? Look at what kids are doing when they are pair programming? There are all kinds of ways of looking for evidence of learning in a course.

Any closing thoughts on what you’ve learned?

I think that the biggest thing is that the field still hasn’t cracked the nut of what is coding vs. what is computational thinking. I think we could still devolve into that we’re “just teaching coding.” So Florida, in March 2016, voted to approve a proposal that would allow coding classes to count for foreign language credit for high school graduation. We still don’t have a good sense of what does it really mean to think like a scientist or think like an engineer. How do you approach a problem computationally so it can be solved, and then understand what are the big ideas in computer science? What are the contributions that it’s made? I think a great book on the key ideas is one written by computer scientist’s themselves: Computer Science: Reflections on the Field, Reflections From the Field.

There are all these different ways that computer science has impacted us, and we’re not communicating that well to kids. Somebody said that it’s easier to understand what an engineer does vs. a computer scientist because everyone has gone to IKEA and put together a piece of furniture!. Well, maybe. So people have been trying to make definitions of what computer science is and what computer scientists do, but he point that we make is that when you have to measure it, you have to be really clear about what it is.