A long time ago, almost a year now, we had the opportunity to conduct a research study at Nemo Science Live. With Science Live, researchers can perform small scientific experiments on visitors of the Nemo Science Museum in Amsterdam, which is a great and relatively easy way to find participants. If you are a Dutch researcher studying things related to kids, I really recommend you to apply! We were lucky to be there in the summer break of ’17, and we recruited over 600 participants for two separate studies! That is really a lot for our type of work! In this post I will explain what one of these two experiments entailed and what we found.

Syntax and semantics

When we teach programming, there are, by and large, two things we need to explain to students.
One is syntax: how to correctly tell a computer what to do. For example, in Python we need to teach students that a colon should follow a for statement, otherwise the program will not run. This program for example looks okay, but will not run, as indicated by the ‘squiggly line’:

In block-based programming languages (BBPL) like Scratch, syntax errors cannot occur, since users drag blocks to create, for example a loop that does the same as the Python code above:

That seems a clear win for BBPL, we can skip the syntax explanation and go right to the hard stuff: semantics: What is a loop exactly? How does it work, and when do we use it? In other words: what are the hard concepts when programming.

Metaphors!

When teachers explain concepts they often use metaphors, which of course, is a good idea. Metaphors from the real world can be really helpful to explain the abstract world of programming. A loop is like a song, where we repeat some lines but not others. Or a condition is like a rule in a game, when you roll 6, you get to place a pawn on the board.

Or… a variable is like a box, you can store something in it, and write the name of the variable on the outside of the box. I myself (video in Dutch) have used the metaphor because it is nice and tangible.

But in working with children in classrooms, we found that this metaphor can also lead to confusion. A box can hold multiple coins or numbers or names. We observed learners suffering for the multiple values misconception where they took the metaphor too far and assumed it would store all values and not just the latest one. This misconception is number 9 on the large list that Sorva described in his thesis*.

Our experiment

At Nemo Live, we recruited 496 participants with no previous programming experiment for this study**, both adults and children. We randomly assigned people one of two groups:

The participants in the label group received an introductory programming lesson in which we explain a variable as being a label that one can place on one value, like a temperature or the age of a person. The other half, the box group, got a lesson in which we explain a variable as a box, like a piggy bank or a shoe box.  We consistently used the metaphor in both lessons, for example, we used “x contains 5″ for the box group and “x is 5″ for the label group.

After the programming lesson, the participants receive questions testing their understanding of programming. These including simple questions testing participants’ understanding of variables, with one variable, like this:

We also added questions to to the presence of the ‘multiple values’ misconception specifically, like:

Results

All in all the study has three interesting results:

1) For simple questions with one assignment, relating to variables, the box group did better.

2) But… analyzing the presence of the multiple values misconception, people in the box group were more likely to suffer from the misconception.

There were the results we expected, at least we expected that there would be a difference between te groups. But we also found something unexpected:

3) We found a new misconception. Novices in the experiment in both groups, were likely to think the values needed to be added! They approached the question as a math puzzle and applied the simplest thing they could think of: addition.

Milage may vary for novices!

That final result might be the most intriguing. Sorva’s list and many other words on misconceptions focus on a particular group: university students in programming courses. Our results show that when we teach novices, new misconceptions can occur that we should be aware of.

Want to know more? Read the full paper!

—

*Sorva’s whole thesis, which is available open access is worth a read, but a list of 162 common misconceptions gathered from literature can be found on page 359 is really educational if you teach programming!

** As I stated above, we ran two experiments, we had a different se tof questions for people with programming experience.