Logo vs Scratch: quantitative evaluation of learning the concept of angle
Foreword: This post is a technical report of a research project in progress. I’m writing it in part for me, in order to set ideas while I am distracted by a number of projects, but also and mainly for giving a first acknowledgement to the students who have collaborated up to now: Martina Fini (Massa), Fabiola Izzo (Lucca), Marianna Fazzino (Grosseto) and Silvia Ercoli (Florence). Thus, first of all, a big thank you to Martina, Fabiola, Marianna and Silvia, because without their fundamental contribution this work would not be possible.
I have been working with Logo and Scratch for three years with my Primary Education students, about 250 students a year. I have also had the good luck to work in some third and fourth grade primary classes. I then came up with some ideas, summarized in a position paper entitled A step Back into the Future (PDF), which led me to formulate the following research question:
In a work aimed at reinforcing the learning of a specific mathematical concept, is there a difference between the two languages, Logo and Scratch?
There is scientific literature about this: the question of whether you learn more with Logo or Scratch is ambiguous. There is some evidence that the practice of Logo improves the deep understanding of concepts while Scratch works better to quickly assimilate specific constructs. Moreover, it seems that girls and boys who had an experience with Logo are more likely to be willing to tackle more complex challenges, while those who have worked with Scratch feel they have had just fun. Again, in summary, Logo is perceived as a cognitive work, Scratch as a kind of video game.
In this brief anticipation we need only two references: the work of Richard Noss (Children’s learning of geometrical concept through Logo, Journal for reasearch in mathematics education, 18(5): 343-362, 1987) and that of Colleen M. Lewis (How programming environment shapes perception, learning and goals: Logo vs. Scratch – SIGCSE’10, 2010, Milwaukee, Wisconsin, USA). From the first one we took the experimental design to study the effect of Logo on learning the concept of angle. From the second one, a comparison between Logo and Scratch, but referred to more general aspects of learning, including personal perception.
Trying to answer the research question by means of a controlled statistical study, the problem of sample size arises, which is the Achilles heel of all studies of this kind. Hence the idea of increasing the sample through a series of experimental studies carried out by the students during their internships, whose results are then reported in their final theses. Thanks to the collaboration of the students we have been able to reach a sample of 90 children, so far.
All the students apply the same experimental design. In short, the class is divided into two equal groups: the first one explores the concept of angle with Logo, whereas the second one does the same with Scratch. At the middle of the course the roles are exchanged and the same things are done again with the other language. During the whole experience, three tests are performed: one at the beginning, one between the first and second phases and one at the end. For the purposes of the specific research question, the data of the first and the second tests are used. The test consists of 13 different questions. What is measured is the number of correct answers, for each child and for each question. The results are evaluated with a Student test for paired data.
In this short report we are interested in the following three comparisons:
- Comparing test 1 and test 2 in the Logo-first group: results improve significantly (p=0.0015)
- Comparing test 1 and test 2 in the Scratch-first group: values are not significantly different (p=0.64)
- Comparing results of test 1 between Logo-first and Scratch-first: values are not significantly different (p=0.23)
The influence of Logo appears to be statistically significant at a level close to 1 per thousand. This means that the probability that this difference is due to chance (hypothesis H0) is less than 1.5 per thousand. On the contrary, the result obtained with Scratch does not reveal any improvement, rather a slight worsening, which, however, is not statistically significant (p=0.64). This result is in agreement to the empirical observation deriving from the experience of several dozen students who have worked with their children with both systems, according to which the interface of Scratch is extremely attractive and powerful but creates a context that distracts too much the children. On the other hand, the paradigm proposed by Logo is that of a blank sheet of paper: not buttons to press and pre-packaged blocks to browse but instructions to write.
The third plot shows the difference in the pre-test competencies between the two groups. It seems that the group which was entering the Scratch-first path had somewhat better competencies. However this difference is not statistically significant, as it should be.
The claim we are making here, that Logo is better in supporting the knowledge of angle, appears to be confirmed with a pretty good statistical significance. This means that the probability of getting false positives is low. However a good test should not only have a good significance but also a high power, that is a high probability of catching true positives. The test power is influenced by the sampling size, by the effect size and by the required significativity. We can act on the sampling size but we need to increase it by a factor of about 5, probably.
Further work to be done…
Andreas Formiconi 