What is most likely to grab headlines this week in response to the publication of the latest PISA results is that England’s scores have dropped in both reading and mathematics. A lot of weight has been put on these – and especially the former – as evidence of the success of government policy (and indeed of the legacy of one minister in particular). But what caught my eye is the fall in our science scores.
Why?
Because it’s hard to put it down to just the effect of the pandemic. While reading and mathematics scores have fallen this time around, it was after gains the last time, and not part of a broader trend.
The same can’t be said for science, with England’s PISA score declining in this subject steadily since 2012 (see Figure 1 below). This finding was highlighted prominently by the OECD in their UK country note, where they said: “In science, the most recent PISA results are close to the results observed in 2018, but below those of any previous assessment confirming a decade-long decline that began around 2012.”
But how worried should we be by this apparent decline?
England is not alone
The first important point to note is that England is not alone in this decline in science scores. As Figure 2 below illustrates, the trend for England has paralleled the decline in the OECD average (the average across industrialised countries).
Now, as I have written about previously I think part of this apparent decline is methodological. When PISA moved from a paper to a computer-based assessment in 2015, there was evidence of a significant ‘mode effect’ (the test being harder on computer than paper). Although the OECD tried to “correct” for this problem at the time, in my view, they did not fully resolve this issue.
That’s why we observed a seven-point decline in the OECD average between 2012 and 2015 (when the switch from paper to computer assessment took place). Interestingly, this led to a bigger decline in science than this time around – and thus apparently bigger than the effect of the pandemic (which does seem a bit odd). All in all, it looks to me like England’s fall is therefore likely mostly due to factors affecting all countries (methodological changes to PISA and the pandemic) more than anything else.
Evidence from TIMSS
Whenever something interesting appears in one international assessment, I always look for whether there is consistent evidence from another.
TIMSS is another major international study that England has participated in for several years. And – although PISA gets all the media – an argument can be made for TIMSS providing more robust information about pupils’ science skills.
Currently, TIMSS only provides information up to 2019, with the 2023 (and first post-pandemic) results coming in November or December next year.
However, as Figure 3 illustrates, there is no evidence of any downward shift in England’s TIMSS science scores. For year 5 pupils, there has actually been an increase over the past decade, while for year 9 pupils scores have been completely flat.
It’s therefore clear that the “decade-long decline” observed in PISA is not replicated in TIMSS.
Summing up
My guess is that – putting aside debated over political legacies – England’s falling PISA science scores will generate a fair amount of attention among future-minded analysts and policy makers over the next few days.
However, the broader evidence suggests that, in spite of all the challenges of the past decade – austerity, spending cuts, the pandemic and teacher strikes to name a few – England’s science performance is holding up pretty well. We remain above international average, the fall observed in PISA is not observed in other data sources, and a similar trend can be observed in other countries.
So, what immediately grabbed my attention from the PISA results may not actually be that big a deal after all.
Your thoughts