The world (or at least America) is buzzing about the students in this city! In a recent aptitude test, students in Shanghai produced “stunning”ly high scores, beating American averages even in English and generally setting off another flurry of worry over a “Sputnik” moment and American complacency.
The test 15-year-old Shanghainese kids, American kids and other kids all over the world took was called the Program for International Student Assessment, known as PISA. As the New York Times breathlessly quotes other people exclaiming, the results are to be believed and to be awed by:
“We have to see this as a wake-up call,” Secretary of Education Arne Duncan said in an interview on Monday.
“I know skeptics will want to argue with the results, but we consider them to be accurate and reliable, and we have to see them as a challenge to get better,” he added. “The United States came in 23rd or 24th in most subjects. We can quibble, or we can face the brutal truth that we’re being out-educated.”
But are “we,” really?
First off, I want to emphasize that the test scores are pretty impressive. They speak to the ability of students here to study their asses off – something I’ve never had the motivation (or forceful enough parents) to do. And while a part of me feels for any kid forced to do homework for umpteen hours a day, there’s no denying that really, really helps you memorize stuff. So good on you, Shanghai students, for being hard working and disciplined.
I also want to emphasize that I agree that the U.S. really does need to figure out a better educational system than what it has right now. If this is the wake up call politicians need to finally do mass studies on effective teaching and educational ecouragement policies, then good. But I doubt it will be – do you know how many “wake up calls” they’ve had already? The U.S. has been declining in world test scores for decades now, wailing about it all the way. But for every program they institute, they have idiots whining about how immigrants should learn English without ESL programs or how evolution shouldn’t be in textbooks.
Okay, so that’s only a subset of the population – but therein lies the problem. If the sample from America took kids from from Mississippi as well as kids from Massachusetts, it’s hardly a fair comparison to the richest city in the richest area of China – the article seems to forget (or wave off in passing) that there a about as many variations in education quality here that there are in America. Granted, there’s a chance math scores would still be higher even if the rest of China was still involved (America, for some reason, is terrible at teaching kids math), but would Singapore have been beat?
Also, there’s some evidence that China, never known to not try and make themselves look as good as possible, may have gamed the system. Not cheated, mind you, but helped kids to be as prepared as they can be for a standardized test. It’s a tried and true method with everything from the SATs to TOEFLs, and it’s not that hard to figure something out for a PISA too.
James Fallows at The Atlantic quotes the reaction of one scientist:
First, all numerical measurements have intrinsic variance or uncertainty. So if we split the sample test taking group (according to this article ~5000 students) into several random subgroups we would expect that the averages would vary. This would allow an estimate of the intrinsic random uncertainty. No error estimates are quoted here
Second, how were the representative groups of 5,000 students selected in each study? In a city of 20 million, there might be a population of half a million 15 year olds enrolled in secondary education. That means there would be many ways to select a “representative” group.
Third, to what extent were students trained specifically for this test? Even in the case of the SAT which was designed to be an “Aptitude” test and for which preparation was supposed to be useless, Kaplan and other test training services claimed that they could improve scores by an average of 30-50 points on the old 800 point scale (much more significant in upgrading a student’s percentile ranking if the student moved from 450 to 500 or even from moved 730 to 780)…
The gap between Shanghai and the second place finisher for the math scores is somewhat more spread out than the corresponding segments in reading and math. The rest of the distributions appear fairly constant. So the most likely explanation for this would be that the Shanghai schools were specifically trained for the “PISA” exam and this training was most effective for the math subject test. Once again the jumbling and narrow range of scores in the top third in all test categories (except Shanghai which appears to be an outlier) is also supportive of the idea that Shanghai is “gaming” the exam.
So, in the end, what we have found out about Shanghai is something most people here knew all along: training for test taking has benefits, and the Chinese really know how to train their teens to take tests. That’s not a bad thing – and there is some evidence that these 5000 students have great abilities outside of test taking too – and heck if the U.S. education system doesn’t need some giant overhaul – but, as Fallows says, “take this seriously, and recognize that China is moving ahead in many, many ways. But recognize the fallibilities in this study, and don’t go nuts.”