Here’s a puzzle: if U.S. students do so badly on international tests, especially in math, how can it be that the U.S. economy is so strong? An educated workforce is supposedly a big predictor of a country’s income and annual growth. Yet the performance of American 15-year-olds on the Program for International Student Assessment, or PISA, has always been lackluster. Since 2012, U.S. math scores have slumped down into the bottom half. Meanwhile, the U.S. remains the top economy in the world this year with over $19 trillion of goods and services produced. No other country even comes close.
A group of behavioral economists wondered if U.S. students are actually not as incompetent as their scores would suggest, but simply lazy when they’re taking the PISA exam. To test this, they created a PISA-like exam of just 25 questions and asked 447 sophomores at two different high schools to take it. Seconds before the test started, they surprised half the students at each school with an envelope of 25 one-dollar bills. The researchers told those students they would take away one dollar for each incorrect or unanswered question.
Guess what? Scores rose for the American teens who were bribed. The economists estimated that if U.S. students had put this much effort into the real PISA test, they would have scored 22 to 24 points higher in math, moving the U.S. from 36th to 19th in the 2012 international rankings. (The U.S. ranked 39th in 2015.)
The researchers conducted the same experiment in Shanghai, China, where students had posted the highest scores in the world on the actual 2012 PISA test. However, the bribe (in renminbi instead of U.S. dollars) didn’t make a difference. The bribed Chinese students scored the same as those who weren’t bribed. They both got almost twice as many questions right as the incentivized American students. (Click here if you want to try the test yourself.)
“We’re by no means fully closing the gap,” said Sally Sadoff, a behavioral economist at the Rady School of Management at the University of California at San Diego and one of six authors of the study. “But the incentive is a tool to show that U.S. students aren’t really trying as hard as they could.”
“We’re not saying we should throw out PISA. But the gaps we see are not just about ability, but [about] some combination of ability and motivation,” Sadoff added.
The working paper, “Measuring Success in Education: The Role of Effort on the Test Itself,” was distributed by the National Bureau of Economic Research in November 2017.
There’s no reason for U.S. students to try their best on the PISA test. It won’t help them get into college. They don’t even get to see their individual scores afterward. But the scores often influence policymakers. Often, there’s a rush to copy the educational models of countries that rank at the top. Or there are policy debates inside a country when scores slide.
A PISA official argues that the United States was properly ranked down low at number 36. “I disagree with the inference that not using monetary incentives distorts international comparisons,” Andreas Schleicher told me by email. “What counts is not the skills that people theoretically have with exceptional incentives, but what they bring to bear in a normal low-stakes environment.”
Schleicher heads the education department at the Organization for Economic Cooperation and Development (OECD), which administers the PISA exam. He suggested that I fly to Shanghai on United Airlines and return via China Eastern Airlines to see first-hand the difference that staff motivation makes in another low-stakes environment. (Unfortunately, the Hechinger Report’s travel budget isn’t that abundant.)
It’s interesting that the Chinese students were unswayed by the bribe in the study — especially since they knew this wasn’t the real PISA and would have no bearing on their region’s or nation’s academic reputation. “Our hypothesis is that Shanghai students have a lot of intrinsic motivation,” said Sadoff. “When they’re given a test, they try hard on it. They’ve developed a habit.”
By contrast, American students lose steam on the real PISA. Another 2016 study found that American teenagers answered fewer questions toward the end of the two-hour test and were lazier about filling out surveys that accompany the PISA. The same was true for Greek and Spanish students, but students in Finland showed more endurance.
Some of you may be scratching your heads and wondering how money succeeded in motivating U.S. students at all. Previous research has generally shown that paying kids for good grades or test scores hasn’t worked. The kids would have done the same if you hadn’t showered money on them. But some behavioral economists have been working on tweaking bribes to make them more effective in the classroom. Sadoff was part of a team, including Freakonomics author Stevin Levitt, that tried different approaches and discovered if you paid the bribe up front and threatened to take it away for poor performance, that was very effective in motivating kids in a Chicago experiment. (That 2016 study is here.) The trick was to hand out the reward immediately, without delay, and to trigger loss aversion. “It feels worse to lose something than it feels good to gain the same thing. You work harder to hold onto something,” Sadoff explained.
The bribes worked better with boys than with girls in both the Chicago and the PISA studies. That suggests girls generally put more effort into their work on a day-to-day basis, but boys typically try their best only sometimes.
The bribes also worked better with higher-achieving students than lower-achieving ones. One of the schools in the PISA study was a high-performing boarding school of wealthy children. Yet the thought of losing $25 was strong enough that the bribed students scored much higher than the unbribed students. The researchers also saw a smaller bribery bump among lower-achieving students at a public school. That might be because the PISA-like test was challenging. Even fully motivated, a low-achieving student might not have the ability to answer many more of the questions correctly.
Sadoff isn’t suggesting that schools should start bribing kids before tests. It would be logistically difficult, for one thing. You’d have to give all your tests by computer to calculate scores right away. Plus, it’s hard to take cash away from kids. More importantly, Sadoff agrees with experts who say that educators and parents should help children build their own internal motivation to do well. So if you were going to try a version of this at home, she advises bestowing a reward for a good habit, as opposed to an end result, and then slowly taking the external incentives away. She admits this hasn’t worked with her own two kids, though. “I’m not sure how much my research has improved my bribing,” Sadoff said, laughing.
When one looks at all the problems with PISA, it is difficult to avoid the conclusion that it is junk science. It tests 15-year-olds in school, not the general population. That means it compares results for a sample of about 100% of American students with results for a sample of 30% of Chinese students. It tests a wealthy city like Shanghai (with less than 5% of China’s population) and then adds it to a list of countries, while conflating the city with China.
It compares the performance of schools in Chinese cities, which under national law exclude the children of migrant workers (Chinese nationals, all) with school performance in the U.S., where schools are required by U.S. Supreme Court mandate to educate all children, including undocumented children, living in their attendance areas. And PISA administrators ignored the concerns of member nations when they pointed out that lowered scores across the board in developed western nations in the 2015 administration of PISA were a function of switching from paper-and-pencil administration of the test to digital, computer-based administration.
As he does in this article, Mr. Schleicher dismisses all of these concerns as mistaken or overblown. But his refusal to acknowledge these realities diminishes the credibility of the results he reports, year after year.
” if U.S. students do so badly on international tests, especially in math, how can it be that the U.S. economy is so strong?”
The answer is that low test scores, not high scores, lead to future national economic success in developed countries, as I showed in a 2007 article in The Phi Delta Kappan. The most likely explanation is that placing too much emphasis on test scores reduces classroom time in doing those things that develop innovativness and entrepreneurship, the keys to driving a highly successful economy, and abilities that are not tested on tests.
There is no doubt that the never confirmed belief that high test scores are good for our economy is false. It is a myth. It is wrong. It is socially harmful to believe it. It is incompetent to perpetuate it.
On the other hand, the explanation of the fact that high test scores harm our nation is, at present, only an hypothesis but should be the center of attention of educational reforms.