Here’s a puzzle: if U.S. students do so badly on international tests, especially in math, how can it be that the U.S. economy is so strong? An educated workforce is supposedly a big predictor of a country’s income and annual growth. Yet the performance of American 15-year-olds on the Program for International Student Assessment, or PISA, has always been lackluster. Since 2012, U.S. math scores have slumped down into the bottom half. Meanwhile, the U.S. remains the top economy in the world this year with over $19 trillion of goods and services produced. No other country even comes close.
A group of behavioral economists wondered if U.S. students are actually not as incompetent as their scores would suggest, but simply lazy when they’re taking the PISA exam. To test this, they created a PISA-like exam of just 25 questions and asked 447 sophomores at two different high schools to take it. Seconds before the test started, they surprised half the students at each school with an envelope of 25 one-dollar bills. The researchers told those students they would take away one dollar for each incorrect or unanswered question.
Guess what? Scores rose for the American teens who were bribed. The economists estimated that if U.S. students had put this much effort into the real PISA test, they would have scored 22 to 24 points higher in math, moving the U.S. from 36th to 19th in the 2012 international rankings. (The U.S. ranked 39th in 2015.)
The researchers conducted the same experiment in Shanghai, China, where students had posted the highest scores in the world on the actual 2012 PISA test. However, the bribe (in renminbi instead of U.S. dollars) didn’t make a difference. The bribed Chinese students scored the same as those who weren’t bribed. They both got almost twice as many questions right as the incentivized American students. (Click here if you want to try the test yourself.)
“We’re by no means fully closing the gap,” said Sally Sadoff, a behavioral economist at the Rady School of Management at the University of California at San Diego and one of six authors of the study. “But the incentive is a tool to show that U.S. students aren’t really trying as hard as they could.”
“We’re not saying we should throw out PISA. But the gaps we see are not just about ability, but [about] some combination of ability and motivation,” Sadoff added.
The working paper, “Measuring Success in Education: The Role of Effort on the Test Itself,” was distributed by the National Bureau of Economic Research in November 2017.
There’s no reason for U.S. students to try their best on the PISA test. It won’t help them get into college. They don’t even get to see their individual scores afterward. But the scores often influence policymakers. Often, there’s a rush to copy the educational models of countries that rank at the top. Or there are policy debates inside a country when scores slide.
A PISA official argues that the United States was properly ranked down low at number 36. “I disagree with the inference that not using monetary incentives distorts international comparisons,” Andreas Schleicher told me by email. “What counts is not the skills that people theoretically have with exceptional incentives, but what they bring to bear in a normal low-stakes environment.”
Schleicher heads the education department at the Organization for Economic Cooperation and Development (OECD), which administers the PISA exam. He suggested that I fly to Shanghai on United Airlines and return via China Eastern Airlines to see first-hand the difference that staff motivation makes in another low-stakes environment. (Unfortunately, the Hechinger Report’s travel budget isn’t that abundant.)
It’s interesting that the Chinese students were unswayed by the bribe in the study — especially since they knew this wasn’t the real PISA and would have no bearing on their region’s or nation’s academic reputation. “Our hypothesis is that Shanghai students have a lot of intrinsic motivation,” said Sadoff. “When they’re given a test, they try hard on it. They’ve developed a habit.”
By contrast, American students lose steam on the real PISA. Another 2016 study found that American teenagers answered fewer questions toward the end of the two-hour test and were lazier about filling out surveys that accompany the PISA. The same was true for Greek and Spanish students, but students in Finland showed more endurance.
Some of you may be scratching your heads and wondering how money succeeded in motivating U.S. students at all. Previous research has generally shown that paying kids for good grades or test scores hasn’t worked. The kids would have done the same if you hadn’t showered money on them. But some behavioral economists have been working on tweaking bribes to make them more effective in the classroom. Sadoff was part of a team, including Freakonomics author Stevin Levitt, that tried different approaches and discovered if you paid the bribe up front and threatened to take it away for poor performance, that was very effective in motivating kids in a Chicago experiment. (That 2016 study is here.) The trick was to hand out the reward immediately, without delay, and to trigger loss aversion. “It feels worse to lose something than it feels good to gain the same thing. You work harder to hold onto something,” Sadoff explained.
The bribes worked better with boys than with girls in both the Chicago and the PISA studies. That suggests girls generally put more effort into their work on a day-to-day basis, but boys typically try their best only sometimes.
The bribes also worked better with higher-achieving students than lower-achieving ones. One of the schools in the PISA study was a high-performing boarding school of wealthy children. Yet the thought of losing $25 was strong enough that the bribed students scored much higher than the unbribed students. The researchers also saw a smaller bribery bump among lower-achieving students at a public school. That might be because the PISA-like test was challenging. Even fully motivated, a low-achieving student might not have the ability to answer many more of the questions correctly.
Sadoff isn’t suggesting that schools should start bribing kids before tests. It would be logistically difficult, for one thing. You’d have to give all your tests by computer to calculate scores right away. Plus, it’s hard to take cash away from kids. More importantly, Sadoff agrees with experts who say that educators and parents should help children build their own internal motivation to do well. So if you were going to try a version of this at home, she advises bestowing a reward for a good habit, as opposed to an end result, and then slowly taking the external incentives away. She admits this hasn’t worked with her own two kids, though. “I’m not sure how much my research has improved my bribing,” Sadoff said, laughing.