Comparing American students to high-flying students in South Korea or Finland is pointless, a new study by the Washington, D.C.-based Economic Policy Institute argues.
Education reformers have long noted that American students do poorly on PISA tests, administered to 15-year-olds in nearly 60 economically advanced countries by the Organization for Economic Cooperation and Development.
In 2012, a handful of Asian countries, including China, Hong Kong, Singapore, Taiwan and South Korea, dominated the PISA tests on all subjects, with Finland joining the high fliers on reading and science but not math.
President Obama has often warned that a nation that "out-educates us today will out-compete us tomorrow," and the administration has used flagging PISA performance as a key talking point in its push for national testing and school and teacher accountability.
But the EPI study argues that such comparisons are distracting and don't lead to useful policy insights. "Such tests can't really tell us why students in any country have higher or lower test scores than in another," said Stanford education professor Martin Carnoy, the EPI studys lead author. "Connecticut should be looking at Massachusetts, not Finland or Hong Kong."
Poor comparisons
Texas, Carnoy notes, has outperformed California, which is similar in many key demographics, while Massachusetts has risen steadily above Connecticut, both again similar in most respects.
Carnoy notes that in Asian countries like South Korea, parents spend enormous amounts of time and money on test preparation for their children. Meanwhile, in Finland there may be localized cultural factors in an ethnically homogenous and prosperous country.
Another poor comparison is Germany, notes Emma Garca, an economist at EPI and a coauthor of the report. "Germany tracks students in high school by ability, which the U.S. does not do. So it's hard to import any policy lessons from how German students are doing."
"The United States does not have an educational system," Garcia said. "Instead, it has 51 different systems, when you include the District of Columbia."
This, she argues, means that any attempt to compare U.S. performance to other, often much smaller or more homogenous countries tends to obscure more than it reveals.
"Their histories and social characteristics are very different," Garcia said.
A new perspective
The realization that test score comparisons hide more than they reveal is becoming more mainstream.
At the same time the EPI was putting the finishing touches on its report, Matt Chingos, a senior fellow at the Washington, D.C.-based Urban Institute, released a similar but smaller study making a parallel point.
Chingos' paper, which targeted misconceptions about how states compare to each other, rather than how they compare to other countries, was timed for the 2013 state-by-state National Assessment of Educational Progress (NAEP) scores.
He adjusted the data based on demographics, including poverty. "When you adjust the data, the list looks pretty different," Chingos said. "There are still some of the usual suspects at the top, like Massachusetts, but you have a whole bunch of states that get bumped down because they have lots of students who tend to do better on the exams."
Utah, Iowa and Connecticut, among others, appear to be well above average in raw data but fall to middling or worse when adjusted.
Chingos also showed that while Texas and Florida appear to be mediocre at best in the raw data, both do quite well when adjusted for socioeconomic status.
The new EPI scores differ from Chingos' report in a couple of ways, digging deeper along similar lines, combing over multiple decades of NAEP scores, comparing those to PISA and a second international test, and looking at policy changes at the state level that may account for outcome changes over time.
Chingos sees his adjustments as a first cut, drawing attention to the problem and putting data in place to allow real comparisons to be done.
Real levers
The authors of the EPI study, meanwhile, think they may have found some promising data. Using a battery of controls, including multiple decades of scores and details about the schools and the individual students, allows the EPI researchers to rough out some real levers of change.
EPI found that poor children perform less well than their wealthier peers on average, but they do better in an economically mixed school. They also do better in a wealthier state, regardless of how segregated their school is.
That last point is quite counterintuitive, but EPI found that students who are identical in all relevant variables actually perform better, on average, in states that are wealthier, on average.
"We tested for several variables," Garcia said, "and the most striking one was poverty."
The poverty impact, Garcia said, reached even to better off students. "Even better off students in a rich state are going to perform better than similar students in a poorer state," she said.
Theres a player effect, a team effect and a league effect, Carney said.
And the league effect of the wealthier state does not hinge on per pupil spending, which Carney said had little or no impact. Teacher qualifications, such as certification and master's degrees, also had little impact.
The absence of impact from per pupil spending was "puzzling," Garcia said.
Aside from poverty levels in the school and the state, what kinds of policy changes made a difference? The answer, Garcia said, seems to be that states committed to accountability testing and consequences for school failure do better than those that resist that form of tough oversight.
EPI created an accountability index, she said, and that index had more explanatory power than anything else they tested besides poverty.
That conclusion, if confirmed over time, would be music to the ears of educational reformers of both parties who, beginning with George W. Bush's 2001 No Child Left Behind Act and ratcheting up with Race to the Top under the Obama administration, have pushed for ever greater accountability based largely on models first tested in Florida and Texas.
Obama has often warned that a nation that "out-educates us today will out-compete us tomorrow," and the administration has used flagging U.S. PISA performance as a key talking point in its push for national testing and school and teacher accountability.
Education reformers have long noted that American students do poorly on PISA tests, administered to 15-year-olds in nearly 60 economically advanced countries by the Organization for Economic Cooperation and Development.
In 2012, a handful of Asian countries, including China, Hong Kong, Singapore, Taiwan and South Korea, dominated the PISA tests on all subjects, with Finland joining the high fliers on reading and science but not math.
President Obama has often warned that a nation that "out-educates us today will out-compete us tomorrow," and the administration has used flagging PISA performance as a key talking point in its push for national testing and school and teacher accountability.
But the EPI study argues that such comparisons are distracting and don't lead to useful policy insights. "Such tests can't really tell us why students in any country have higher or lower test scores than in another," said Stanford education professor Martin Carnoy, the EPI studys lead author. "Connecticut should be looking at Massachusetts, not Finland or Hong Kong."
Poor comparisons
Texas, Carnoy notes, has outperformed California, which is similar in many key demographics, while Massachusetts has risen steadily above Connecticut, both again similar in most respects.
Carnoy notes that in Asian countries like South Korea, parents spend enormous amounts of time and money on test preparation for their children. Meanwhile, in Finland there may be localized cultural factors in an ethnically homogenous and prosperous country.
Another poor comparison is Germany, notes Emma Garca, an economist at EPI and a coauthor of the report. "Germany tracks students in high school by ability, which the U.S. does not do. So it's hard to import any policy lessons from how German students are doing."
"The United States does not have an educational system," Garcia said. "Instead, it has 51 different systems, when you include the District of Columbia."
This, she argues, means that any attempt to compare U.S. performance to other, often much smaller or more homogenous countries tends to obscure more than it reveals.
"Their histories and social characteristics are very different," Garcia said.
A new perspective
The realization that test score comparisons hide more than they reveal is becoming more mainstream.
At the same time the EPI was putting the finishing touches on its report, Matt Chingos, a senior fellow at the Washington, D.C.-based Urban Institute, released a similar but smaller study making a parallel point.
Chingos' paper, which targeted misconceptions about how states compare to each other, rather than how they compare to other countries, was timed for the 2013 state-by-state National Assessment of Educational Progress (NAEP) scores.
He adjusted the data based on demographics, including poverty. "When you adjust the data, the list looks pretty different," Chingos said. "There are still some of the usual suspects at the top, like Massachusetts, but you have a whole bunch of states that get bumped down because they have lots of students who tend to do better on the exams."
Utah, Iowa and Connecticut, among others, appear to be well above average in raw data but fall to middling or worse when adjusted.
Chingos also showed that while Texas and Florida appear to be mediocre at best in the raw data, both do quite well when adjusted for socioeconomic status.
The new EPI scores differ from Chingos' report in a couple of ways, digging deeper along similar lines, combing over multiple decades of NAEP scores, comparing those to PISA and a second international test, and looking at policy changes at the state level that may account for outcome changes over time.
Chingos sees his adjustments as a first cut, drawing attention to the problem and putting data in place to allow real comparisons to be done.
Real levers
The authors of the EPI study, meanwhile, think they may have found some promising data. Using a battery of controls, including multiple decades of scores and details about the schools and the individual students, allows the EPI researchers to rough out some real levers of change.
EPI found that poor children perform less well than their wealthier peers on average, but they do better in an economically mixed school. They also do better in a wealthier state, regardless of how segregated their school is.
That last point is quite counterintuitive, but EPI found that students who are identical in all relevant variables actually perform better, on average, in states that are wealthier, on average.
"We tested for several variables," Garcia said, "and the most striking one was poverty."
The poverty impact, Garcia said, reached even to better off students. "Even better off students in a rich state are going to perform better than similar students in a poorer state," she said.
Theres a player effect, a team effect and a league effect, Carney said.
And the league effect of the wealthier state does not hinge on per pupil spending, which Carney said had little or no impact. Teacher qualifications, such as certification and master's degrees, also had little impact.
The absence of impact from per pupil spending was "puzzling," Garcia said.
Aside from poverty levels in the school and the state, what kinds of policy changes made a difference? The answer, Garcia said, seems to be that states committed to accountability testing and consequences for school failure do better than those that resist that form of tough oversight.
EPI created an accountability index, she said, and that index had more explanatory power than anything else they tested besides poverty.
That conclusion, if confirmed over time, would be music to the ears of educational reformers of both parties who, beginning with George W. Bush's 2001 No Child Left Behind Act and ratcheting up with Race to the Top under the Obama administration, have pushed for ever greater accountability based largely on models first tested in Florida and Texas.
Obama has often warned that a nation that "out-educates us today will out-compete us tomorrow," and the administration has used flagging U.S. PISA performance as a key talking point in its push for national testing and school and teacher accountability.
