The U.S. Department of Education has released college-by-college graduation rates for low-income students receiving federal grants, for the first time ever—information advocates, and Congress, have long demanded.
But there turns out to be a problem: A lot of the numbers are wrong.
In comparison to reviews of the same data by independent organizations, the Department of Education figures—released in conjunction with the Obama administration’s long-awaited College Scorecard, which is meant to provide consumers with helpful information about universities and colleges—are off by an average of 10, and as much as 59, percentage points.
That’s a wide margin of error in the realm of statistics.
“Ten percentage points is pretty worrisome for me,” said Mamie Voight, director of policy research at the nonpartisan Institute for Higher Education Policy.
The figures measure the success of students who get Pell grants, which provide up to $5,775 a year and typically go to Americans from low-income families to pay for tuition or other college expenses at a cost to the federal government of more than $31 billion a year. Knowing how many of these students ever graduate not only tells taxpayers what they’re getting for their money; it helps to measure the effectiveness of colleges and universities at helping all their undergraduates earn degrees.
Yet while schools are required by law to provide the graduation rates of Pell recipients to any applicants who ask, a loophole protects them from having to report the same figures to the government. So the Department of Education used something called the National Student Loan Database System, or NSLDS, to calculate the percentage of people with Pell grants who earn degrees in four, five and six years.
Trouble is, the NSLDS was designed to keep track of student loans, not to monitor the graduation rates of Pell recipients. A student who gets a Pell grant, but doesn’t get another federal loan, for instance, will likely be missed.
“It’s not the best fix,” said Andrew Nichols, director of higher education research and data analytics for The Education Trust, which advocates for low-income students.
Related: Billions in Pell dollars go to students who never graduate
Higher-education lobbyists have objected to the College Scorecard on the grounds the information it provides is inaccurate or can be misleading. And, at least in this case, they appear to be right.
Knowing how many of these students ever graduate not only tells taxpayers what they’re getting for their money; it helps to measure the effectiveness of colleges and universities at helping all their undergraduates get degrees.
A comparison of the figures provided by the government with a report released last month by The Education Trust using data from state education systems, colleges and universities themselves, and other sources shows that, on average, there’s a 10 percentage point difference between what the College Scorecard materials estimate to be the percentage of Pell recipients who graduate within six years and what The Education Trust found they were at the 1,088 four-year institutions for which both have results.
Boston University had the largest gap. According to the Department of Education numbers, only 25 percent of BU’s students who get Pell grants graduated in 2013 with four-year degrees after even six years. The university’s self-reported figure, however, is 84 percent. BU confirmed that the Department of Education’s data was incorrect.
Related: An unprecedented look at Pell Grant graduation rates from 1,149 schools
A separate Hechinger Report analysis of Pell graduation rates obtained directly from the nation’s 50 largest public and 32 biggest private colleges and universities—18 privates refused to provide them—also shows the same discrepancies with the Department of Education figures.
An Education Department official, who spoke on condition that her name not be used, acknowledged that the data were imperfect and said that the department will work with universities and colleges to identify mistakes so that the rates grow more precise with time.
Voight and other experts said there’s no reason the data can’t be accurate right now.
They said one solution would be to simply call for universities and colleges to give the government the graduation rates of their Pell students, just as they already submit their graduation rates broken down by race.
While that’s an option, the Department of Education official said, the agency is continuing to work with the existing data in NSLDS “in the interest of minimizing the burden on institutions.”
Related: While Washington wavers, states provide detailed college ratings
But experts said requiring that universities and colleges provide their Pell Grant graduation rates shouldn’t be an additional burden, since they’re already supposed to be calculating this number to provide to prospective students.
The Department of Education figures—released in conjunction with the Obama administration’s long-awaited College Scorecard—are off by an average of 10, and as much as 58, percentage points.
“That is not a real high bar to be able to do that,” said Christine Keller, executive director of the Student Achievement Measure, which tracks individual students through their educations, and which is a project of two national associations of public universities, the Carnegie Corporation, and the Bill & Melinda Gates Foundation. (Carnegie and Gates are both among the funders of The Hechinger Report, which produced this story.) The data comes from a private, nonprofit, nongovernmental organization called the National Student Clearinghouse.
The groups involved in the Student Achievement Measure have called for a similar federal data system. There’s a big obstacle to doing that, however: current law, pushed by higher-education groups, bans the Department of Education from collecting student-level data.
Changing that might be a political hurdle, but Keller is hopeful.
“If we could lift the ban, if we could create from the beginning a data set that was intended to do these sort of calculations, we’d have more trustworthy, more valid, more accurate information rather than struggling with the weaknesses of some of the current systems and trying to fix them on the back end.”
This story was produced by The Hechinger Report, a nonprofit, independent news organization focused on inequality and innovation in education. Read more about higher education.
Reproduction of this story is not permitted.
At The Hechinger Report, we publish thoughtful letters from readers that contribute to the ongoing discussion about the education topics we cover. Please read our guidelines for more information. We will not consider letters that do not contain a full name and valid email address. You may submit news tips or ideas here without a full name, but not letters.
By submitting your name, you grant us permission to publish it with your letter. We will never publish your email address. You must fill out all fields to submit a letter.