The Hechinger Report is a national nonprofit newsroom that reports on one topic: education. Sign up for our weekly newsletters to get stories like this delivered directly to your inbox.

A handful of colleges think they’ve found the secret to closing the gap between the types of graduates they’re turning out and the types employers say they need.


Not the hairy, creepy kind. The colleges are using artificial-intelligence spiders that crawl through search engines and read thousands of online “help wanted” ads to check up on the job market in real time—not two years after the fact, which is how long the federal government can take to report on labor trends.

future workforce
The Seattle satellite campus of Northeastern University, which moved into that market when “spidering” technology showed there was demand there. (Photo courtesy of Phototainment)

The technology is helping institutions add and update programs on short notice so their graduates can land real-world jobs. And at the same time, schools are using the new information to eliminate programs that leave students in debt with skills employers don’t want.

So far, the use of such technology is limited, but it is likely to increase as colleges and universities face growing pressure to help drive economic recovery and justify the cost of higher education by matching graduates’ skills with workforce needs.

“It’s not just good enough any more to educate a student,” said Elaine Gaertner, director of a system of regional centers that use spidering technology to collect real-time job-market information for California’s community colleges. “You have to educate him with a purpose.”

That’s often hard to do with federal labor data, which can be years out of date.
“It’s like looking in the rearview mirror to get a sense of where you’re going,” said John Dorrer, a program director at the Boston-based advocacy group Jobs for the Future.

“We’re in a very disruptive time,” said Dorrer. “The past doesn’t equal the future.” And using outdated information from the government means, “We’re training people for jobs that don’t exist, and not training people for jobs that do.”

Employers say that’s why, in a time of persistent unemployment, there are nearly 3.7 million jobs nationwide sitting empty, according to the U.S. Department of Labor.

Learn and Earn

This story is part of a series about workforce development and higher education.

Read the rest of the series

The National Association of Manufacturers estimates that 600,000 manufacturing jobs alone remain unfilled because companies can’t find applicants with the right skills. Ninety-three percent of IT employers say they’re having trouble attracting qualified employees, according to the Computing Technology Industry Association.

“I hear from business leaders all the time who want to hire in the United States, but at the moment, they cannot always find workers with the right skills,” President Barack Obama told an audience last year at Northern Virginia Community College. Students should be able to learn those skills at colleges and universities, the president said, “and companies looking to hire should be able to count on these schools to provide them with a steady stream of workers.”

Yet the vast majority of colleges aren’t using the latest technologies, like spidering, to track workforce demand.

“They only care about filling seats,” said Matthew Harris, cofounder of a newly launched website called “The demand for college is so high, they don’t have any problem getting applicants. So why should they change?”

Seventy-two percent of educators believe they’re doing a good job of preparing students for the workforce, while only 45 percent of graduates and 42 percent of employers think so, a survey released in December by the consulting firm McKinsey & Company found.

Even if they do pay attention to industry needs, colleges and universities are not particularly fast moving, while the pace of change in the economy has been accelerating.

“In business, if you don’t have the next product ready by the next quarter, you’re in trouble,” said Lee Todd Jr., former president of the University of Kentucky, who founded two high-tech companies before that. “In academics, you’ve got seven years to make tenure.”

But market forces driven by information like that provided by Harris’s website—which uses spidering technology to put updated labor-market data straight into the hands of students and parents—could eventually speed things up.

“We’re trying to create a way for people to get that return-on-investment type of analysis,” Harris said.

And just as universities whose graduates are having trouble getting jobs risk losing business, those that are nimble and responsive may gain, said Matthew Sigelman, CEO of Burning Glass Technologies, which provides job-market spidering services to colleges.

“Educational institutions have always thought of themselves, I think, as being quite removed from the labor market, and that’s changing rapidly,” Sigelman said. “There’s so much awareness around workforce readiness and the skills gap. Schools that can make sure their graduates do well are seeing a meteoric rise in their standing.”

One of those is Northeastern University, whose so-called “experiential learning” approach requires undergraduates to work in real-world settings for as many as 18 months while in school. More than half go on to full-time jobs in those places, and more than 90 percent are employed or in graduate school within nine months of earning their degrees.

Nationally, only 42 percent of the Class of 2010, the most recent for which the figure is available, had jobs at graduation, according to the National Association of Colleges and Employers. Fewer than two-thirds were employed six months later.

Northeastern’s focus on preparing its students for the workplace has driven a 46 percent increase in applications to the university over the last five years.

The school is using spidering technology to strategize an expansion from its base in Boston to other cities, such as Seattle and Charlotte, N.C., where real-time “help wanted” listings reveal high demand for workers with certain skills, but not enough supply, and where Northeastern has opened satellite campuses to lure lucrative, tuition-paying graduate students. More are planned.

“We can see how metropolitan regions compare in terms of the hiring that’s going on,” said Sean Gallagher, senior strategist and market-development officer at Northeastern. “Then we can dig down and see who are the top employers, so we can meet with those employers and find out what the skills are that they need.”

Like other schools, Northeastern previously used state and federal government labor data, “but a lot of those are fairly basic forecasts on a really long time-scale,” Gallagher said. “The inflection point, from my perspective, was the economic downtown in 2008. After that, you could see that these forecasts were entirely out of date.”

Other higher-education institutions still rely on such outdated information, however, Gallagher said. “The way they operate—their orientation, their culture—is just not to incorporate labor-market information in a routine way.” But he said that will change, “especially as more technology like this becomes available.”

Based on real-time labor-market information, the Lone Star College System in Texas will close three programs next fall, in aviation management, hospitality management and computer support. It found that employers prefer four-year to two-year degrees in the first two cases, and were outsourcing work in the third. But it is adding programs to train oil and gas drillers and CT-scan technicians, for which there is burgeoning demand.

“My job is to make sure that the college career programs that we have are the ones that are needed, and that we don’t offer the ones any more where there aren’t jobs at the end,” said Linda Head, Lone Star’s associate vice chancellor for workforce education.

Cabrillo College in California thought its program in medical assisting was doing well—until spidering technology showed there wasn’t much hiring going on in the field, and a survey of graduates confirmed that fewer than 30 percent had jobs in it. So the college raised the program’s standards to a level employers told them they needed.

“We didn’t pay a great deal of attention to this data while the economy was growing and unemployment was low—but when the recession began, it became clear that we could no longer assume that program completion would result in employment,” said Rock Pfotenhauer, dean of career education and economic development at Cabrillo College.

So quickly have employers’ demands shifted that Archana Mani found her master’s degree in information systems, which she earned in 2001, insufficient to get her a job after she took a break to raise her children. So she enrolled at Oakland Community College near Detroit, which had discovered—through spidering technology—an urgent need for programmers who could build and test new software applications. Within three months, it began offering an accelerated course to train some.

“I can see that demand, now that I’m at work,” said Mani, who completed the program and got a position with a quickly expanding branch of Hewlett-Packard in Pontiac, Mich. “They are looking to fill a lot of jobs.”

The Hechinger Report provides in-depth, fact-based, unbiased reporting on education that is free to all readers. But that doesn't mean it's free to produce. Our work keeps educators and the public informed about pressing issues at schools and on campuses throughout the country. We tell the whole story, even when the details are inconvenient. Help us keep doing that.

Join us today.

Letters to the Editor

3 Letters

At The Hechinger Report, we publish thoughtful letters from readers that contribute to the ongoing discussion about the education topics we cover. Please read our guidelines for more information. We will not consider letters that do not contain a full name and valid email address. You may submit news tips or ideas here without a full name, but not letters.

By submitting your name, you grant us permission to publish it with your letter. We will never publish your email address. You must fill out all fields to submit a letter.

  1. Pingback: News | HEpenguin
Submit a letter

Your email address will not be published. Required fields are marked *