Note: Through March 27th I’ll be sharing this Digital/Edu space with some excellent professionals in the area of learning and innovation for one weekly guest post. In addition to bringing new voices and ideas to our readers, this will help me as I finish the first draft of my forthcoming book, The Test, on the past, present and future of testing in public schools, to be published by Public Affairs in 2015. Today’s guest poster, Justin Reich, is the Richard L. Menschel HarvardX Research Fellow and a Fellow at the Berkman Center for Internet & Society. He writes the EdTechResearcher blog for Education Week.
Framing MOOC Research: How Comparisons Shape Our Judgments
Last month, my colleagues and I on the HarvardX and MITx research teams jointly released a series of reports about the first 17 courses launched by HarvardX and MITx on the edX platform. We released a synthesis report with findings about all of the courses, and then 15 additional reports examining individual courses in more detail.
We tried to provide the public and our internal stakeholders with data that instructors can use to create better courses and that people can use to judge the state of the enterprise. We are keenly aware, however, that our data don’t have a single story to tell, and how people read our research depends upon how they approach the subject.
Here are two sets of facts, two possible frames for thinking about HarvardX courses:
Set 1: In September 2009, Boston’s WGBH published to YouTube a series of 12 videos from Michael Sandel’s Justice course at Harvard University. Each hourlong video is a combination of lecture from Sandel and facilitated Socratic dialogue among the hundreds of students who take his class every semester.
On YouTube, the first video has been played almost 5 million times. The second video over a million times. The next ten videos have been played about 300,000 to 400,000 times each. All told, the series has about 10 million views.
As a back of the envelope calculation (ignoring people who watch videos multiple times, who don’t finish videos, etc.), it seems unlikely that more than 6% of people who started the series finished the whole thing.
Set 2: Researchers at the Community College Research Center at Columbia University’s Teachers College have done some very interesting work examining online courses in community colleges.
Di Xu and Shanna Smith Jaggers have recently published studies that show that online course completion rates in two large systems in Virginia and Washington are lower than face-to-face course completion rates. In Virginia, completion rates in face-to-face courses were 81 percent, while online completion rates were 68 percent; in Washington, the rates of completion were 90 percent for face-to-face versus 82 percent for online. While online completion rates lag behind on-campus counterparts, the vast majority of students in both conditions earn a passing grade in the courses in which they enroll.
The first set of facts concerns online media available to anyone for learning and personal growth. The second set of facts concerns structured learning experiences offered by institutions of higher education. Both are are potential frames of reference for interpreting our research about HarvardX and MITx courses, each could lead to different comparisons and different judgments.
One of the most important findings from our research is that people use materials in edX courses in all kinds of ways. The 2012-2013 MITx and HarvardX courses had a little over 800,000 registrations. A little over 43,000 people earned a certification of completion in a course. Nearly 36,000 people opened up more than half of the units of a course, but did not earn a certificate. Over 450,000 people viewed less than half of the units of the course (without earning a certificate), and nearly 300,000 people who registered for a course never entered the courseware at all.
The figure below is a scatterplot of all 800,000 registrations. On the y-axis is the student’s grade in the course, and on the x-axis is the percentage of the units (or chapters) in a course that the student opening. As you can see, nearly the entire possibility space is full of points. There are people who are completing every snippet of course content, people who are auditing courses and ignore assessments, people dabbling in a fraction of the course, and people who are never showing up at all.
How should we judge these findings?
One thing we can do is compare these patterns of behavior to the patterns of behavior in community colleges. In community colleges, we are keenly concerned with completion rates. Courses are expensive, and students who fail and drop out not only miss the benefits of learning and certification, but they also lose the money they invested in enrollment. We might make the comparison that only about 6% of HarvardX and MITx registrants finish a course, but in Virginia 68% of online community college registrants finish a course.
We could also compare the online learning content hosted on edX to the PBS Justice videos hosted on YouTube. In online content, we expect to see a funnel of participation. We expect, that on any website, there will be some number of people who navigate to the site, a smaller number of people who register, a smaller number of people who participate in some way, and then a smaller yet number of people who engage the deepest possible ways. When we compare the HarvardX version of JusticeX with the PBS version of Justice, we find very similar patterns of participation.
Patterns of persistence and completion in edX look pretty typical when compared with engagement funnels of online media, and they look pretty lousy when compared to community college retention rates. So which is the right comparison?
Faculty intent should play an important role in deciding the right frame of reference, the right yardstick, for judging open online courses.
Some MOOC faculty are primarily interested in sharing their ideas and relatively uninterested in certifying learners’ competency. Michael Sandel, through PBS Justice, justiceharvard.org, and JusticeX, is primarily interested in helping more people learn more about moral reasoning rather than certifying people at a particular level of skill or knowledge about moral reasoning. If he’s trying to maximize the number of learning experiences that people have, of both lighter touch and deeper engagement, then the frame of online learning media seems to be a fairer point of reference for judging the public impact of JusticeX.
By contrast, many of Sebastian Thrun’s courses at Udacity, especially his pilot programs with San Jose State University and Georgia Tech, are explicitly designed to replace or complement typical courses of study in higher education. Familiar higher education settings make a more sensible frame of reference in understanding these efforts.
As new forms of online learning proliferate, no doubt there will be even more sensible ways of contextualizing and framing new courses and new platforms for learning. As we debate how open online courses might reshape the future of higher education and lifelong learning, it’s worth paying close attention to how the points of comparison that we choose frame our interpretations and judgments.
At The Hechinger Report, we publish thoughtful letters from readers that contribute to the ongoing discussion about the education topics we cover. Please read our guidelines for more information. We will not consider letters that do not contain a full name and valid email address. You may submit news tips or ideas here without a full name, but not letters.
By submitting your name, you grant us permission to publish it with your letter. We will never publish your email address. You must fill out all fields to submit a letter.