A Question of Quality

If students in a face-to-face course emailed their provost with concerns that their professor had stopped lecturing, chances are that someone -- a department head or an administrator -- would intervene. But what if the students were scattered across different countries and time zones in a not-for-credit massive open online course?

The issue of MOOC quality control has resurfaced in the wake of the #MassiveTeaching debacle, the MOOC-turned-social experiment that last week inspired a scavenger hunt across the internet.

By Tuesday afternoon, one observant Inside Higher Ed commenter had cracked the case. After a successful first week of “Teaching Goes Massive: New Skills Required,” Paul-Olivier Dehaye, assistant professor of mathematics at the University of Zurich, deleted the course content as part of a social experiment to show students how their data can be manipulated online. But since Dehaye had not notified anyone of his intentions, the experiment raised confusion rather than awareness.

This is not the first time a MOOC -- or even an online education MOOC -- has gone off the rails. Last year, a course on the same subject collapsed after its creators underestimated the technology required to accommodate tens of thousands of students.

Last week’s case, however, was one man’s doing. “[I don’t] mean to destroy [C]oursera, just insist to those students that there are dangers,” Dehaye said in one of many clues scattered around the internet.

George Siemens, a researcher based at the University of Texas at Arlington, said Dehaye actually deserves some credit for raising awareness about MOOC providers’ ability (or lack thereof) to control quality and manage crises.

“Coursera has been revealed as a house of cards in terms of governance and procedures for dealing with unusual situations,” Siemens wrote in a blog post. “MOOCs were developed so quickly and with such breathless optimism that the architects didn’t pay much attention to boring stuff like foundations and plumbing. What is the governance model at Coursera? Is there anything like a due process to resolve conflicts?”

Siemens is not alone in his criticism of Coursera. Although the mystery may have been solved, the discussion has flourished over the last week, with many researchers wondering if Coursera could have done more to prevent the incident.

Vivek Goel, Coursera’s chief academic strategist, discussed what went on behind the scenes in an interview last week. For about two confusing days, he said, Coursera and the university worked across their nine-hour time difference to identify the problem and restore the content to salvage the last week and a half of the course. The perception that Coursera took ages to react was “driven by the sense of urgency that things like live blogging and Twitter draw in” -- and the fact that the situation occurred during the Fourth of July weekend, he added.

“In my own experiences, when we have something disruptive happen in a classroom, it could take a lot longer for universities to figure out how to resolve it,” Goel, former provost of the University of Toronto, said. “I think this was not an event that any set of agreements or policies could necessarily anticipate.”

Ronald Legon, a former faculty member and administrator who now serves as executive director of the Quality Matters Program, said he could sympathize with Coursera. During his time as provost at the University of Baltimore, a faculty member in an online creative writing course claimed she was intentionally not involving herself in the course to let students themselves form a learning community. The instructor was eventually replaced by “someone who was going to teach [the course] in a more conventional way,” he said.

“I would say what happened in this course is not that unusual in higher ed,” Legon said. “But given the publicity attendant to Coursera and the MOOC movement... this becomes an international scandal.”

Goel also challenged the assertion that Coursera doesn’t have standard operating procedures to deal with such events. It’s a process that may take more time than if the MOOC provider had a secret back door into every course on its platform, Goel said, but one that leaves the responsibility of quality control with partner universities.

“As much as possible, we would prefer not to be the quality censor,” Goel said. “We want it to be done within the academic community.”

If Coursera were to monitor every course, Goel said, it would risk infringing on instructors' academic freedom. “There’s a very careful balance between ensuring learners have the best possible experience and the professor’s ability to guide that experience,” he said.

Coursera’s policy is more than a convenient excuse. Belle S. Wheelan, president of the commission on colleges of the Southern Association of Colleges and Universities, said quality control is a responsibility of the institution, regardless of medium.

“If they are the ones in whose name the course is offered, then they have the responsibility of ensuring the quality of that course,” Wheelan, whose organization accredits institutions in the South, said. “We as accreditors hold the institutions responsible, because it’s the institutions we accredit -- not Coursera.”

That said, Coursera isn’t absolving itself of all responsibility. It requires all users, from students to instructors, to abide by its terms of service, which contain a section on “Rules for Online Conduct.” Then there are the agreements it signs with partner universities, which Goel described as a “failsafe” that “provide[s] for ensuring the pedagogical and technical quality of the course materials.” There are also course development agreements, which includes best practices and guidelines instructors are expected to follow when they create and teach MOOCs.

Of course, agreements, guidelines and provisions are only useful if they are actually followed.

“If someone’s not going to follow the practices, that’s something that society faces in general, and that’s why we have police and courts,” Goel said. “In the academic world, we don’t need police and courts, but we have academic discipline, we have academic institutional standards, we have research standards. If people don’t comply with those, they do need to be deal with. In our set of relationships, that’s between the university and the instructor.”