Research into speech therapy interventions usually requires trained listeners to evaluate a student’s progress by listening to his or her speech and rating the sounds for correctness, but ratings from speech language pathologists can be costly to obtain and take a long time to gather.
Enter the world of crowdsourcing. A study published in the January/February edition of the Journal of Communication Disorders finds that the aggregated opinions of nine untrained listeners recruited through a crowdsourcing platform can provide equally valid assessments compared to the averaged results of three trained listeners.
The findings could help speech researchers, who in turn could be able to develop interventions more quickly, said the authors of the study, who are all based at New York University.
And eventually, crowdsourcing could become a tool for speech pathologists as well, who could use crowdsourcing both to evaluate student’s intelligibility and to gauge that student’s progress in therapy by using unbiased listeners, said Tara McAllister Byun,the study’s lead author and an assistant professor in the university’s department of communicative sciences and disorders.
“We do hope, in the future, that it could be easy to upload sound samples so that a speech pathologist could keep track of how they’re doing,” Byun said.
Speech and language disorders are second only to learning disabilities in the percentage of students who are receive services under the Individuals with Disabilities Education Act. About 18 percent of students were in this category in 2011-12, the most recent year for which statistics are available (learning disabilities accounted for about 40 percent of students served under IDEA that year.) For young children, speech and language disorders were more prevalent—about 45 percent of children ages 3 to 5 who received special education services did so for some kind of speech impairment.
Recruiting listeners for pennies
For this report, researchers used 153 amateurs recruited through Amazon Mechanical Turk program, which allows requesters to submit “microtasks” that are better done by humans than by computers. The listeners received 75 cents for going through a 20-word training task, listening to 100 samples of children reading words with the “R” sound, and rating those samples for correct pronunciation. The process took 23 hours and cost $167, including fees to Amazon.
The researchers then compared the results to those provided by 25 trained listeners. The final result was that the averaged results from nine or more amateur listeners on a given sample had a high level of agreement with the averaged results of three trained listeners, all of whom had training in speech pathology, most at the master’s level.
Untrained listeners can offer another benefit, the study said: Speech therapy is intended to make a person more intelligible to the average listener, so measuring therapy results using real-word listeners is likely to show if the intervention is making a significant difference.