Reviewing research on Mechanical Turk
IRBs must consider anonymity, informed consent
Amazon's Mechanical Turk offers investigators the chance to survey thousands of respondents quickly and cheaply via computer while protecting their anonymity. Once IRBs understand how the system works, approval should be a slam dunk, right?
Not always. Panos Ipeirotis, PhD, an associate professor at the Leonard N. Stern School of Business at New York University in New York City, who has studied the use of Mechanical Turk, says IRBs have raised specific questions about its use in research.
Mechanical Turk allows an investigator (or "requester" in the platform's parlance) to post a Human Intelligence Task (HIT) and open it up to thousands of workers ("Turkers") accessing the site from all over the world.
A Turker can choose a HIT, read through it before deciding to perform it, and carry it out, usually in only a few minutes. Once the task has been completed, the requester is able to view it, decide whether it has been fulfilled properly, and authorize a small payment from his or her account to the worker's account. The requester never learns personally identifiable information about the worker.
"I think Amazon's terms of service are really very nicely in line with the type of thing you'd want for an experimental design where you want to keep the subjects anonymous," says Chris Callison-Burch, PhD, an associate research professor in the computer science department at Johns Hopkins University in Baltimore, who has used Mechanical Turk for research. "It's pretty much how you would 'anonymize' data from a study that you did in person where you would give each subject a serial number."
Ipeirotis says he's heard of IRBs asking for more detail about precisely how Amazon protects the personal information that it holds details that Amazon tends to be tight-lipped about providing. In fact, Amazon officials did not respond to a request from IRB Advisor about how the company protects data.
"They're not going to provide explicit information about how they monitor data and who has access," he says. "I just don't think they're going to answer such questions."
Compiling a profile
Ipeirotis says there is another concern about confidentiality that has nothing to do with Amazon's data protection system. Each Turker is issued an individual ID number, which does not change. While a researcher may not know the name of the person assigned to a particular number, if the subject participates in many studies, a fairly detailed personal profile could still be derived from the answers, he says.
"You know from one set of data that Subject No. 123456 is a male and from another experiment what his age is and from another survey you might learn his sexual orientation," he says. "You can ask about zip code, family status, attitude about movies, websites visited. You could build a very comprehensive profile of an individual by using connecting information."
IRBs concerned about this possibility can require investigators to delete the worker's official Amazon ID number and assign them a different ID that is unique to a specific study once the data is collected.
"If people are really concerned about the privacy of their subjects, this should be a standard practice in IRB proposals," Ipeirotis says.
Other issues raised by the use of Mechanical Turk:
Informed consent because the entire interaction between the investigator and subject is online and anonymous, it is impossible to obtain a signed consent form. Consent can be obtained in a few ways, Ipeirotis says. If the IRB permits it, a requester can include consent in the request, explicitly stating that the task is research and that acceptance of it constitutes giving consent.
A more in-depth consent statement may be part of a "qualification," which is a test or other screening question used to determine whether the worker is eligible to work on a HIT. That qualification would show up as a separate screen before the worker can participate. He or she could read the consent statement and would have to click a button saying "I consent" before being able to continue to the study itself.
Payment for unsatisfactory work Requesters have the option of declining to pay for work completed by a Turker. The reason for this is to weed out "bad responses" either by people who simply randomly press buttons or more sophisticated frauds in which people create computer software that makes it appear as though a human is answering.
The latter problem is not a serious concern for researchers, Ipeirotis says, since there's not much money in filling out short research surveys. But it is possible to throw out responses that do not appear to have been done properly and to refuse to pay the Turker. He notes that IRBs usually require that a person be paid regardless of whether he or she withdraws from a study.
One answer to that problem, he says, is a separate qualification that forces participants to answer some questions unrelated to the research showing that they're paying attention and to refuse to let them participate in the study if they don't pass the test.
Amazon's Mechanical Turk offers investigators the chance to survey thousands of respondents quickly and cheaply via computer while protecting their anonymity. Once IRBs understand how the system works, approval should be a slam dunk, right?You have reached your article limit for the month. Subscribe now to access this article plus other member-only content.
- Award-winning Medical Content
- Latest Advances & Development in Medicine
- Unbiased Content