Ethical Approaches to Obtain High-Quality Crowdsourced Data

By Stacey Kusterbeck

Researchers increasingly are using online recruitment (“crowdsourcing”) for studies. Rather than relying on undergraduate panels, such as college freshmen completing studies for credit, or basic convenience sampling using social media posts or flyers in classrooms, platforms such as Amazon’s Mechanical Turk and Prolific allow researchers to post their studies as “jobs” for online workers to complete.

This approach can provide researchers with access to large samples of participants from diverse demographic groups.

“In many cases, data can be obtained quickly, relative to more traditional recruitment methods,” says Jon Agley, PhD, deputy director of research at Prevention Insights and an associate professor in the School of Public Health at Indiana University Bloomington. However, crowdsourcing also poses some unique ethical concerns, many of which are related to data quality.

Crowdsourced “workers” are completing studies remotely, rather than under direct supervision in a laboratory environment. Individuals are signing up for a task for which they expect to be paid a specified rate, which carries different expectations than someone being paid an incentive for participating in a study.

“There is a delicate balance between the need to ensure that participants are completing a study as intended, and the temporary employee-contractor relationship associated with posting a job to be crowdsourced,” says Agley.

On the one hand, researchers have an ethical obligation to use high-quality data when conducting research. Yet, researchers also have ethical obligations to study participants who expect to be paid for the data they provided. “It’s a complicated topic. Both perspectives have merit, but there is a degree of tension between them,” says Agley.

Agley and colleagues co-authored a recent paper examining the competing interests of study participants and researchers in this scenario.¹ The authors offer strategies that allow researchers to reject low-quality data while at the same time treating participants fairly. “It’s a balancing act for researchers to avoid low-quality data while at the same time meeting ethical standards for human subjects research,” says Agley.

Without good quality control, researchers may spend time and money — both their own or others’ — on a crowdsourced study only to find out that participants provided low-quality data when they review the data or attempt to analyze it. In extreme cases, a dataset may turn out to be composed primarily of low-quality data and, therefore, be unusable.

This poses an ethical concern because costs, in terms of researchers’ time and financial resources, are invested in obtaining the data. “And it is important to ensure that we exhibit good stewardship of these resources,” argues Agley.

Furthermore, if researchers do not detect bad data, the study’s findings are published, and the study makes recommendations about clinical practices, there can be negative downstream effects on decisions about whether clinical interventions are effective. For example, a study using low-quality data may suggest incorrectly that a set of questions can be used to effectively assess for a healthcare risk. In addition, such studies also may inappropriately suggest the need for more research on the same topic or idea. “Studies rarely exist in isolation. A conclusion based on bad data may spark additional studies, which then consume more resources,” says Agley.

When researchers are considering data quality issues, they also need to consider ethical obligations to crowdsourced study participants. “Participants drawn from crowdsourcing platforms deserve to be treated fairly and to have confidence that they can expect payment if they do good work,” says Agley.

Here are some of the practices the authors recommend:

• Be clear that there is a chance participants could be rejected. This information could be stated in a study information sheet or in other disclosure materials that are presented to participants before the study begins.

• Situate quality control questions early in the study, before the participant spends additional time or provides more than a nominal amount of data. This way, if someone is rejected, they are not being denied compensation for work that has been completed.

• When designing study protocols, be clear about how and when you will reject cases. For instance, disclose to the institutional review board (IRB) that you only intend to pay participants “who pass quality control checks and who complete the study fully.”

“The goal is to facilitate maximum transparency, so the IRB is aware of the things that could lead to a participant not being compensated after enrolling,” says Agley.

• Reject requests for payment that clearly are fraudulent. Some people demand payment inappropriately, either from the researcher or the IRB.

If a participant fails a quality check, leaves the study, and then submits a false completion verification form, such requests should be rejected.

“It’s important to be clear with your IRB beforehand how this will work, to ensure everyone is on the same page,” emphasizes Agley.

REFERENCE

1. Agley J, Mumaw C, Johnson B. Rationale and study checklist for ethical rejection of participants on crowdsourcing research platforms. Ethics Hum Res 2024;46:38-46.