By Melinda Young
The creators of an online survey, designed to explore breast cancer and reconstruction among African American women, began enrolling participants slowly. Surveys trickled in, accumulating 30 completed surveys over several weeks. Suddenly, the survey rose in popularity. Within two days, more than 80 completed surveys came in.
“I immediately knew what was happening, and shut down the survey,” says Lorraine R. Reitzel, PhD, FAAHB, FSRNT, IRB chair and professor at the University of Houston. Marketing for the survey had not changed, so investigators knew it could not have recruited so many new, eligible participants that quickly.
Another sign that something was wrong with the completed surveys was the timing in which were submitted. Batches of surveys arrived in two-minute intervals, says Shahnjayla K. Connors, PhD, MPH, CPH, assistant professor of health and behavioral science in the department of social sciences at the University of Houston-Downtown.
Other signs of a breach included suspicious responses, unusual email addresses and patterns, responses from outside the United States, and missing contact information.1
Incentives Attract Bots, Scammers
Connors and Reitzel believed the survey had been breached. One possible cause was the way investigators marketed the survey. A link went out to groups connected to breast cancer survivors — but anyone could share the survey link, and they probably shared it with the wrong people.
Another factor was the $20 Amazon gift card investigators offered participants as an incentive for their time completing the survey. “We sent the gift cards electronically,” Connors says.
That small incentive was enough to attract electronic bots, which are software programs that can execute commands and perform routine tasks. The fact the surveys were dropped two minutes apart, and some were blank or did not include contact information, suggested they were not filled out by a person.
“We had to go through every single survey; it was pretty labor intensive,” Connors explains. “At first, I had to look for things that didn’t look right, something in the diagnosis, or something like they said they made $80,000 a year, but had Medicaid insurance.”
If any red flags arise, the researchers put the survey in a separate pile.
Connors began the laborious work of verifying each survey. She called the numbers they provided to see if the number was connected to a woman. In some cases, the numbers were disconnected or were attached to a doctor’s office or another random place.
“We sent each woman an email, saying that we looked at their application and needed further review and would they mind contacting us,” Connors says.
Several people responded with anger, saying the researchers were running a scam and that they had put in their time and wanted their gift card, she recalls. Two people went through the additional verification and were shown to have completed legitimate surveys, but the rest were likely fake.
Connors was determined to fix the breach and ensure the only survey data collected would be from eligible participants. This study was important to her, personally, and to the body of knowledge about Black women and breast cancer.
“Particularly for women of color, there’s not a lot of research out there, and I wanted my data to be valid, useful, and meaningful,” Connors says. “Otherwise, it defeats the purpose.”
Ensure Data Integrity
As the breast cancer reconstruction survey continued, researchers asked potential participants to contact a member of the research staff. “Then, we created an individualized survey link for that person,” Reitzel says. “We thought bots were less likely to reach out to the investigator to get a link.”
While this created another barrier between participants and research staff, it was an effective step toward ensuring data integrity. “We did not reach the target of 100 breast cancer survivors, but we reached 59 people in the end, when funding was over,” Reitzel says. “The extent to which we were hampered is unknown.”
Issue Is Widespread
Shortly before the study enrollment was breached, Reitzel heard a cautionary tale from a colleague about seeing surveys churning out every couple of hours. Something had changed in the enrollment, and it was not because investigators had changed their recruitment strategy or tapped into a new well of potential participants.
“As she described this issue with her study, I started asking her questions,” Reitzel says. “I asked if she had alerted the NIH [National Institutes of Health] and the IRB.”
In these breached survey studies, the earliest surveys were returned by the people targeted in recruitment. But there was evidence the later rush of completed surveys was completed by electronic bots or people who did not meet the study’s enrollment criteria.
“They wonder why they’re getting this large input of people when, really, they should have only 30 participants or some smaller numbers in chunks of time, depending on how they’re rolling out their marketing,” says Danielle A. Griffin, EdD, CIP, associate director of research integrity and oversight at the University of Houston.
Griffin and Reitzel learned increasing numbers of investigators were experiencing similar issues. “In our compliance office, we learned this had happened to even more people,” Reitzel says. “All these students, doing student research, trying to get through their dissertations and theses, and when their surveys are breached, they’re probably very excited about making their survey numbers.”
Unfortunately, what looks like good news actually is a mess they will have to clean up. They may never learn exactly what was behind the breach, although incentive payments are the most likely explanation.
Even studies designed to recruit students and offering course credit as incentive might be hacked, Griffin notes. Many universities use cloud-based participant management software to recruit students for studies and to offer them course credit. “We’ve had reports, and one was through a whistleblower, who knew their friend hacked the system and completed a survey twice so they could get additional credit,” Griffin says. “This student sent an anonymous email, saying, ‘Hey, you may want to check this data because I know my friend hacked the system and completed the survey and put in garbage data’ — not exact words, but a heads-up.”
Many researchers rely on these platforms and expect students to complete the survey at no cost to investigators so they are collecting data for free. “But how many are hacking it? Hopefully, it’s a small number, but if they were able to do that for one survey, then they could do it for others,” Griffin says. “This just questions the reliability of the data.”
Breached surveys disrupt research and turn a fairly straightforward electronic process into one that requires much more time than expected. “First, they have to go through all of the surveys to weed out the ones that are valid,” Griffin says.
Sometimes, it requires investigators to perform additional research, such as contacting participants with new questions. Researchers may have to email all participants and ask them to take an additional step, such as calling into the research office, before they can receive their incentive payment. There may be no response from the false surveys, but the participants who meet the criteria may respond emotionally to the added inconvenience.
“They might be upset you’re withholding their money because they didn’t see the email asking them to call in, and so they didn’t send in a response,” Griffin says. “You’ll have participant complaints.”
In one breached study, it took several months for the valid participants to receive their incentive payment. This can affect relationships and trust with a community investigators may want to work with during future studies.
Verify Each Survey
If researchers are not cautious in verifying each completed survey, they run the risk of harming their data integrity with bad data. When researchers suspect a survey breach, they should contact appropriate regulatory entities, including NIH (if applicable) and the IRB.
The IRB asked Reitzel and Connors to show how they could protect the study’s integrity — and participants — as the study continued. (See story on preventing survey security breaches in this issue.)
“Lorraine and I came up with a plan, moving forward, of having women contact us first,” Connors says. “Before, we had a survey link on the flyer. I think that was what the issue was because we distributed it to breast cancer groups, and once it goes out you don’t know where it goes.”
After the breach, they removed the link from the flyer and required women who were interested in participating to contact investigators for more information. When they called, Connors verified they met eligibility for the study and sent them a personalized link to the survey.
Each participant received a separate link so it could not be shared with other people.
“Of course, nothing is 100%,” Connors says. “But we didn’t have any issues after sending out that link.”
REFERENCE
- Griffin DA, Castro Y, Reitzel LR. Methods to address and prevent compromises to online survey data from internet bots and fraudsters. Presented at the PRIM&R Advancing Ethical Research Virtual Conference, Dec. 1-2, 8-9, 15-16, 2020. Poster/Abstract:57.
Researchers at the University of Houston discovered a survey study had been breached. Large number of surveys poured in, with batches arriving in two-minute intervals. Other signs of a breach included suspicious responses, unusual email addresses and patterns, responses from outside the United States, and missing contact information.
You have reached your article limit for the month. Subscribe now to access this article plus other member-only content.
- Award-winning Medical Content
- Latest Advances & Development in Medicine
- Unbiased Content