Observation index ensures efficient audit
IRB New Year's Resolution: Focus on QI
Site issues easily tracked, measured
Research protection programs that conduct audits or on-site reviews for monitoring compliance typically collect a wealth of information about research practices and trends. However, the wealth often remains untapped because the review comments and findings are left in individual data silos.
The key to making this information work for a research program is to collect information that can be generalized, using analyses that can translate from one review to the next.
The quality improvement program at Partners HealthCare in Boston developed a tool to mine these data. Called an observation index, the tool contains more than 100 different observations that can be made by an on-site reviewer, says Sarah White, MPH, CIP, director of the human research QI program at Partners HealthCare.
"The QI teams are collecting such great information that it seems like a no-brainer to have a database for storing all of this stuff," White says.
The electronic tool has cut the time on-site review specialists spend on writing reports down to two or three hours from five or six, White says.
"It was a significant time saver and enabled us to be more efficient and do more audits," she says. "Also, it enabled us to do more education."
The database contains the itemized observations made during on-site reviews.
"It's a standardized index of observations we might make during an on-site visit," White says. "For instance, one is that the study rep has not signed a consent form or that a consent form is missing."
The data collected from the observations makes it possible for the institution to track its activities and see how many IRB request reviews have occurred. They also can see how much time is spent with any particular research group and generate information about where there is a need for additional training and information, she adds.
Other observations involve broad categories of observations related to federal regulations, the institution's policies, and good clinical practice guidelines. Here are several more observations on the index:
- Study documentation has been inappropriately obscured.
- Exceptions to inclusion and exclusion criteria have been made for subjects. There is no documentation that these exceptions were submitted to the IRB.
- Subject dates are not documented on all the consent forms.
"Because it's broad and somewhat generic, the specialist will make an observation and put it in a bucket, and then we can query back into those buckets and see how many times this [finding] occurred," White explains.
With data from the itemized observations, the QI program can assess whether to make system changes, such as revamping a policy or improving education, she adds.
Before creating the index, the standard practice was to write reports in Word, change them to PDF files, and send the files to investigators without storing and categorizing any of the findings in a database, White says.
"We'd have to scramble and cobble things together to get the information we needed, and now all we have to do is put in a query, and the information comes out," White says. "It's great."
The observations are organized according to topics, such as "study staff training" and "adverse events." QI specialists provide the detail to the broad observation in the actual report to the investigator. Examples might include:
- Training: There is no documentation that study staff have been trained on the protocol.
- Adverse Events: Tracking, assessment, and/or reporting of adverse events to the IRB and/or sponsor is not adequately documented.
Each of these observations also has a column for the applicable regulatory reference. The QI program references federal regulations, institutional policy/guidance and ICH Good Clinical Practice. Examples of these from the Partners' index include:
- The investigator should ensure that all persons assisting with the trial are adequately informed about the protocol, the investigational products, and their trial-related duties and functions. [Partners Guidance: Principal Investigator's Responsibilities; GCP 4.2.4]
- The principal investigator is responsible for documenting and reporting adverse events in accordance with institutional policies, federal regulations and good clinical practice guidelines. [Partners Policy; Adverse Event Reporting; GCP 4.11.1]
A final column is for suggested corrective actions, including these:
- Document study staff training.
- Obtain documentation and/or report adverse events as applicable. A log can be used to ensure adverse events are reported in a timely manner. A template can be found on the Partners' website.
The index has multiple uses, White notes.
"As far as trends we can identify by use of the database, we can identify a common observation, make a systematic change, such as a clarification in a policy, and track how/if that systematic change worked," she says.
Creating an observation index is a long-term time saver, and it opens up many possibilities for a more efficient human research protection process and program, but it does require a significant up-front commitment of time and resources, White notes.
"It was a huge undertaking," she says. "I would say there were two of us over several months who were really looking at this thing and developing it from our knowledge and observations over the years."
The QI team also referred to federal regulations to develop some of the general observations.
"If I opened the federal regulations and looked at what a clinical investigator is responsible for, like ensuring the subject receives a copy of the informed consent, then that became an observation in a broad category, and that went into the index," White explains.
"It's important to point out that we use that index as a means to track and analyze data, but once a report is generated from the database, the specialists go in and include more details of the findings," she adds. "For example, the generic observation might be 'study procedures were changed without IRB approval'; in the report, the analyst would identify how the survey questionnaire and blood test were not collected."
Once the QI team compiled a long list of general observations, an analyst and data expert took the information and adapted it for an electronic checklist that is flexible enough that new observations can be added as necessary, White says.
The electronic observation index is user-friendly, she notes.
"When specialists come back from an audit, they plug in all the observations they made into our database, which houses the index, and it generates a report in Word," White explains. "Then they go in and insert their details; I review the report and it is complete."
