Electronic medical records (EMR) were always supposed to make things better for emergency clinicians, but they have turned into a major source of aggravation. Critics say they operate too slowly, interfere with the natural workflow of a busy ED, and forces clinicians to work harder to find information. New information suggests why, at least in part, this may be the case.
An analysis of the usability tests performed by 41 of some of the largest EMR vendors shows that 34% of vendors did not meet certification standards established by the Office of the National Coordinator for Health Information Technology (ONC), specifying that they state their user-centered design process. Further, 63% of the vendors failed to include at least 15 representative end-user participants. In fact, only 15% of the vendors used at least 15 participants who had clinical backgrounds. The authors, led by Raj Ratwani, PhD, scientific director of the National Center for Human Factors in Healthcare at the MedStar Institute for Innovation in Washington, DC, note that one vendor didn’t use any clinicians in its usability testing, 17% of the vendors didn’t have any physician participants, and 5% used their own employees to conduct the usability tests.1
“Certainly our results don’t speak to all vendors, but the results do show that there seems to be tremendous variability in the vendors’ user-centered design processes, and their employment of different usability processes overall,” Ratwani observes. “Unfortunately, we are seeing some vendors that are running [usability tests with] only a handful of participants, and they are not stating their user-centered design process.”
Ratwani stresses that some vendors do an exceptional job with their usability tests, going above and beyond the certification requirements. When following such practices, Ratwani notes that it is clear from the literature that 95% of any usability challenges that a user might face are captured. However, he notes that the high variability in the user-centered design processes used by vendors is a problem.
“If all vendors were adhering to the certification requirements, we really should not be seeing so much variability,” he says.
What is not clear from the study is why EMR products that have not met the standards for usability testing have been certified.
“It could be that the authorized certification bodies that are tasked with reviewing the actual reports from the vendors may not have clear guidance as to what the requirements actually are,” Ratwani suggests. “The requirements may be a little bit unclear and there could be some confusion on what vendors are actually supposed to do, but that is an open question as to why there are vendors who appear to not be meeting the requirements who are being certified.”
Despite the troubling findings, there are things that emergency providers can do to ensure that their own organizations select products that meet certification standards, and that these products are optimized to meet their needs. Further, it is important to understand that while individual clinicians may prefer one EMR product over another, whether or not a product meets usability standards is another matter entirely.
Consider components of usability
Usability is not in the eye of the beholder, Ratwani stresses.
“It is a subcomponent of human factors, which is a science, and there are quantitative measures of usability,” he explains, noting that the measures generally fall under three categories:
-
effectiveness, which captures what kinds of errors participants make;
-
efficiency, which is how long it takes for people to complete tasks;
-
satisfaction.
Ratwani notes that the certification requirements reflect guidelines that have been established by the National Institute of Standards and Technology (NIST), which recognizes the three components of usability.
While there are well-developed measures around each of these three different components, there will be variability in the participants and the speed with which they perform tasks, Ratwani says.
“Include enough participants to ensure that you are capturing the majority of what people would do when they are working with these tasks,” he explains. “When the tests are performed properly, you generally end up with a usable and safe product.”
Usability testing is certainly not unique to healthcare, Ratwani notes.
“If you looked at any other industry, whether it be defense, aviation, or nuclear regulation … each of those different domains takes usability very seriously and measures it in a quantitative way to ensure that a product is usable and safe,” he says.
Ratwani acknowledges that sometimes there are tradeoffs between safety and efficiency, and that those decisions should be made by the developers and the designers of the software that goes into EMR systems.
“That is not a new problem. We have seen that in several industries,” he says. “But all those components are measured.”
Participate in usability testing
While some of the chief critics of EMR products are the physicians and nurses who use them, EMR vendors say it is difficult to get these end-users to actually participate in usability testing. Ratwani has visited a number of EMR vendors to observe their user-centered design practices and hear about their challenges. He acknowledges that a number of these companies struggle to assemble representative groups of users to test out their products.
“The ability to get physicians and nurses to engage in the user-centered design processes and be participants in these studies … is difficult and expensive,” he says.
However, Ratwani emphasizes that usability tests are a core component of the process.
“If we suggest and even require vendors to engage in these usability processes, which is the right path, that means that end-users need to be willing to participate in these studies,” he says. “We can’t blame only vendors for the challenges we are seeing in [EMR] usability. This is not a vendor-centric problem. There are several different things we need to change in order to improve these products and one of those is getting end-users to participate in these studies.”
What else can frontline emergency clinicians and ED administrators do to improve the usability of their EMR systems? Ratwani says that it is critically important to document the challenges with the system and to convey these challenges to the vendor.
Busy clinicians develop work-arounds, becoming accustomed to various capabilities and functionalities within the EMR that are not working properly, Ratwani notes.
“However, without reporting those issues, documenting them, and sharing them with the [EMR] vendor, it is very difficult for the vendor to make improvements,” he says.
It will take extra time, and clinicians may not receive immediate feedback from such reports, but without capturing and reporting problems, the vendors will not be aware of them, Ratwani adds.
Ask about user- centered processes
For hospital administrators or ED professionals who are in the market for a new EMR, be sure to investigate the usability processes that each vendor has employed in developing their product, Ratwani advises.
“There is currently no Consumer Reports for [EMR] vendor products, but that shouldn’t stop clinicians from querying vendors about the user-centered processes that were employed,” he notes.
For instance, Ratwani suggests providers ask questions such as:
-
How do you build a usable product?
-
How many participants tested your product?
-
What were the backgrounds of the testing participants?
-
Was the EMR tested with EPs and emergency nurses?
-
Under what conditions was the EMR tested?
-
Was the EMR product tested with appropriate use cases that represent the way EPs and nurses perform their work?
Such questions reveal the usability and safety of a product, and provide clinicians a better understanding of what processes vendors are using, Ratwani observes.
“That can shape your decision-making in selecting a product,” he says. (See below: “Look out for rapid task-switching, potential for errors")
While some vendors give clinicians an opportunity to use their EMR products before a purchase decision, the value of such an experience is limited because the EMR on display does not actually represent what the clinicians will use once the customization process has occurred, Ratwani explains.
“What you are using early on during the decision-making process doesn’t always match what you get when you actually purchase and use the product,” he says. “That discrepancy can be a real challenge … and points to one area where we need to focus when we start talking about how we might change or modify certification requirements to really make … usability more optimal.”
Engage with EMR vendors
Christopher Corbit, MD, FACEP, the chief medical informatics officer for Emergency Medicine Physicians, suggests the alignment of financial incentives affects the usability of EMRs.
“The [EMR] vendors don’t sell to end-users. Typically, they sell to hospital boards and CEOs, and a lot of the decisions come from the CFO on which products to go with,” he explains. “I have been through many vendor selection committees, and there is always talk about how the end-users would feel. [The committees] bring the vendor in and perform a demo. The doctors will take a quick peek, but they never engage in a real, in-depth, hands-on, full analysis of exactly how [the EMR product] is going to affect workflows.”
Therefore, EMR development is not geared toward clinicians, but rather what makes the system work best for administrators and hospital executives, Corbit notes.
“That is the environment that makes it difficult for physicians to use [EMR systems] because they don’t put the majority of the work into fine-tuning the workflows and the user interfaces for clinicians to use the system,” he says.
However, Corbit explains there are some steps that could mitigate this problem.
“The issue lies both with administrators and the physicians. A lot of physicians don’t put the time in that they need to evaluate the systems because they are very busy, and [usually] this is uncompensated time,” he explains. “Hospitals don’t budget for physicians to take a significant amount of time to look at the workflows.”
Corbit also emphasizes that EPs need to be part of the vendor selection process, and not just because they would pick a better system for their own workflow purposes.
“If they are engaged from the beginning, even if they still end up with the same EMR system … the amount of effort that they will put into it will be higher because they have been part of the process.”
Further, once an EMR vendor is selected, clinicians should be brought into the process right away to discuss building the system.
“Some systems are more built out than others, but each one still requires work. Getting clinicians [involved] at the very beginning and mapping out workflows is not done as well as it could,” Corbit says. “Many physicians don’t want to do this without some sort of compensation, and, often, hospitals don’t offer that, or don’t budget it into their planning, so they don’t have it.”
However, Corbit observes that if clinicians are not engaged in this build-out process, there is often dissatisfaction with the system.
Assemble in-house expertise
Corbit agrees with the notion that there is not enough communication between end-users and EMR vendors.
“Typically, end-users don’t talk to vendors, so there needs to be some work on [the part of] hospitals to create a committee that looks at these issues, has a direct line to the vendor, and to work directly with the vendor,” Corbit suggests.
However, Corbit adds that what many physicians don’t understand is that a lot of the improvements that clinicians would like to see are actually passed on to a hospital’s IT staff to carry out.
“Each hospital can make tweaks in the system to make it better,” Corbit notes, but adds that hospitals often don’t have the staff or in-house expertise to make the needed improvements. “A systems analyst may not have the training or knowledge to really perform some of the more advanced work in the EMR. I have seen some hospitals that do have [this expertise], and the clinicians are much happier.”
Consequently, Corbit would like to see hospitals take steps to ensure they have the appropriate expertise to be responsive to requested tweaks to their EMR systems, and to develop processes for handling such requests. But he reiterates that physicians need to be more engaged.
“Employ someone who has an interest in [health IT], wants to take up the cause, and wants work on this,” he advises. “They need to develop the networking and the relationships to get their voices heard in the hospital, and to get connected with the vendor.”
Corbit explains that this physician representative needs to work with the hospital’s IT staff, get to know both the hospital’s CIO and CEO, and to push to get on committees that talk about information technology and EMRs. Further, the physician representative needs to be on the team at the hospital that talks directly with the vendors, Corbit observes.
“Be there at the table because the vendors need to hear from the hospital,” he says.
There is no question that clinicians tend to get frustrated with the process, but there are times when vendors do listen, Corbit observes.
“Vendors aren’t evil empires,” he notes. “The problem is the financial incentives and who the vendors are trying to please, and unfortunately, right now [a lot of them] are not focused on the users.”
Participate in user groups
One tactic that can be very helpful in guiding improvements to EMR systems is for clinicians to engage in EMR user groups. Various EMR vendors offer some of these, but practice groups can also develop their own user groups. For instance, Corbit notes that EMP has developed a user group around each EMR used so that clinicians from different hospitals can come together and share ideas they have employed to improve the system.
“There is a lot of variability with the same EMR between different institutions,” Corbit notes. “I have seen how each hospital handles their build and what they have done with their build. Some have done amazing things with the tracker board, and some have done amazing things with CPOE [Computerized Physician Order Entry].”
In fact, this sharing of ideas within the user groups gives Corbit hope that EMRs will eventually fulfill their promise.
“I have seen a lot of good things,” he says. “The vendors take pride in their systems and they want them to be the best, so over time [they] will chip away. [The improvement] is just slower than what people were hoping for.”
-
Ratwani R, et al. Electronic health record vendor adherence to usability certification requirements and testing standards. JAMA 2015;314:1070-1071.
-
Christopher Corbit, MD, FACEP, Chief Medical Informatics Officer, Emergency Medicine Physicians, Canton, OH. E-mail: [email protected].
-
Raj Ratwani, PhD, Scientific Director, National Center for Human Factors in Healthcare, MedStar Institute for Innovation, Washington, DC. E-mail: [email protected].
