Researchers Say Opioid Risk Tool Has ‘Too Many False Alarms’
/By Pat Anson
The use of artificial intelligence (AI) continues to grow in healthcare, with patient health data and behavior increasingly being used to assess whether a patient is at risk of an illness or chronic health condition.
NarxCare and Epic, for example, scan electronic health records and prescription drug databases to create Opioid Risk Scores (ORS) for patients, which are then shared with healthcare providers to flag patients at risk of opioid misuse or an overdose. Patients deemed to be at high risk may not be able to get a prescription for opioids or they may be abandoned as “too risky.”
But a new study – the first of its kind – suggests that using opioid risk scores to predict patient outcomes is flawed, with unacceptably high rates of false positives and false negatives.
The study, recently published in the Journal of General Internal Medicine, looked at Epic’s opioid risk scores for over 700,000 U.S. patients being treated by primary care providers. The vast majority of patients (99.6%) were classified as low risk, with only 0.4% considered at high risk of an overdose or OUD.
While it’s reassuring to see so many patients deemed low risk, how accurate were Epic’s risk scores in predicting patient outcomes?
Of the 702,099 patients deemed low risk, only 2,177 went on to have an overdose or OUD diagnosis within the next 12 months. That means the system correctly predicted outcomes about 99.7% of the time.
Conversely, of the 2,665 patients deemed high risk, only 185 later had an overdose or OUD diagnosis. That means Epic’s scoring system correctly predicted outcomes only about 7% of the time.
Researchers say the false positive rate of 92.2% in the high risk category means that Epic’s ORS “produces too many false alarms” and is of little value to providers.
“In this study, most high-risk patients were false positives, and most who developed OUD or overdosed were false negatives. Because these outcomes are rare, achieving adequate PPV (the proportion of cases that are accurate) is challenging. The ORS’s misclassification could undermine its external validity, leading to misallocated resources and missed interventions,” wrote lead author Stephanie Hooker, PhD, a Research Investigator at HealthPartners Institute.
“Missed interventions” in this case could mean a patient being denied opioid medication or being referred to addiction treatment, when neither move is justified.
On the flip side, Epic’s 99.7% success rate in identifying low risk patients also isn’t foolproof.
Of the 2,362 patients who experienced an overdose or OUD diagnosis, Epic’s system flagged only 185 of them as high risk.
In the end, “low risk” doesn’t mean no risk, and “high risk” doesn’t provide any certainty either.
Pain management expert Dr. Lynn Webster says no ORS — whether Epic’s or NarxCare’s – should be viewed as authoritative by doctors and pharmacists in making clinical decisions.
“Both tools can be harmful if used punitively. The NarxCare scores have shown that overestimated risk may lead to forced tapering, abandonment, or other punitive responses, which could paradoxically increase overdose risk. With Epic, the harm is a bit different: the score can both stigmatize flagged patients and falsely reassure clinicians about the much larger group labeled low risk,” said Webster, a Senior Fellow at the Center for U.S. Policy (CUSP).
In 2023, CUSP petitioned the FDA to take Narxcare’s software off the market as an invalid and misbranded medical device. The FDA rejected the petition on procedural grounds.
In the case of Epic’s ORS, Webster says it is a mistake to combine OUD and overdoses into the same prediction model. Someone can overdose without having OUD, while someone can have OUD without ever experiencing an overdose.
“Opioid risk tools will always struggle to predict overdose death risk because overdoses can occur in patients who have no opioid use disorder and no aberrant drug-related behavior,” Webster told PNN. “Some patients overdose even when they take their medications exactly as prescribed. Overdose can also occur because of comorbid medical conditions or other factors unrelated to OUD.”
As flawed as they might be, Epic and NarxCare are already embedded in the U.S. healthcare system. Epic’s MyChart software stores data on over 190 million patients, while NarxCare is used by Walmart, Rite Aid, CVS and other major pharmacy chains to analyze patient risk.
“Whether the score comes from NarxCare or Epic, the core danger is the same: once a proprietary risk label is embedded in the chart, it can take on a false authority that changes how patients are treated,” says Webster.
