Patient reviews may not be reliable way to compare hospitals

While patient-reported measures can provide valuable insight into an individual’s experience with hemophilia, a study suggests they may not be reliable for comparing outcomes across different hospitals.

“Patient-centered outcomes demonstrate low to moderate reliability for hospital-level comparisons,” researchers wrote in the paper, “Assessing the Real-World Value of Hemophilia Care: How Suitable Are Patient-Centered Outcomes for Hospital-Level Comparisons?”  published in the International Journal for Quality in Health Care.

Hemophilia encompasses several disorders marked by abnormally low activity of certain clotting proteins, leading to hemophilia symptoms such as unusually excessive bleeding and bleeding into joints. To assess hemophilia severity, doctors typically rely on objective, quantitative measures, such as the annual bleeding rate.

There’s been a push in recent years to incorporate patient-reported outcomes into clinical care for hemophilia. These measures can help researchers and clinicians better understand the lived experiences of people with the disease, and data suggest they can be useful for comparing outcomes among individuals — for example, participants in clinical trials.

However, it hasn’t been clear whether patient-reported outcomes can be relied on to compare hospitals. Hospital-to-hospital comparisons are more complicated than comparing outcomes among individuals, because each hospital will see a different mix of patients with varied disease severity. Factors such as location, socioeconomics, and random variation can affect hospital-level outcomes and complicate comparisons.

Recommended Reading

July 14, 2025 Columns by Allyx Formalejo

When hospitals aren’t prepared for bleeding disorders

Analyzing results from Netherlands center

Meaningful comparisons “require careful consideration of these methodological challenges,” the researchers wrote. “Before such comparisons can be made, the suitability of patient-centered outcomes must be evaluated to ensure that they are capable of distinguishing true between-hospital differences from random variation.”

To address this issue, the scientists analyzed data from a nationwide study of more than 800 hemophilia patients in the Netherlands. Using statistical analyses, the researchers evaluated 12 types of patient-centered outcomes to assess their reliability for comparisons between hospitals.

“This study provides a quantitative evaluation of the suitability of patient-centered outcomes for hospital-level comparisons in hemophilia care,” they wrote.

The scientists looked at two key factors. One was case mix, referring to the types of patients seen at a given hospital. Results showed that most of the outcomes assessed were not substantially affected by differences in the types of cases seen at each hospital.

The second key factor was rankability, a statistical measure that assesses the extent to which outcomes are affected by random chance. On this test, most of the patient-reported outcomes scored poorly, indicating that at the hospital level, most patient-reported outcomes are affected by random chance more than by actual differences between hospitals. The only notable exceptions were objective measures assessing annual rates of bleeding or joint bleeding.

“Our findings suggest that the value of PROs [patient-reported outcomes] in clinical care does not automatically imply their suitability for hospital-level comparisons, particularly in the context of hemophilia,” the researchers wrote. “This was reflected by the superior performance of [annual bleed rates and annual joint bleed rates] compared with PROs.”

The researchers said the data “should not be interpreted as a reason to disregard PROs altogether,” as these measures can still be useful for understanding patients’ experiences and making comparisons among individuals. But when comparing different hospitals for hemophilia, the data suggest that it is likely more useful to look at objective measures of bleeding rates.

A notable limitation of the analysis is that the researchers examined patient-centered outcomes reported at a single point in time, rather than tied to specific clinical events. “Further research is needed to assess whether PROs demonstrate greater reliability for hospital-level comparisons when collected around clinically relevant moments in the patient care trajectory,” they said.