Are usability measurements reliable for all kinds of systems?

TL;DR: A recent study examined the impact of critical feedback on usability measurements in consumer loan application systems. They have found that the system’s feedback influences the usability ratings. Research in other domains is advised.

Critical feedback administering systems

Critical feedback administering systems are those that make significant decisions affecting users’ lives, such as automated medical diagnoses or bank loan approvals. Critical feedback can be administered for the purpose of learning or as a final response.

A medical diagnosis system will give as critical feedback the patient's diagnosis.

A online bank loan application will give as critical feedback the loan approval or rejection.

Methodology

To investigate this potential bias in usability measurements, researchers conducted a study using a web-based system designed for consumer loan applications. The system provided critical feedback to applicants as a final response, either approving or rejecting their loan requests. After receiving this feedback, 332 participants completed the UMUX-LITE questionnaire, a commonly used tool for measuring perceived usability.

Results

Participants who received positive critical feedback (i.e., their loans were approved) provided significantly higher UMUX-LITE scores compared to those who received negative critical feedback (i.e., their loans were rejected). This finding suggests that the type of critical feedback a system delivers can influence users’ perceived usability ratings. The difference in ratings was witnessed in both questions of the UMUX-Lite.

Two bar charts.
The one on the left shows the ratings of the question "[This system's] capabilities meet my requirements". For the exp1: loan rejected score was ~3, loan approved score was ~6. For the exp2: loan rejected score was ~3, loan approved score was ~5.5. For the exp3: loan rejected score was ~3, loan approved score was ~5.5.

The one on the right shows the ratings of the question "[This system] is easy to use". For the exp1: loan rejected score was ~4, loan approved score was ~6. For the exp2: loan rejected score was ~4, loan approved score was ~6. For the exp3: loan rejected score was ~3.5, loan approved score was ~5.5.

These results raise important questions about the reliability of usability measurements in systems that provide critical feedback. It suggests that users’ emotional responses and perceptions may be tied to the outcome of their interactions with the system, rather than solely reflecting the system’s inherent usability.

Implications and Future Research

The implications of this study extend to usability researchers and practitioners. When assessing systems that deliver critical feedback, it is crucial to consider the potential bias in perceived usability ratings. To mitigate this bias, researchers may need to segment usability data by feedback type or explore alternative evaluation methods.

Furthermore, the study calls for further research to confirm and expand upon these findings. Future investigations could explore the impact of critical feedback on user-reported emotional ratings and whether similar biases exist in other domains beyond consumer loan applications.