TL;DR: A recent study examined the impact of critical feedback on usability measurements in consumer loan application systems. They have found that the system’s feedback influences the usability ratings. Research in other domains is advised.
Critical feedback administering systems are those that make significant decisions affecting users’ lives, such as automated medical diagnoses or bank loan approvals. Critical feedback can be administered for the purpose of learning or as a final response.
To investigate this potential bias in usability measurements, researchers conducted a study using a web-based system designed for consumer loan applications. The system provided critical feedback to applicants as a final response, either approving or rejecting their loan requests. After receiving this feedback, 332 participants completed the UMUX-LITE questionnaire, a commonly used tool for measuring perceived usability.
Participants who received positive critical feedback (i.e., their loans were approved) provided significantly higher UMUX-LITE scores compared to those who received negative critical feedback (i.e., their loans were rejected). This finding suggests that the type of critical feedback a system delivers can influence users’ perceived usability ratings. The difference in ratings was witnessed in both questions of the UMUX-Lite.
These results raise important questions about the reliability of usability measurements in systems that provide critical feedback. It suggests that users’ emotional responses and perceptions may be tied to the outcome of their interactions with the system, rather than solely reflecting the system’s inherent usability.
The implications of this study extend to usability researchers and practitioners. When assessing systems that deliver critical feedback, it is crucial to consider the potential bias in perceived usability ratings. To mitigate this bias, researchers may need to segment usability data by feedback type or explore alternative evaluation methods.
Furthermore, the study calls for further research to confirm and expand upon these findings. Future investigations could explore the impact of critical feedback on user-reported emotional ratings and whether similar biases exist in other domains beyond consumer loan applications.